I'm currently running an icecast server for streaming audio on an EC2 Instance.
Currently all my .mp3 files are stored on the EC2 instance and I want to move them to AWS S3 for storage. So far I've been able to find scripts that will update the playlist but will not make the server request external sources.
Is it possible to setup this architecture? Any help would be appreciated.
How about mounting the S3 bucket as a directory and just using that?
https://github.com/s3fs-fuse/s3fs-fuse / https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon
https://github.com/russross/s3fslite
As you only read the files, this should be without major issues.
Related
Problem:
We need to perform a task under which we have to transfer all files ( CSV format) stored in AWS S3 bucket to a on-premise LAN folder using the Lambda functions. This will be a scheduled tasks which will be carried out after every 1 hour, and the file will again be transferred from S3 to on-premise LAN folder while replacing the existing ones. Size of these files is not large (preferably under few MBs).
I am not able to find out any AWS managed service to accomplish this task.
I am a newbie to AWS, any solution to this problem is most welcome.
Thanks,
Actually, I am looking for a solution by which I can push S3 files to on-premise folder automatically
For that you need to make the on-premise network visible to the logic (lambda, whatever..) "pushing" the content. The default solution is using the AWS site-to-site VPN.
There are multiple options for setting up the VPN, you could choose based on the needs.
Then the on-premise network will look just like another subnet.
However - VPN has its complexity and cost. In most of the cases it is much easier to "pull" data from the on-premise environment.
To sync data there are multiple options. For a managed service, I could point out the S3 Gateway which based on your description sounds like an insane overkill.
Maybe you could start with a simple cron job (or a task timer if working with windows) and run a CLI command to sync the S3 content or just copy specified files.
Check out S3 Sync, I think it will help you accomplish this task: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html#examples
To run any AWS CLI in your computer, you will need to setup credentials, and the setup account/roles should have permissions to do the task (e.g. access S3)
Check out AWS CLI setup here: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html
I have one spring boot application which is in our internal data center, which process files from a specific folder on the host.
we wanted to deploy this to aws and wanted to use s3 bucket to upload files for processing.
is there any way we can add s3 bucket space as docker volume?
UPD: See at the bottom of this answer.
Other answers mistakenly say that AWS S3 is an object store and you can not mount it as volume to docker. Which is not correct. AWS S3 has a 3rd party FUSE driver, which allows it to be mounted as local filesystem and operate on objects as if those were files.
However it does not seem this FUSE driver has been made available as storage plugin for Docker just yet.
Edit: well, i have to correct myself after just a couple of minutes posting this. There in fact is a FUSE based driver for Docker to get volume mounted from AWS S3. See REX-ray and also here for possible configuration issue.
Other answers have correctly pointed out that :
AWS S3 is an object store and you can not mount it as volume to docker.
That being said, using S3 with spring application is super easy and there is framework developed called spring-cloud. spring-cloud works excellent with AWS.
Here is sample code :
public void uploadFiles(File file, String s3Url) throws IOException {
WritableResource resource = (WritableResource) resourceLoader.getResource(s3Url);
try (OutputStream outputStream = resource.getOutputStream()) {
Files.copy(file.toPath(), outputStream);
}
}
You can find detailed blog over here.
S3 is an object store, not a file system. You should have S3 trigger a message to SQS when new objects are added to the bucket. Then you can code your application running in the Docker container to poll SQS for new messages, and us the S3 location in the message to copy the object from S3 to local storage (using the appropriate AWS SDK) for processing.
No docker volume is for mounting drives on the machine (https://docs.docker.com/storage/volumes/)
You can use the S3 api to manage your bucket from the docker container (https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html)
I receive some large data to process and I would like to copy the files to my remote GPU server for processing.
the data contains 8000 files x 9GB/per file which is quite large.
Is it possible to copy the files from aws directly to the remote server (used with ssh)
I have googled it online and did not find anyone come up with the question..
If anyone could kindly provide a guide/url example I would appreciate a lot.
Thanks.
I assume your files are residing in S3.
If that is the case then you can simply install AWS CLI on your remote machine and use aws s3 cp command
For more details click here
I need to transfer all our files (With folders structure) to AWS S3. I have researched lot about how this done.
Most of the places mentioned s3fs. But looks like this is bit old. And I have tried to install s3fs to my exsisting CentOS 6 web server. But its stuck on $ make command. (Yes there is Makefile.in)
And as per this answer AWS S3 Transfer Acceleration is the next better option. But still I have to write a PHP script (My application is PHP) to transfer all folders and files to S3. It is working same as how file save in S3 (API putObject), but faster. Please correct me if I am wrong.
Is there any other better solution (I prefer FTP) to transfer 1TB files with folders from CentOS 6 server to AWS S3? Is there any way to use FTP client in EC2 to transfer files from outside CentOS 6 to AWS S3?
Use the aws s3 sync command of the AWS Command-Line Interface (CLI).
This will preserve your directory structure and can be restarted in case of disconnection. Each execution will only copy new, changed or missing files.
Be aware that 1TB is a lot of data and can take significant time to copy.
An alternative is to use AWS Snowball, which is a device that AWS can send to you. It can hold 50TB or 80TB of data. Simply copy your data to the device, then ship it back to AWS and they will copy the data to Amazon S3.
I have an application deployed on EB that need to download a file from a remote server then serve to visitors
As I understand, Its recommended to save files to S3 instead then grant users access to these files. However, I believe there is no option for S3 to initiate the download of a file on a remote server therefore the process would be :
EB application get the files => EB application upload the files to S3.
That would double the wait time for users.
Should I save files directly to the application directory instead as I will only use 200-300MB max then clean it daily.
Is there any risk or better approach to this problem?
Why would it double the time? The upload to S3 would be extremely quick. You could even stream the file to S3 as it is being downloaded.
Saving the files to the server will prevent you from scaling your application beyond a single server.