I have a java application deployed at elastic beanstalk tomcat and the purpose of the application is to serve resources from S3 in zipped bundles. For instance I have 30 audio files that I zip up and return in the response.
I've used the getObject request from the AWS SDK, however its super slow, I assume it's requesting each object over the network. Is it possible to access the S3 resources directly? The bucket with my resources is located next to the beanstalk bucket.
Transfer from S3 to EC2 is fast, if they are in the same region.
If you still want faster (and reliable) delivery of files, consider keeping files pre-zipped on S3 and serve from S3 directly rather than going through your server. You can use signed URL scheme here, so that the bucket need not be public.
Next level is speed up is by keeping the S3 behind Cloudfront as an origin server. Here the files are cached in locations near your users. Serving Private Content through CloudFront
Related
This is more of a theoretical question for AWS S3 website hosting.
Say I have a website hosted in S3. Obviously I want the content to be public, but I don't want people to be able to download the backend scripts, images, css by simply changing the domain url. I want to hide those folders, but if I deny GetObject access in the bucket policy for the folders the application "breaks" because it can't reach those folders.
How can I secure my content to ensure the most security for my backend when it sits in an S3 bucket?
You need to access the website via cloudfront with restricted access, better known as Origin Access Identity. This will allow cloudfront distribution access to s3 bucket.
More details can be found in the AWS Docs or https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html#private-content-creating-oai
There are 9 local on premises servers with unique output. Seems like the AWS storage gateway will expose one share per AWS S3 bucket.
It means that there would be total of 9 shares with 9 S3 buckets.
Is there any way to do this using just one file share?
You can setup a single s3 bucket as your data hub on aws and setup a nfs file-share for this bucket through storage gateway. Locally setup sub-folders for each data source/destination under this common nfs share.
Have servers/processes write to their own folders. Now all these folders will be replicated to the single s3 bucket
have a lambda function on s3 bucket that will replicate (aws-cli s3 sync) each of the folders to their corresponding target s3 bucket.
this approach will handle traffic for all your servers with a single bucket share. Storage gateway only supports 10 file-shares/10buckets per gateway instance. Above approach lets you go past that limit.
All the best.
I have a few EC2 instances (t2.micro) behind a load balancer on the us-east-1 region (N. Virginia) and my users are accessing the application from South America. This is my current setup mainly because costs are about 50% of what I would pay for the same services here in Brasil.
My uploads all go to S3 buckets, also in the us-east-1 region.
When a user requests a file from my app, I check for permission because the buckets are not public (hence why I need all data to go through EC2 instances) and I stream the file from S3 to the user. The download speeds for the users are fine and usually reach the maximum the user connection can handle, since I have transfer acceleration enabled for my buckets.
My issue is uploading files through the EC2 instances. The upload speeds suffer a lot and, in this case, having transfer acceleration enabled on S3 does not help in any way. It feels like I'm being throttled by AWS, because the maximum speed is capped around 1Mb/s.
I could maybe transfer files directly from the user to S3, then update my databases, but that would introduce a few issues to my main workflow.
So, I have two questions:
1) Is it normal for upload speeds to EC2 instances to suffer like that?
2) What options do I have, other than moving all services to South America, closer to my users?
Thanks in advance!
There is no need to 'stream' data from Amazon S3 via an Amazon EC2 instance. Nor is there any need to 'upload' via Amazon EC2.
Instead, you should be using Pre-signed URLs. These are URLs that grant time-limited access to upload to, or download from, Amazon S3.
The way it works is:
Your application verifies whether the user is permitted to upload/download a file
The application then generates a Pre-signed URL with an expiry time (eg 5 minutes)
The application supplied the URL to the client (eg a mobile app) or includes it in an HTML page (as a link for downloads or as a form for uploads)
The user then uploads/downloads the file directly to Amazon S3
The result is a highly scalable system because your EC2 system does not need to be involved in the actual data transfer.
See:
Share an Object with Others - Amazon Simple Storage Service
Uploading Objects Using Pre-Signed URLs - Amazon Simple Storage Service
Is it possible to use EBS like S3? By that I mean can you allow users to download files from a link like you can in S3?
The reason for this is because my videos NEED to be on the same domain/server to work correctly. I am creating a Virtual Reality video website however, IOS does not support cross-origin resource sharing through WebGL (which is used to create VR).
Because of this, my S3 bucket file system will not work as it will be classed as cross origin, but looking into EBS briefly it seems that it attaches to the all your instances as local storage which would get past the cross-origin problem I am facing.
Would it be simply like a folder on my web server, that could be reached by 'www.domain.com/ebs-file-system/videos/video.mp4'?
Thanks in advance for any comments.
Amazon S3 CORS
You can configure your Amazon S3 bucket to support Cross-Origin Resource Sharing (CORS):
Cross-origin resource sharing (CORS) defines a way for client web applications that are loaded in one domain to interact with resources in a different domain. With CORS support in Amazon S3, you can build rich client-side web applications with Amazon S3 and selectively allow cross-origin access to your Amazon S3 resources.
CloudFront Behaviours
Another option is to use Amazon CloudFront, which can present multiple systems as a single URL. For example, example.com/video could point to an S3 bucket, while example.com/stream could point to a web server. This should circumvent CORS problems.
See:
Format of URLs for CloudFront Objects
Values that You Specify When You Create or Update a Web Distribution
Worst Case
Worst case, you could serve everything via your EC2 instance. You could copy your S3 content to the instance (eg using the AWS Command-Line Interface (CLI) aws s3 sync command) and serve it to your users. However, this negates all the benefits that Amazon S3 provides.
I have a Python/Flask application that I've deployed in elastic beanstalk. I have been deploying updates via git aws.push, which includes my static js libraries, css, and images.
I now have about 1 GB of static content in the form of images. I want to serve that content from the same location as my application, that is, from the same place I was serving them before, in a /static/img/ folder. However, I obviously don't want to add the images to source control or deploy them with the git macro.
Ideally, I would like to connect to the instance where the files are hosted and upload them manually. However, I do not know how to do this. I have searched through the s3 associated with the elastic beanstalk app, but there is no sign of my app there, only a repository of zipped deployments.
I could create a new bucket and handle things that way, but I haven't been able to map a domain to a new bucket. Whenever I try to add a CNAME record to the bucket, it is rejected because "URL/IP cannot be added as a CNAME." In any case, the process that seems most intuitive is to manually put unversioned static content in place next to versioned, deployed code.
You're correct, this type of static content should not be part of your repository and certainly not stored on EC2 instance's volumes.
AWS' best practice for this use case would be to use S3 and directly link to S3 objects from your HTML code. S3 is a natively HTTP enabled object storage service.
In order to use S3 as web server, you must create a bucket on S3.
You can either use the S3 provided URL <bucket-name>.s3-website-<AWS-region>.amazonaws.com to link to your content from your web pages.
Or you can use your own domain name. In this case, your bucket name must be named after your domain name and you must enable "Website Hosting" option at the bucket level. This is required to let S3 know how to map HTTP requests to buckets.
A high level scenario is described here : http://docs.aws.amazon.com/gettingstarted/latest/swh/website-hosting-intro.html
And more details are provided by S3 documentation.
As an added benefit, storage in S3 costs less money than EBS storage.