I have a large bucket on AWS. The program I'm using EVS requires that all videos and photos must be inone bucket instead of creating new buckets.
So my bucket now has a crapload of stuff in there. Is there a way that I can just skip to the item I want in the bucket without having to scroll through the entire bucket.
thanks
If you're using the AWS web console, you can just start typing and it will filter the files in the bucket.
Related
Use-case
We basically want to collect files from external customers into a file server.
We were thinking of using the S3 bucket as the file server that customers can interact with directly.
Question
Is it possible to accomplish this where we create a bucket for each customer, and he can be given a link to the S3 bucket that also serves as the UI for him to drag and drop his files into directly?
He shouldn't have to log-in to AWS or create an AWS account
He should directly interact with only his S3 bucket (drag-drop, add, delete files), there shouldn't be a way for him to check other buckets. We will probably create many S3 buckets for our customers in the same AWS account. His entry point into the S3 bucket UI is via a link (S3 bucket URL perhaps)
If such a thing is possible - would love some general pointers as to what more I should do (see my approach below)
My work so far
I've been able to create an S3 bucket - grant public access
Set policies to Get, List and PutObject into the S3 bucket.
I've been able to give public access to objects inside the bucket using their link, but never their bucket itself.
Is there something more I can build on or am I hitting a dead-end and this is not possible to accomplish?
P.S: This may not be a coding question, but maybe your answer could have code to accomplish it if at all possible, general pointers if possible
S3 presigned url can help in such cases, but you have write your own custom frontend application for drag and drop features.
Link: https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html
I am using RESTful API, API provider having images on S3 bucket more than 80GB size.
I need to download these images and upload in my AWS S3 bucket, its time taking job.
Is there any way to copy image from API to my S3 bucket instead of I download and upload again.
I talked with API support they saying you are getting image URL, so its up to you how you handle,
I am using laravel.
is it way to get the sourced images url's and directly move images to S3 instead of first I download and upload.
Thanks
I think downloading and re-uploading to different accounts would be inefficient plus pricey for the API Provider. Instead of that I would talk to the respective API Provider and try to replicate the images across accounts.
Post replicate you can Amazon S3 inventory for various information related to the objects in the bucket.
Configuring replication when the source and destination buckets are owned by different accounts
You want "S3 Batch Operations". Search for "xcopy".
You do not say how many images you have, but 1000 at 80GB is 80TB, and for that size you would not even want to be downloading to a temporary EC2 instance in the same region file by file which might be a one or two day option otherwise, you will still pay for ingress/egress.
I am sure AWS will do this in an ad-hoc manner for a price, as they would do if you were migrating from the platform.
It may also be easier to allow access to the original bucket from the alternative account, but this is no the question.
The problem:
I have an old S3 bucket: bucket A and a new S3 bucket: bucket B. These buckets are in separate accounts. Up until now, I have been serving assets from bucket A. Moving forward, I want to serve assets from bucket B. I must still support pushing to bucket A. However, those assets pushed to bucket A must be retrievable from bucket B.
Possible solutions:
On every new push to bucket A (PutObject), I must sync that object from bucket A to bucket B. As I understand it, there are two ways to achieve this:
Using AWS Lambda with Amazon S3
Using DataSync <-- preferred solution
Issue with solution 2:
I have a feeling the path using DataSync will be less complex. However, it's not clear to me how to accomplish this, or if it is even possible. The examples I see in the documentation (granted there is a lot to sift through) are not quite the same as this use-case. In the console, it does not seem to allow a task across multiple AWS accounts.
The disconnect I'm seeing here is, the documentation implies it is possible. However, when you navigate to DataSync Locations in the AWS Console, there is only the option to add locations in your AWS accounts S3 bucket list.
I try and succeed to upload a file using AWS Amplify quick start doc and I used this example to set my graphql schema, my resolvers and dataSources correctly: https://github.com/aws-samples/aws-amplify-graphql.
I was stuck for a long time because of an error response "Access Denied" when my image was uploading into the S3 bucket. I finally went to my S3 console, selected the right bucket, went to the Authorization tab, and clicked on "Everyone" and finally selected "Write Object". With that done, everything works fine.
But I don't really understand why it's working, and Amazon show me a big and scary alert on my S3 console now saying "We don't recommend at all to make a S3 bucket public".
I used Amazon Cognito userPool with Appsync and it's inside my resolvers that the image is upload to my S3 bucket if i understood correctly.
So what is the right configuration to make the upload of an image work?
I already try to put my users in a group with the access to the S3 bucket, but it was not working (I guess since the user don't really directly interact with my S3 bucket, it's my resolvers who do).
I would like my users to be able to upload an image, and after displaying the image on the app for everybody to see (very classical), so I'm just looking for the right way to do that, since the big alert on my S3 console seems to tell me that turning a bucket public is dangerous.
Thanks!
I'm guessing you're using an IAM role to upload files to S3. You can set the bucket policy to allow that role with certain permissions whether that is ReadOnly, WriteOnly, etc.
Take a look here: https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html
Ok I find where it was going wrong. I was uploading my image taking the address of my S3 bucket with the address that was given by aws-exports.js.
BUT, when you go to your IAM role policy, and you check the role of your authorize user of your cognito pool, you can see the different strategies and the one that allow to put objects on your S3 bucket use the folders "public", "protected" and "private".
So you have to change those path or add these folder at the end of your bucket address you use on your front-end app.
Hope it will help someone!
I have two buckets with large quantity of pdf files. And, I wanted to make these searchable with file name and content after indexing all documents. I tried using the CloudSearch but it appeared to be useful for same data type. Please guide me how I can make documents searchable in amazon s3 bucket for a domain name or using any web browser.
CloudSearch can index PDFs. You can submit that data from S3 buckets using the AWS CLI or the web console. This functionality is documented here http://docs.aws.amazon.com/cloudsearch/latest/developerguide/uploading-data.html
If you want something automated, AWS Lambdas can monitor your buckets for changes and submit new documents for indexing.