I've looked all over but can't seem to see how this might be done.
I have a GCP bucket, which is publicly accessible. I need a link to give to an associate which they can use to upload some files into that bucket. There is no need for authentication, the files are just public domain anyway. I need the process to be super simple for the associate.
Once the files are uploaded I can grab them and destroy the bucket/project anyway.
Is this possible?
Use a signed URL. From the documentation (emphasis mine):
In some scenarios, you might not want to require your users to have a Google account in order to access Cloud Storage, but you still want to control access using your application-specific logic. The typical way to address this use case is to provide a signed URL to a user, which gives the user read, write, or delete access to that resource for a limited time. You specify an expiration time when you create the signed URL. Anyone who knows the URL can access the resource until the expiration time for the URL is reached or the key used to sign the URL is rotated.
Related
I'm creating a platform whereby users upload and download data. The amount of data uploaded isn't trivial---this could be on the order of GB.
Users should be able to download a subset of this data via hyperlinks.
If I'm not mistaken, my AWS account will be charged for the egress of downloaded these files. If that's true, I'm concerned about two related scenarios:
Users who abuse this, and constantly click on the download hyperlinks (more than reasonable)
More concerning, robots which would click the download links every few seconds.
I had planned to make the downloads accessible to anyone who visits the website as a public resource. Naturally, if users logged in to the platform, I could easily restrict the amount of data downloaded over a period of time.
For public websites, how could I stop users from downloading too much? Could I use IP addresses maybe?
Any insight appreciated.
IP address can be easily changed. Thus, its a poor control, but probably better than nothing.
For robots, use capcha. This is an effective way of preventing automated scraping of your links.
In addition, you could considered providing access to your links through API gateway. The gateway has throttling limits which you can set (e.g. 10 invocations per minute). This way you can ensure that you will not go over some pre-defined.
On top of this you could use S3 pre-signed URLs. They have expiration time so you could adjust this time to be valid for short time. This also prevents users from sharing links as they would expire after a set time. In this scenario, he users would obtained the S3 pre-signed urls through a lambda function, which would be invoked from API gateway.
You basically need to decide whether your files are accessible to everyone in the world (like a normal website), or whether they should only be accessible to logged-in users.
As an example, let's say that you were running a photo-sharing website. Users want their photos to be private, but they want to be able to access their own photos and share selected photos with other specific users. In this case, all content should be kept as private by default. The flow would then be:
Users login to the application
When a user wants a link to one of their files, or if the application wants to use an <img> tag within an HTML page (eg to show photo thumbnails), the application can generate an Amazon S3 pre-signed URLs, which is a time-limited URL that grants temporary access to a private object
The user can follow that link, or the browser can use the link within the HTML page
When Amazon S3 receives the pre-signed URL, it verifies that it is correctly created and the expiry time has not been exceeded. If so, it provides access to the file.
When a user shares a photo with another user, your application can track this in a database. If a user requests to see a photo for which they have been granted access, the application can generate a pre-signed URL.
It basically means that your application is in control of which users can access which objects stored in Amazon S3.
Alternatively, if you choose to make all content in Amazon S3 publicly accessible, there is no capability to limit the downloads of the files.
I uploaded an image to Google Cloud Storage, but someone used the URL of the image for their website without my consent, which caused me to lose money. How can this be prohibited?
As #Doug and gauillaume# said in the comments, by default, a storage URL is accessible for anyone connected to internet who has the URL.
If you want only entitled persons to have access to you Storage objects you can use Signed URLs.
A signed URL is a URL that provides limited permission and time to make a request. Signed URLs contain authentication information in their query string, allowing users without credentials to perform specific actions on a resource.
Here you can read more about how to process of generating signed URLs looks like and how to achieve it.
There are also other security considerations for Storage buckets that may be worth reading about. For example if you do not need a 100% secure bucket you can simply choose some very difficult name for buckets and objects. That way it will be very difficult for a link to be guessed by a person which is not entitled. More strong solutions would be ACLs and different IAM roles attributions.
I'm storing user images on S3 which are readable by default.
I need to access the images directly from the web as well.
However, I'd like to prevent hackers from brute forcing the URL and downloading my images.
For example, my S3 image url is at http://s3.aws.com/test.png
They can brute force test and download all the contents?
I cannot set the items inside my buckets to be private because I need to access directly from the web.
Any idea how to prevent it?
Using good security does not impact your ability to "access directly from the web". All content in Amazon S3 can be accessed from the web if appropriate permissions are used.
By default, all content in Amazon S3 is private.
Permissions to access content can then be assigned in several ways:
Directly on the object (eg make an object 'public')
Via a Bucket Policy (eg permit access to a subdirectory if accessed from a specific range of IP addresses, during a particular time of day, but only via HTTPS)
Via a policy assigned to an IAM User (which requires the user to authenticate when accessing Amazon S3)
Via a time-limited Pre-signed URL
The most interesting is the Pre-Signed URL. This is a calculated URL that permits access to an Amazon S3 object for a limited period of time. Applications can generate a Pre-signed URL and include the link in a web page (eg as part of a <img> tag). That way, your application determines whether a user is permitted to access an object and can limit the time duration that the link will work.
You should keep your content secure, and use Pre-signed URLs to allow access only for authorized visitors to your web site. You do have to write some code to make it work, but it's secure.
Let's say that I want to create a simplistic version of Dropbox' website, where you can sign up and perform operations on files such as upload, download, delete, rename, etc. - pretty much like in this question. I want to use Amazon S3 for the storage of the files. This is all quite easy with the AWS SDK, except for one thing: security.
Obviously user A should not be allowed to access user B's files. I can kind of add "security through obscurity" by handling permissions in my application, but it is not good enough to have public files and rely on that, because then anyone with the right URL could access files that they should not be able to. Therefore I have searched and looked through the AWS documentation for a solution, but I have been unable to find a suitable one. The problem is that everything I could find relates to permissions based on AWS accounts, and it is not appropriate for me to create many thousand IAM users. I considered IAM users, bucket policies, S3 ACLs, pre-signed URLs, etc.
I could indeed solve this by authorizing everything in my application and setting permissions on my bucket so that only my application can access the objects, and then having users download files through my application. However, this would put increased load on my application, where I really want people to download the files directly through Amazon S3 to make use of its scalability.
Is there a way that I can do this? To clarify, I want to give a given user in my application access to only a subset of the objects in Amazon S3, without creating thousands of IAM users, which is not so scalable.
Have the users download the files with the help of your application, but not through your application.
Provide each link as a link the points to an endpoint of your application. When each request comes in, evaluate whether the user is authorized to download the file. Evaluate this with the user's session data.
If not, return an error response.
If so, pre-sign a download URL for the object, with a very short expiration time (e.g. 5 seconds) and redirect the user's browser with 302 Found and set the signed URL in the Location: response header. As long as the download is started before the signed URL expires, it won't be interrupted if the URL expires while the download is already in progress.
If the connection to your app, and the scheme of the signed URL are both HTTPS, this provides a substantial level of security against any unauthorized download, at very low resource cost.
I work on a SaaS application where Creators can create Groups and invite others to their Group to share files, chat and so on. Only people within specific group should have access to this group's files.
People from other group must not have access to not their group's files.
And of course all files permission should be set to 'Private', i.e. they should not be searchable/visible/accessable by anonymous users of Internet since information in those files is for personal use only.
I am new to Amazon S3 and don't know how to achieve it... Should I create only 1 main bucket? Or create for each group a new Amazon Bucket?
It is not recommended to use AWS Identity and Access Management (IAM) for storing application users. Application users should be maintained in a separate database (or LDAP, Active directory, etc).
Therefore, creating "one bucket per group" is not feasible, since it is not possible to assign your applications users to permissions within Amazon S3.
The better method would be to manage permissions within your application. When a user requests access to a file, the application can determine whether they should be permitted access. If they are permitted, then the application can generate a Pre-Signed URL.
A Pre-Signed URL permits access to private objects stored on Amazon S3. It is a means of keeping objects secure, yet granting temporary access to a specific object.
When listing available files, your application would generate links that include the pre-signed URL. Then, when a user clicks the link, they can access the file. Then, after a certain time has expired (eg 10 minutes), the link will no longer function. So, if a user shares a link with somebody else, it will probably have timed-out.
See: Creating a pre-signed URL in Ruby