How to access, upload and delete objects of the S3 bucket from the web URL securely?
We are accessing the objects in S3 from our Application. But that bucket is public which is not secure.
I have tried CloudFront with OAI on the s3 bucket and putting bucket private but access is denied from the application when trying to upload an object to the s3 bucket.
We want to upload and delete objects in s3 bucket. We want that bucket to private only. And we want to do this from web applications only not from CLI, not from any tool. How could we achieve this?
Any suggestion will be appreciated.
Thanks!
Your application can use an AWS SDK to communicate directly with AWS services.
Your application will require a set of credentials to gain access to resources in your AWS Account. This can be done in one of two ways:
If your application is running on an Amazon EC2 instance, assign an IAM Role to the instance
Otherwise, create an IAM User and store the AWS credentials on the application server by using the AWS CLI aws configure command
You can control the exact permissions and access given to the IAM Role / IAM User. This can include granting the application permission to access your Amazon S3 buckets. This way, the buckets can be kept private, but the application will be able to upload/download/delete objects in the bucket.
To add more to the previous answer, you can find many S3 SDK examples in the AWS Github located here:
https://github.com/awsdocs/aws-doc-sdk-examples
If you look under each programming language, you will find Amazon S3 examples. You can use the AWS SDK to perform actions on a bucket when its private. You can take security a step further too and use encryption as shown here:
https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/java/example_code/s3/src/main/java/aws/example/s3/S3EncryptV2.java
Also, you can interact with Amazon S3 bucket from a web app as well by using the AWS SDK. Here is an example of building a web app using Spring Boot that interacts with an Amazon S3 bucket by reading all of the objects in the bucket.
Creating an example AWS photo analyzer application using the AWS SDK for Java
It's a bad practice to use long term credentials. AWS recommends to use short term credentials along with STS. Here is an article using Python/Flask to upload a file into private S3 bucket using STS/short term credentials.
Connect on-premise Python application with AWS services using Roles
I could have listed down all the steps in this post. But, it's a bit too long and so the reference to the above link.
Related
I have a Laravel application that is hosted on AWS. I am using an S3 bucket to store files. I know that I have successfully connected to this bucket because when I upload files, they appear as I would expect inside the bucket's directories.
However, when I try to use the URL attached to the uploaded file to display it, I receive a 403 Forbidden error.
I have an IAM user set up named laravel which has the permission AmazonS3FullAccess applied to it, and I am using that key/secret.
I have the Object URL like so:
https://<BUCKET NAME>.s3.eu-west-1.amazonaws.com/<DIR>/<FILENAME>.webm
But if I try to access that either in my app (fed into an audio player) or just via the link directly, I get a 403. None of the tutorials I've followed to get this working involve Bucket Policies, but when I've googled the problems I'm having, Bucket Policy seems to come up.
Is there a single source of truth on how I am to do this? My AWS knowledge is very limited, but I am trying to get better!
When you request a URL of the form https://bucket.s3.amazonaws.com/dog/snoopy.png, that request is unauthenticated. Your S3 bucket policy does not allow unauthenticated access to the contents of the bucket so that request is denied with 403.
If you want your files to be downloadable by an unauthenticated/anonymous client then create an S3 bucket policy to allow that.
Alternatively, your server can create signed URLs and share those with the client.
Otherwise, your client's requests need to be authenticated, which means having correctly-permissioned credentials and using an AWS SDK.
Typically, back-end applications that you write that need access to data in S3 (or other AWS resources) would be given AWS credentials allowing the necessary access. If your back-end application runs in AWS then you would do that by launching the compute with an IAM role.
Typically, front-end applications would not have AWS credentials. Instead they would either authenticate to a back-end that then does work with AWS resources on their behalf. There are other options, however, such as AWS Amplify apps.
I'm trying to make a private bucket on AWS s3 and read and write to it from an Android App.
I found many of resources and questions but nothing of it gave me a complete answer.
I made a private one but don't know what should I write in Bucket Privacy, and what I need to access it from an Android app.
How can I do it?
More details:
My app is an e-commerce app has users, they upload photos to the app and view it from a website and from the app too.
All data on my server except photos, I want to upload the photos to a private bucket on s3 and view those by URLs saved on my server. Can I do it?
Restrict access to IAM role for S3 bucket. Use this role in Android App.
The normal architecture for this is:
Keep the Amazon S3 bucket private (no Bucket Policy)
Users of the Android app provide their login information to the app, which authenticates against your back-end service
Once authenticated, your back-end service can generate temporary credentials using AWS Security Token Service (STS) — permissions are assigned to these credentials that grant access to Amazon S3 (eg only for a certain path within a particular bucket)
The mobile app can then use these temporary credentials to directly communicate with Amazon S3 to upload/download objects
You could use Amazon Cognito for authentication and provisioning of credentials, or you could code your own authentication process (eg checking against your own database).
References:
Using Amazon Cognito for Mobile Apps - AWS Identity and Access Management
AWS IAM Now Supports Amazon, Facebook, and Google Identity Federation | AWS News Blog
About SAML 2.0-based Federation - AWS Identity and Access Management
If you only wish to view photos, then a simpler method would be for the back-end to generate Amazon S3 pre-signed URLs, which permit time-limited access to private objects in Amazon S3.
See: Amazon S3 pre-signed URLs
You can do this both by:
creating a bucket policy for specific IAM user
creating an ACL policy for an account and then delegating permissions to a specific user
I'm in the process of developing a web application (using Angular6) that uses aws amplify.
The storage module provided by amplify lets you store your files in three protective levels (public, protected & private). I have a requirement to process an uploaded file via a lambda function.
My questions is whether the s3 buckets (and 'folders') created via the amplify available to Lambda functions (as the buckets are encrypted to use only via the app)??
would changing CORS on the S3 bucket do the trick?. Any help appreciated.
An S3 bucket that is created by Amplify CLI is like any other S3 bucket. You can access it provided that appropriate permissions are in place. https://docs.aws.amazon.com/AmazonS3/latest/user-guide/set-permissions.html
I've been using AWS Codedeploy using github as the revision source. I have couple of configuration files that contains credentials(e.g. NewRelic and other third party license key) which I do not want to add it to my github repository. But, I need them in the EC2 instances.
What is a standard way of managing these configurations. Or, what tools do you guys use for the same purpose?
First, use IAM roles. That removes 90% of your credentials. Once you've done that, you can store (encrypted!) credentials in an S3 bucket and carefully control access. Here's a good primer from AWS:
https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2
The previous answers are useful for managing AWS roles/credential specifically. However, your question is more about general non-AWS credentials, and how to manage them securely using AWS.
What works well for us is to secure the credentials in a properties file in a S3 bucket. Using same technique as suggested by tedder42 in A safer way to distribute AWS credentials to EC2, you can upload your credentials in a properties file into a highly secured S3 bucket, only available to your instance, which has been configured with the appropriate IAM role.
Then using CodeDeploy, you can add a BeforeInstall lifecycle hook to download the credential files to a local directory via the AWS CLI. For example:
aws s3 cp s3://credentials-example-com/credentials.properties
c:\credentials
Then when the application starts, it can read those credentials from the local file.
Launch your EC2 instances with an instance profile and then give the associated role access to all the things your service needs access to. That's what the CodeDeploy agent is using to make calls, but it's really there for any service you are running to use.
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html
Can you connect to S3 via s3cmd or mount S3 to and ec2 instance with IAM users and not using access keys?
All the tutorials I see say to use access keys but what if you can't create your own access keys (IT policy).
There are two ways to access data in Amazon S3: Via an API, or via URLs.
Via an API
When accessing Amazon S3 via API (which includes code using an AWS SDK and also the AWS Command-Line Interface (CLI)), user credentials must be provided in the form of an Access Key and a Secret Key.
The aws and s3cmd utilities, and also software that mounts Amazon S3 as a drive, require access to the API and therefore require credentials.
If you have been given a login to an AWS account, you should be able to ask your administrators to also create credentials that are associated with your User. These credentials will have exactly the same permissions as your normal login (via Username/password), so it's strange that they would be disallowing it. They can be very useful for automating AWS activities, such as starting/stopping Amazon EC2 instances.
Via URLs
Objects stored in Amazon S3 can also be made available via a URL that points directly to the data, eg s3.amazonaws.com/bucket-name/object.txt
To provide public access to these objects without requiring credentials, either add permission to each object or create a Bucket Policy that grants access to content within the bucket.
This access method can be used to retrieve individual objects, but is not sufficient to mount Amazon S3 as a drive.