My application runs on the client PC. It produces log files including error reports and user's action.
To collect and analyze log files, I try to upload log files to Amazon S3 from the client PC.
But is it safe? My app has no authentication so that users can upload unlimited number of files. I am concerned with maricious user upload a fake error report and huge file. I'd like s3 bucket not to exceed free quota. Is there any best practice for this task?
Just make sure that the files you are uploading to Amazon S3 are kept as Private and the Amazon S3 bucket is kept as private. These are the default settings and are enforced by Amazon S3 block public access unless somebody has specifically changed the settings.
With this configuration, the files are only accessible to people with AWS credentials that have been granted permission to access the S3 bucket.
Additionally to John's answer you can use AWS KMS (https://aws.amazon.com/kms/?nc1=h_ls) to encrypt your data at rest.
With regards of the file size, you should limit the size of the uploaded file in your application I would say.
Related
I am currently looking for a solution to store temporary files in AWS. I want to create a functionality on my app that allows my customers to upload a file and send it by email (something like WeTranfer or Send Anywhere).
I want to save the file temporarily on my AWS storage, for 10 hours and then remove it permanently. If the file has not expired, the user can click the link (provided by AWS) on the email and download the file.
I recently came across S3 Bucket Lifecycle rules, but I can only specify days for the expiration and not hours.
I would appreciate any suggestion. Thank you!
Amazon S3 is the appropriate place to store these files.
If you want access controls (to control which users can access the file) and fine-grained control over when the object 'expires', then you would need to code this yourself.
The files should be stored in a private Amazon S3 bucket. You would then need a back-end app that manages user authentication. When an authorized user requests access to a file, the app can generate an Amazon S3 pre-signed URL, which provides time-limited access to private objects in Amazon S3 (eg 10 hours). This is the link you would put into the email.
Deletion could still be handled by S3 Lifecycle rules, but it is less important when the file is actually deleted because the pre-signed URL would block access to the file after 10 hours anyway.
I can't find some information about Amazon S3, hope you will help me. When is a file available for user to download, after the POST upload? I mean some small JSON file that doesn't require much processing. Is it available to download immediately after uploading? Or maybe amazon s3 works in some sessions and it always takes a few hours?
According to the doc,
Amazon S3 provides strong read-after-write consistency for PUTs and DELETEs of objects in your Amazon S3 bucket in all AWS Regions.
This means that your objects are available to download immediately after it's uploaded.
An object that is uploaded to an Amazon S3 bucket is available right away. There is no time period that you have to wait. That means if you are writing a client app that uses these objects, you can access them as soon as they are uploaded.
In case anyone is wondering how to programmatically interact with objects located in an Amazon S3 bucket through code, here is an example of uploading and reading objects in an Amazon S3 bucket from a client web app....
Creating an example AWS photo analyzer application using the AWS SDK for Java
I have created an S3 bucket, not sure what am I missing with IAM lifecycle policies.
Files in s3 bucket are automatically moving to tombstone folder after few days. how to stop this?
I have enabled only "Server access logging" in properties tab. And there are no life cycle rules are attached.
You can enable Amazon S3 Server Access Logging. following these instructions
Server access logging provides detailed records for the requests that are made to a bucket. Server access logs are useful for many applications. For example, access log information can be useful in security and access audits.
I have to upload video files into an S3 bucket from my React web application. I am currently developing a simple react application and from this application, I am trying to upload video files into an S3 bucket so I have decided two approaches for implementing the uploading part.
1) Amazon EC2 instance: From the front-end, I am hitting the API and the server is running in the Amazon EC2 instance. So I can upload the files into S3 bucket from the ec2 instance.
2) Amazon API Gateway + Lambda: I am directly sending the local files into an S3 bucket through API + Lambda function by calling the https URL with data.
But I am not happy with these two methods because both are more costly. I have to upload files into an S3 bucket, and the files are more than 200MB. I don't know I can optimize this uploading process. Video uploading part is necessary for my application and I should be very careful to do this part and also I have to increase the performance and cost-effective.
If someone knows any solution please share with me, I will be very helpful for me to continue my process.
Thanks in advance.
you can directly upload files from your react app to s3 using aws javascript sdk and cognito identity pools and for the optimization part you can use AWS multipart upload capability to upload file in multiple parts I'm providing links to read about it further
AWS javascript upload image example
cognito identity pools
multipart upload to S3
also consider a look at aws managed upload made for javascript sdk
aws managed upload javascript
In order to bypass the EC2, you can use a pre-authenticated POST request to directly upload you content from the browser to the S3 bucket.
As per my project requirement, I want to fetch some files from on-prem FTP server & put them into a S3 bucket. Files are of size 1-2 GB. Once the file will be put into the FTP server folder, I want that file to be uploaded to S3 bucket.
Please suggest the easiest way to achieve this?
Note- Mostly the files will be put into FTP server only once in a day, hence i dont want continuously scan the FTP server. once the files will be uploaded to S3 from FTP server, i want to terminate any resources (like EC2) created in AWS.
These are my ideas:
I think you could create an agent on your FTP server that will upload the files every N seconds/minutes/hours/Etc using the AWS CLI. This way you're avoiding external access to your FTP server.
Another approach is a Lambda function for pulling process, but like you said the FTP server doesn't allow external access.
Create a VPN between your on-prem and the cloud infra, create a Cloudwatch event and through a Lambda execute the pulling process.
Here you can configure a timeout:
Create a VPN between your on-prem and the cloud infra, from your FTP server upload the files using AWS CLI (pay attention to sync option). Take a look at this link: https://aws.amazon.com/answers/networking/accessing-vpc-endpoints-from-remote-networks/
With Jenkins create a task to execute a process that will upload the files.
You can use Storage gateway, visit its site here: https://aws.amazon.com/es/storagegateway/
Here is how we solved it.
Enable S3 acceleration on your S3 bucket. This is very much needed, since you are pushing large file.
If you have access to the server install aws cli and perform a sync on the folder to s3 bucket. AWS CLI will automatically sync your folder to bucket. This way if you change any of your existing files, it will keep in sync with S3 bucket. This is ideal and simplest way if you have access to the server and able to install aws cli.
https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration-examples.html#transfer-acceleration-examples-aws-cli
aws s3api put-bucket-accelerate-configuration --bucket bucketname --accelerate-configuration Status=Enabled
If you want to enable for specific or default profile,
aws configure set default.s3.use_accelerate_endpoint true
If you don't have access to ftp server in your premisis, you need an external server to perform this process. In this case you need to perform a poll or share file system, copy the file locally and move it to s3 bucket. There will be lot of failure points with this process.
Hope it helps.