I want to expose an API (preferably using AWS API gateway/ Lambda/Go) to the users.
Using this API, the users can download a binary file from S3 bucket.
I want to capture the metrics like, which user has started download of the file, the time at which the file download had started and finished.
I want to record these timestamps in DynamoDB.
S3 has support for Events for creating/modifying/deleting files, so I can write a lambda function for these events.
But S3 doesn't seems to have support for read actions ( e.g. download a file)
I am thinking to write a Lambda function, which will be invoked when the user calls the API to download the file. In the lambda, I want to record the timestamp, read the file into a buffer, encode it and then send it as as base64 encoded response to the client.
Let me know if there is any better alternative approach.
use Amazon S3 Server Access Logging
don't use DynamoDB, if you need to query the logs in the target bucket setup Spectrum to query the logs which are also in S3
Maybe you can use S3 Access Logs?
And configure event based on new records in log bucket. However, this logs will not tell you if user has finished download or not.
Related
I am looking to create an AWS solution where a lambda function will transform some excel data from a S3 bucket. When thinking about how I'm going to create the architecture background, I need to think of a way where I can get non-technical users, which don't have access to the AWS account, to upload data into a S3 bucket. One possible solution is using an S3 API and creating a UI to allow the users to upload the data. However, I do not have much experience with front end programming skills such as JS and HTML. Are there any other possible solutions we can use?
I've thought about creating a simple UI and using a S3 API to ingest data into the bucket but I do not have front end programming experience.
Some ideas:
Serve a web page from somewhere (S3?) and have them upload via the browser, or
Give them a simple program like CyberDuck and they can drag & drop files, or
Setup the AWS Command-Line Interface (CLI) on their computer and have them double-click a script file to sync local disk folder to the s3 bucket, or
Use Dropbox and Invoke Lambda Function on New File from Dropbox - Pipedream
The main thing to think about is how you want to secure the upload. For example, do they need to authenticate first, or do you want anyone in the world to be able to upload to the bucket (not a good idea!). Should each user have their own IAM User credentials, or should they authenticate to an application that manages its own logins?
Also, have a think about what you want to happen after they upload the file. If they need to download something after the file has been processed, then you'll need a way to 'give back' the new file.
In my AWS project, I created webservices using API Gateway, Lambda, DynamoDB and S3, that are called by an Android app.
Now, I want to log specific actions on my webservices (in my lambda functions), so I can download them from an Android app.
Here is what I was thinking of:
append my logs to a text file (or multiple text files) in S3, but then I have to download the file, append the logs, then upload the file, each time I need to add a log (doesn't sound very optimized)
store my logs in a DynamoDB table, but it doesn't look like a clean solution, and might be pricy
using CloudWatch Logs to log everything I want, but then I need to only extract the logs I need, and it seems quite complex, and not sure it's the best solution either
So what is the most suitable solution to log actions in lambda functions, so I can then download them from an app?
Thanks.
I think you can use AWS Kinesis Stream if you want to analyze your logs on the fly, or use Kinesis Firehose if you just want to aggregate your logs and store them in the same place.
Kinesis Firehose can receive logs from multiple sources, aggregate your logs and save it in S3. When the log is saved in S3, you can use AWS Athena to do queries in these logs files. To connect it with an Android Device to download the logs, you can build an API to communicate with Athena.
And if you to personalize the view for each Android Device, just make sure to include a unique ID in each log and query for this ID in Athena.
I have a requirement to send files from S3 bucket to an external client. FTP or SFTP can be used for this. Based on certain research I found this can be done using Lambda or using EC2 but couldn't find detailed steps for it. Please let me know how this can be done.
Amazon S3 cannot "send" files anywhere.
Therefore, you will need some code running 'somewhere' that will:
Download the file(s) from Amazon S3
Send the file(s) to the external client via SFTP
This is all easily scriptable. The difficulty probably comes in deciding which files to send and how to handle any errors.
You probably couldn't find any documentation on the topic because sending files via SFTP has nothing specifically related to AWS. Just do it the way you would from anywhere.
For example, let's say you wanted to do it via a Python program running either on an Amazon EC2 instance or as an AWS Lambda function:
Download the desired files by using the AWS SDK for Python (boto3). See: Amazon S3 examples
Send the files via SFTP. See: Transfer file from AWS S3 to SFTP using Boto 3
Came across a similar requirement, and this can be done very easily with the lambda function.
functional requirement for our use case was automated transfer of the files when it's ready to send back to the customer.
Architecture
We came up with this simplistic architecture for the basic use case.
Workflow
Upload a file to the S3 bucket
Trigger Push event notification for the lambda function. Prefer to have a separate lambda function for each client so that we can store all SFTP connection details in environment variables.
Env variables will be used to store Server details, credentials, file path, etc...
Lambda function will fetch a file from the S3 bucket
Lambda will transfer the file to External Server.
Worthy Addition
Worth Considering changes on top of this simple approach
If the Lambda function failed to fetch a file then it should do a couple of retries and if it still fails, they should send a notification to the client who is uploading the file to S3 bucket.
If the external transfer fails then Lambda should add that to any SQS queue from that any application can process messages and notify the system and also we can setup retry after few days again.
We provide REST api to upload user files, on backend we use S3 Bucket to store the user uploads.
As our REST api has timeout of 30sec, user may get timeouts based on filesize and his network.
So we thought of providing him a S3 Pre-Sign URLs for uploads, through which user can upload his large files using AJAX calls from frontend or using backend scheduled script.
Everything looks OK but we don't have any clue about provided s3 Pre-Signs:
- whether he attempted to upload or not:
if attempted whether the upload is success or not
if failed, what was the error (url expired or something other..)
We can know about success case by searching the object key in our bucket. But in case of failures we don't have any clue.
Please let me know if there is anyway to track the S3 pre-sign access/uploads.
You will not know when a pre-signed URL is used, but another couple of options are:
You can configure an Amazon S3 Event to trigger when a new file is uploaded to a bucket. This could trigger an AWS Lambda function that can process the file or, at least, make a log that the file was uploaded.
You can use Amazon S3 Server Access Logging to track access to the bucket.
how to save voice message of customer number and store in an s3 bucket using aws connect. I made a contact workflow but I am not understanding how to save voice message to s3 bucket?
We've tried many ways to build a voicemail solution, including many of the things you might have found on the web. After much iteration we realized that we had a product that would be useful to others.
For voicemail in Amazon Connect, take a look at https://amazonconnectvoicemail.com as a simple, no-code integration that can be customized to meet the needs of your customers and organization!
As soon as you enabled Voice Recording all recordings are placed automatically in the bucket you defined at the very beginning as you setup your AWS Connect Instance. Just check your S3 Bucket if you can spot the recordings.
By default, AWS creates a new Amazon S3 bucket during the
configuration process, with built-in encryption. You can also use
existing S3 buckets. There are separate buckets for call recordings
and exported reports, and they are configured independently.
(https://docs.aws.amazon.com/connect/latest/adminguide/what-is-amazon-connect.html)
The recording in S3 is only starting when an agent is taking the call. Currently, there is no direct voice mail feature in Amazon connect. You can forward the call to a service that allows it, such as Twillio.