I'm searching for a method to track the identities which are doing modifications on my table besides the application service itself. In the beginning I though there could be two options, but:
CloudTrail - the documentation (Logging DynamoDB Operations by Using AWS CloudTrail) says, as far as I understood, I'd be only able to track changes made to the infrastructure itself, but not to the actual use of a table.
DynamoDB Streams - I'd guessed that the modifying identity is also passed in a stream event, but actually it's not. I'm using NEW_AND_OLD_IMAGES as the stream type.
Am I overlooking something or is there probably another possibility anywhere else? The streams event does pass me an EventID. Is this of use somewhere?
Grateful for any tips on how to solve this, even if it's a complete different approach.
AWS CloudTrail now supports logging for DynamoDB actions!
AWS CloudTrail Adds Logging of Data Events for Amazon DynamoDB
Related
Is this possible?
I did my research but this is the only possible events for RDS:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.Messages.html
Mostly for maintenance type events but what I want is - let's I have a RDS Oracle table called Users. Whenever a record is inserted in the table, an event or stream can be picked up by a Lambda and do the necessary action.
In short, no, not with the existing events you refer to - these are for monitoring the RDS service, not what you actually use it for, i.e. contents auditing (manipulation/tracking)
You can of course create notifications when an insert occurs, but you'll probably need to build/setup a few things.
A couple of ideas:
Building something closer to the database logic, i.e. in your code base add something that fires a SQS / SNS event.
If you can't (or don't want to) modify the logic that handle the database, maybe you could add a trigger that gets fired on INSERTs to the user table. Unfortunately I don't think there's support to execute a Lamdba from a trigger (as it is possible to do with PostgreSQL at the moment).
Set up a database activity stream from RDS to Kinesis to monitor the INSERTS. This is a bit of a additional infrastructure to set up, so it might be a bit too much depending on your use case:
"Database Activity Streams is an Amazon RDS feature that provides a near real-time stream of the activity in your Oracle DB instance. Amazon RDS pushes activities to an Amazon Kinesis data stream."
From Kinesis, you can configure AWS Lambda to consume the stream and take action on INSERT events.
Some references:
https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis-example.html
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/DBActivityStreams.Enabling.html
I would like to debug an issue with DynamoDB.
The provided expression refers to an attribute that does not exist in the item
For that I'd like to log all requests made to a DynamoDB Table from AWS (not from the lambda code).
I have the RequestId in the error and I wish to be able to search for it to find the exact requests with its parameters.
I have looked into AWS Cloudtrail but it seems to only log Management Operations not all gets and all puts done to DynamoDB.
Thanks
You will need to add this level of data plane logging to your application as currently CloudTrail only supports logging of control plane operations for DynamoDB.
I have a AWS dynamodb. How to know who has updated the records in my table (not the table)? Need to know the details like logged-in user id or ARN of the AWS services which has updated the records in the table.
Updated: 8.16.2021
Cloudtrail now suppoerst tracking data events for DynamoDB
https://aws.amazon.com/about-aws/whats-new/2021/03/aws-cloudtrail-adds-logging-of-data-events-for-amazon-dynamoDB/
DynamoDB does not let you inquire which user last modified a certain item. Nor does log these data modification events anywhere. The DynamoDB Detective Security Best Practices explains your options:
If all you want to log are administrative operations, such as table creation and deletion, then AWS CloudTrail is good enough for you. This feature gives you a log of all these administrative operations, and which user did which.
However, you said that you want to know about data-plane operations (PutItem, UpdateItem, etc.), not just control-plane operations. So CloudTrail is not good enough for you. The remaining option is to use DynamoDB Streams. This creates a "stream" of modification events to your database, where each event also records the user who did this modification. A dedicated application can listen to this stream, and either record the information of who-modified-what, or react to suspicious activity, or whatever you want to do with it.
Using Streams as suggested above is neither easy nor free to do. But without doing this, the information of which user modifies which item is simply not recorded anywhere by DynamoDB.
This is where CloudTrail would come in handy. CloudTrail can be attached to services, including DynamoDB, so you can see any operations on your tables.
Here is a tutorial for it:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/logging-using-cloudtrail.html
I need your experience to determine which is the best tool to invoke an AWS lambda from another one. Here are the known tools, if you can give me your pros and cons for the one you know and its efficiency:
Invoke function
DynamoDB Stream
Kinesis Stream
SNS
SQS
S3 Bucket and Put Object
Any other proposal?
Thanks a lot for your help to determine the best strategy.
Note: I am using serverless and NodeJS if it can lead to another compatible option.
In my case, I have no real problem. I just want to take advantage of your experiences using this tool. I both need s3 for PDF files and dynamoDB to store. I just would like to use one of the available tool to communicate between my different components (lambdas) of my API. Maybe some of you think SNS should be the best option. Why. Some other S3? etc. This is not specific if my usage but yours in fact ;-) I think it is just difficult to determine the best adapted choice for a newcomer. In my case I would like to uniformize my communication between my different services (modularity/reproductive method) without any constraint of what service actually does. A kind of universal lambda communication tool.
You're thinking about this in the wrong way. You don't choose one mechanism over another like this. Instead, the mechanism is dictated by how events are being generated. Did an update happen to a DynamoDB table? Then a DynamoDB Streams event triggers a Lambda function. Did a file get uploaded to S3? Then an S3 object uploaded event triggers a Lambda function.
I can't seem to find the documentation on what kinds of events DynamoDB is able to trigger a lambda function based on. All I can find is mentions of when a new record is added to a table or a record is updated. Are those the two "only" actions/events available? Or could I also trigger a lambda function when I request a records that does not exists (which is what I need in my case, where I will be using DynamoDB as a cache)?
Triggering AWS Lambda through events happening in DynamoDB is done by utilizing DynamoDB Streams.
As stated in the documentation:
DynamoDB Streams captures a time-ordered sequence of item-level modifications in any DynamoDB table, and stores this information in a log for up to 24 hours.
So they only capture operations which modify data, which isn't the case for read operations.
Triggering a Lambda function automatically because somebody queried for a key that doesn't exist is not supported by DynamoDB. You would have to handle that in your querying code.