I have a AWS dynamodb. How to know who has updated the records in my table (not the table)? Need to know the details like logged-in user id or ARN of the AWS services which has updated the records in the table.
Updated: 8.16.2021
Cloudtrail now suppoerst tracking data events for DynamoDB
https://aws.amazon.com/about-aws/whats-new/2021/03/aws-cloudtrail-adds-logging-of-data-events-for-amazon-dynamoDB/
DynamoDB does not let you inquire which user last modified a certain item. Nor does log these data modification events anywhere. The DynamoDB Detective Security Best Practices explains your options:
If all you want to log are administrative operations, such as table creation and deletion, then AWS CloudTrail is good enough for you. This feature gives you a log of all these administrative operations, and which user did which.
However, you said that you want to know about data-plane operations (PutItem, UpdateItem, etc.), not just control-plane operations. So CloudTrail is not good enough for you. The remaining option is to use DynamoDB Streams. This creates a "stream" of modification events to your database, where each event also records the user who did this modification. A dedicated application can listen to this stream, and either record the information of who-modified-what, or react to suspicious activity, or whatever you want to do with it.
Using Streams as suggested above is neither easy nor free to do. But without doing this, the information of which user modifies which item is simply not recorded anywhere by DynamoDB.
This is where CloudTrail would come in handy. CloudTrail can be attached to services, including DynamoDB, so you can see any operations on your tables.
Here is a tutorial for it:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/logging-using-cloudtrail.html
Related
Is this possible?
I did my research but this is the only possible events for RDS:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.Messages.html
Mostly for maintenance type events but what I want is - let's I have a RDS Oracle table called Users. Whenever a record is inserted in the table, an event or stream can be picked up by a Lambda and do the necessary action.
In short, no, not with the existing events you refer to - these are for monitoring the RDS service, not what you actually use it for, i.e. contents auditing (manipulation/tracking)
You can of course create notifications when an insert occurs, but you'll probably need to build/setup a few things.
A couple of ideas:
Building something closer to the database logic, i.e. in your code base add something that fires a SQS / SNS event.
If you can't (or don't want to) modify the logic that handle the database, maybe you could add a trigger that gets fired on INSERTs to the user table. Unfortunately I don't think there's support to execute a Lamdba from a trigger (as it is possible to do with PostgreSQL at the moment).
Set up a database activity stream from RDS to Kinesis to monitor the INSERTS. This is a bit of a additional infrastructure to set up, so it might be a bit too much depending on your use case:
"Database Activity Streams is an Amazon RDS feature that provides a near real-time stream of the activity in your Oracle DB instance. Amazon RDS pushes activities to an Amazon Kinesis data stream."
From Kinesis, you can configure AWS Lambda to consume the stream and take action on INSERT events.
Some references:
https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis-example.html
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/DBActivityStreams.Enabling.html
I have information in Amazon DynamoDB, that has frequently updated/added rows (it is updated by receiving events from Kinesis Stream and processing those events with a Lambda).
I want to provide a way for other teams to query that data through Athena.
It has to be as real-time as possible (the period between receiving the event and the query to Athena including that new/updated information).
The best/most cost optimized way to do that?
I know about some of the options:
scan the table regularly and put the information in Athena. This is going to be quite expensive and not real time.
start putting the raw events in S3 as well, not just DynamoDB, and make a glue crawler that scans the new records only. That's going to be closer to real time, but I don't know how to deal with duplicate events. (the information is quite frequently updated in DynamoDB, it updates old records). also not sure if it is the best way.
maybe update the data catalog directly from the lambda? not sure if that is even possible, I'm still new to the tech stack in aws.
Any better ways to do that?
You can use Athena Federated Query for this use-case.
I'm searching for a method to track the identities which are doing modifications on my table besides the application service itself. In the beginning I though there could be two options, but:
CloudTrail - the documentation (Logging DynamoDB Operations by Using AWS CloudTrail) says, as far as I understood, I'd be only able to track changes made to the infrastructure itself, but not to the actual use of a table.
DynamoDB Streams - I'd guessed that the modifying identity is also passed in a stream event, but actually it's not. I'm using NEW_AND_OLD_IMAGES as the stream type.
Am I overlooking something or is there probably another possibility anywhere else? The streams event does pass me an EventID. Is this of use somewhere?
Grateful for any tips on how to solve this, even if it's a complete different approach.
AWS CloudTrail now supports logging for DynamoDB actions!
AWS CloudTrail Adds Logging of Data Events for Amazon DynamoDB
Is there a way to allow creation of a resource like a DynamoDB table only if the table to be created was PAY_PER_REQUEST or was provisioned with capacity below a certain amount?
I initially looked at IAM condition keys, but they appear to only be available for interactions with the table data operations (scan, update, put operations etc.) but not creation operations for the table.
Alternatively, are there ways to reduce service quotas for an account?
Ideally, I'm wondering if it is possible to scope down the ability to create DynamoDB table resources beyond a certain capacity and I’m not sure how to do it proactively instead of retroactively processing CloudTrail logs or listing existing table properties.
AWS Config
You can use AWS Config to retrospectively query AWS resources and their properties, and then determine if they are compliant or not. There are rules already available out of the box, but I can't see one which matches your use case. You will need to then write a Lambda function to implement this yourself. Here is an example.
After your rule is working you can either create a remediation action to
Delete the Table
Scale the Table Down
Send a Notification
Adjust Autoscaling (i.e. reduce max)
AWS Budgets
(My Preference)
For determining if an account is using too much DynamoDB, probably the easiest is to setup a budget for the DynamoDB Service. That would have a couple of benefits:
Auto-Scaling: Developers would be free to use high amounts of capacity (such as load tests) for short periods of time.
Potentially Cheaper: what I have found is that if you put restrictions on projects often developers will allocate 100% of the maximum, as opposed to using only what they need, in fear for another developer coming along and taking all the capacity.
Just like before with AWS Config you can setup Billing Alarms to take action and notify developers that they are using too much DynamoDB, also when the Budget is at 50%, 80% ... and so on.
CloudWatch
You could also create CloudWatch Alarms as well for certain DynamoDB metrics, looking at the capacity which has been used and again responding to excessive use.
Conclusion
You have a lot of flexibility how to approach this, so make sure you have gathered up your requirements and then the appropriate response will be easier to see. AWS Config requires a bit more work than budgets so if you can get what you want out of Budgets I would do that.
I would like to debug an issue with DynamoDB.
The provided expression refers to an attribute that does not exist in the item
For that I'd like to log all requests made to a DynamoDB Table from AWS (not from the lambda code).
I have the RequestId in the error and I wish to be able to search for it to find the exact requests with its parameters.
I have looked into AWS Cloudtrail but it seems to only log Management Operations not all gets and all puts done to DynamoDB.
Thanks
You will need to add this level of data plane logging to your application as currently CloudTrail only supports logging of control plane operations for DynamoDB.