I have a DynamoDB table and I am querying using one of my GSI some condition.
Now what I want is to graph everything that this query returns in Cloudwatch
Is that possible?
Also Dynamo has this limitation of only showing a max of 300 items at the time. Can I at least see the total of items the query is returning?
Related
I want to create a pool of users in Dynamo DB and limit the number of users in it.
For that I want to either restrict the number of entries in the table maintaining the pool or create an alarm as soon as the size crosses a threshold.
I know I can monitor the size through DescribeTable operation but it is not real time and updates every 6 hours.
If this isn't possible with Dynamo DB, what other better options are available in AWS meeting this requirements?
You could create a new DDB table (or use your existing 'pool' table, if you have one) and add an atomic counter field to track how many users are in the pool, just be sure to add a condition to your writes so that it fails when the max number of users has been reached.
However, as you queried, Dynamo isn't well suited to this type of problem. A small RDS or ElastiCache instance will both set you back about $11 a month are both more straightforward for managing state like that.
I have one dynamo table. Any data insert to a table it will trigger one lambda function. So in loop when I hit data to dynamo table. Some time trigger not happening for one or two rows
What is the solution for trigger not happening for loop
i monitor data quality of ingested data in AWS. Example:
ID
customer
Sales
Cost
1
henry
5000
4000
2
melissa
10000
abc
One dataset might have 20 validation checks. Each validation check creates an entry in a result table.
validation run
validation name
checked column
unexpected values
2021-11-04
are values numeric
Sales
0
2021-11-04
are values numeric
Cost
1
Now i want to create an alert when one of these entries are >0 (like in column "unexpected values").
Target should be that the alert goes to the SNS Topic. I know Cloudwatch or DynamoDB can monitor tables, but more on the infrastructure site (is data ingested,...). Do you know how i can monitor and create an event if certain values are present in the table?
Do you know how i can monitor and create an event if certain values are present in the table?
You stream to lambda function, and function monitors the values. If the lambda detects the values of interests, its going to perform further actions, e.g. send SNS notification or trigger some automations.
If you really want CloudWatch Alarms, then the lambda function would populate custom CloudWatch Metrics, for which you would setup your alarms.
I have a table in DynamoDB which has millions of records. I have created a secondary index (GSI) based on the criteria and filtering products based on that. Now, I wanted to use AWS datapipe line to query products from the table and export it to S3.
Questions:
a) Can we Specify GSI name in the pipeline - Because querying on a large table using data pipeline is getting cancelled because of timeout issue. [The pipeline configuration has 6 hrs max wait time, it is reaching that and getting cancelled]?
b) Is there any better way to create an export dumps from the table quickly using the GSI index?
Please share your views.
Regards,
Kishore
You cannot specify the GSI in the pipeline. The list of available options you can specify for a dynamodb node are given here. The data pipeline service actually creates an EMR cluster for the export job which uses parallel table scans. You could try using a larger instance size for your nodes to speed it up the process.
Since your table has millions of records, make sure you have provisioned enough read throughput. Even if your provisioned throughput is high, the speed of export depends on what percent of the provisioned throughput is allocated for the export job. This is described in the AWS pipeline documentation here.
Is there any way to get the DynamoDB data in batches.
I want to fetch 2000 items from Dynamo DB but in batches, like 100 records per minute and feed it to the lambda function.
I've tried AWS batch, but it doesn't seems to be fruitful.