I want to implement atomic transactions over multiple AWS resources -- e.g. uploading an object to S3 and adding a record to a DynamoDB table. Both should happen in lockstep -- or not at all. If one of the operations fails, the other should be rolled back. I understand I can implement it myself, but I was wondering if there is an existing library that does it.
One of the challenges while implementing this is expiry of temporary credentials. What if credentials expire after one of the operations was performed?
Any suggestions?
Transactions are hard! Especially in a distributed system. Transactions are also slow.
If there is any way to redesign your system to not require transactional semantics, I strongly encourage you to try.
If you really need transactions, involving multiple AWS resources, across different services.. you sort of have to roll your own. You can leverage a distributed data store that supports atomic operations and build on top of that.
It won’t be easy.
Related
Does anyone have any performance metric links or tips & tricks lists for using AWS Lambdas in cooperation with a backing RDMS database (Aurora with MySQL or PostgreSQL background) and/or DocumentDB. I want to use this functionality but I want to get at least some idea of what my performance calculations will need to be to determine whether or not it is feasible for certain operations.
I see things like these:
https://docs.aws.amazon.com/lambda/latest/dg/configuration-database.html
https://cloudonaut.io/passwordless-database-authentication-for-aws-lambda
etc.
However, nothing seems to include performance metrics. Since database authc/authz can be expensive, I want to see what I'm in for. I don't want to blow pricing for concurrent connections out of the water nor have excessive delays when a Lambda needs to access something in some cases.
The issue here is trying to run mostly serverless for some things but understanding that it's simply not the best solution for all scenarios.
We have a "shared" layer that has a few resources accessed by different services in the project. There is a table storing shared information (user permission on each of the resources in the project, since it can get big so not being stored in JWT token)
Should we have a Lamba read the dynamoDB table and give other microservices access to the shared lambda only or should we give the microservices access to the table directly so that they can just use a lib method to read the permissions from the table? I am leaning towards direct DynamoDB table access since that avoids the extra hoop through a lambda.
Both approaches have advantages & disadvantages:
Direct Access to DynamoDB - Good Sides
The authors of the other Lambda functions can build on their own phases. Faster teams can sprint and not wait for the slower team
If one lambda function is misbehaving / failing, the other lambdas are still decoupled from it and the blast radius gets limited
Direct Access to DynamoDB - Bad sides
The effort for writing similar stuff is duplicated in different lambda instances.
Each lambda can write their own logic and introduce differences in implementations. This could be intentionally designed to work that way but it could also be that one developer misunderstood the requirements
If this DynamoDB gets poisoned by wrong coding by one of the consuming lambdas, the other lambdas can also go down.
It becomes hard to measure the reserve capacity, Some of the lambdas can easily become greedy when it comes to read units.
Mediating Lambda - Good Sides
Reduces the effort required to implement similar logic for different consumers
If the shared lambda that manages the DynamoDB is performing actions like audit trail storing, you will be able to easily measure the required read & write capacity units.
If it is decoupled from the consumers, then the failure can be reduced and contained within it.
Mediating Lambda - Bad Sides
This shared lambda can easily become a single point of failure if the consuming lambdas are expecting return values from it.
More communication is required between the team managing this lambda and the consuming teams. Politics can easily be introduced by this Lambda :D
If the consuming teams are developing in a much faster rate than the owner of this shared lambda, it could easily be a blocker to other teams if integration is done poorly.
I'm using https triggered Google Cloud Functions to handle client requests to perform database writes. The data is structured in a way that most in parallel writes will not result in corruption.
There are few cases where I need to prevent multiple write actions to happen at once for the same item. What are the common patterns to lock access to some resource on the function level. I'm looking for some "mutex-like" functionality.
I was thinking of some external service that could grant or deny access to the resource for requesting function instances, but the connection overhead would be huge - handshake each time etc.
Added an example as requested. In this specific case, restructuring the data to keep the track of updates isn't a suitable solution.
import * as admin from "firebase-admin";
function updateUserState(userId: string) {
// Query current state
admin
.database()
.ref()
.child(`/users/${userId}/state`)
.once("value")
.then(snapshot => {
return snapshot.val() || 0;
})
.then(currentState =>
// Perform some operation
modifyStateAsync(currentState)
)
.then(newState => {
admin
.database()
.ref()
.child(`/users/${userId}/state`)
.set(newState);
});
}
This is not a pattern that you want to implement in Cloud Functions. Restricting the parallelism of Cloud Functions would limit its scalability, which is counter to the way Cloud Functions works. To learn more about how Cloud Functions scales, watch this video.
If you have a database that needs to have some protection against concurrent access, you should be using the database's own transaction features. Pretty much every database that provides concurrent access to data also provides some ability to perform atomic transactions. Use these transactions, and let the serverless container scale up and down in the way it sees fit.
In the Google Cloud there is an elegant way to have a global distributed mutex for a critical section in a Cloud Function:
gcslock
This is a library written in Go language, hence available for Cloud Functions written in Go, that utilises atomicity guarantees of the Google Cloud Storage service. This approach is apparently not available in AWS because of the lack of such guarantees in the S3 service.
The tool is not applicable for every use case. Acquiring and releasing the lock are operations of order of 10ms, which might be too much for high speed processing use cases.
For a typical batch process, that is not time critical, the tool provides pretty interesting option of guaranteeing that your Cloud Function is not running concurrently over the same target resource. Just create the lock file in GCS with the name that is unique for the operation that you'd like to put into the critical section and release it once its done (or rely on the GCS object lifecycle management to clean the locks up).
Please see more considerations and pros and cons in the original tool GitHub project.
There is also apparently an implementation of the same in Python.
Here is a nice article that summarises use cases for distributed locking on GCP in particular.
I'm building an auction house system for an MMO, the system is pretty straightforward:
Players can post items to sell, item will be retained inside the order and appear on the market
Players can post items to buy, gold will be retained inside the order and system will look for orders, if there is a match, it will swap.
We're considering using AWS to make things simpler to scale. But, we're also having some concerns in matter of security, since any hacks on the auction house would pretty much ruin the game. And, well, since it's an online game, there will be people trying to hack it pretty much from day 1.
Is the serverless architecture recommended for this type of system in terms of security? Are there any unforeseen problems I might run in the future?
Is NoSQL(DynamoDB) recommended for this? I can visualize relational databases keeping things a bit more secure because of integrity restriction and ACID, but I'm a bit uncomfortable with having my code handling all the relationships in such a critical software.
For the DB architecture:
I think the key point here is your system will look for orders, if there is a match sentence. What matters is your indexing to achieve this. If you can create an index that is good (has a large number of distinct values that are requested fairly uniformly) then those items in the mall would be less costly and fast for querying. Otherwise searching items in the mall will be very inefficient, for instance, if your index key is say GoldenSword and 10000+ of items have the same id GoldenSword. I don't know the requirements for your 'Auction house' like in how many ways you can search or add so I can't speculate much on this. But a downside is if your DB architecture changes in time or is weakly designed at first it forces you to use Global Secondary Indexes and become very costly as DynamoDB is costly it self.
Best practices for DynamoDB
For the safety:
It is very fine and secure if you have decent restrictions on your IAM Roles and policies for accessing dynamoDB and for running lambda on your AWS account. Not like giving full access to everything and all dynamoDB operations and tables. As long as your IAM credentials are safe they will be safe. Never seen a security issue regarding to serverless architecture. Plus DynamoDB has a easy to use encryption at rest feature.
I am planning to build a mini Content Management System. Checking the possibility of storing the Content on a DynamoDB, Will the services be able to access the content while Updating the same content? (Scenario of updating the content on CMS and publishing)
Or CloudSearch will be the better solution instead of DynamoDB in such use case?
Thanks in advance!
Please think about your use case and decide whether it requires eventually consistent read or strongly consistent read.
Read Consistency
Eventually Consistent Reads
When you read data from a DynamoDB table, the response might not reflect the results of a recently completed write operation. The response might include some stale data. If you repeat your read request after a short time, the response should return the latest data.
Strongly Consistent Reads
When you request a strongly consistent read, DynamoDB returns a response with the most up-to-date data, reflecting the updates from all prior write operations that were successful. A strongly consistent read might not be available in the case of a network delay or outage.
Note:-
DynamoDB uses eventually consistent reads, unless you specify otherwise. Read operations (such as GetItem, Query, and Scan) provide a ConsistentRead parameter: If you set this parameter to true, DynamoDB will use strongly consistent reads during the operation.
AWS DynamoDB tool (Java) for transaction management:-
Out of the box, DynamoDB provides two of the four ACID properties:
Consistency and Durability. Within a single item, you also get
Atomicity and Isolation, but when your application needs to involve
multiple items you lose those properties. Sometimes that's good
enough, but many applications, especially distributed applications,
would appreciate some of that Atomicity and Isolation as well.
Fortunately, DynamoDB provides the tools (especially optimistic
concurrency control) so that an application can achieve these
properties and have full ACID transactions.
You can use this tool if you are using Java or AWS SDK Java for DynamoDB. I am not sure whether similar tool is available for other languages.
One of the features available on this library is Isolated Reads.
Isolated reads: Read operations to multiple items are not interfered with by other transactions.
Dynamodb transaction library for Java
Transaction design