I am querying my contacts to match a list of contacts (primary keys) on dynamodb to see if any are using my service.
I have two options to go about this:
1) client side: I call the aws sdk directly in my mobile device and handle the response accordingly.
2) via API Gateway: I send a json of my contacts to my backend (aws lambda), which computes off client and responds via json.
I am wondering what are the pros and cons of each, or if one is clearly better?
Thanks
Like many things, it depends. I don't think one is clearly better than the other.
*1 client side sdk is good because it's probably the easiest and quickest way to get going and less to build/configure/maintain.
*2 API gateway is good because it will probably be easier to call your lambda from different clients(browsers, other services,etc) and those clients wouldn't need to depend on the SDK, they could just use RESTful calls if that's how you set it up. You would also be able to support different content-types with a mapping template such as XML, YAML, etc.
It really just comes down to your use case, style, plans for reuse in the near future. You could probably start with #1 and migrate to #2 if you find you need more of the API Gateway features.
Related
I am trying to implement a gateway design to access/abstract the api to my database, which is simply a single HDB and RDB on the same server. Reading through documentation https://code.kx.com/q/wp/gateway-design/ the most basic gateways act as at least one man in the middle. Without aggregation this doubles the data transfer needed and with aggregation it seems that to be generic (implement "select" for example) it would need to pull all data to the gateway anyway (for example to perform an average that required data from both). Is there something I am missing in the design of the gateway so as not to copy the data through it, a simple and elegant solution for a small setup would be ideal. I guess this is the map reduce problem in general but in a KDB+ HDB/RDB setting.
You might want to take a look on https://www.aquaq.co.uk/kdb-gateways/
aquaq torq has implement a good gw setup. If you are setting a new kdb project, torq will be a boost starter for you.
Asyn gw dosent hold any data
Sync gw does hold data and processing can be done on sync gw itself and then return to user/client
The gateway in general doesn't hold the data. Instead it uses Inter-process Communication (IPC) to send requests to the RDB and HDB. The queries are calculated on the RDB and HDB side and the result is then sent to the gateway which is then sent to the client, but the gateway usually doesn't store the data.
I am familiar with firebase platform, but I am relatively a new user of the google cloud platform as whole.
I am working on a project built using a microservices structure, and I do have so many question for which I cannot find an answer or better I cannot find any example.
Unfortunately all the example that I am able to find are way to simple to be able to extrapolate a viable answer for my issues.
I adopted the new cloud run offer, and I decided to play with the full managed version (not kubernetes). I built few microservices (each service is built using express for node or flask for python - depending on what the services does). Each microservices expose it's own endpoint and has it's own api to call the methods - and I use a service account to allow the application to perform the internal calls.
I now want to expose the application to the external (specifically to my client built using vuejs technology), and I was trying to leverage another google product to create and expose an api: the google endpoints.
My question (specifically referred to the cloud run structure) is related to how is possible and what I need to do to create an api endpoints to communicate with the client app, that internally calls multiple services and combine their response in one.
Just to be clear, let's make an example:
Cloud run service 1 -> crud user api
Cloud run service 2 -> crud product api
Cloud endpoint external visible api -> get user from service 1, and after get products from service 2 and return the combined response all green products for user Jane Doe.
How I can aggregate the response directly in the endpoint gateway, check for failure and if everything goes smooth send the aggregate response to the client?
I need to build the aggregate endpoint in something else, like a cloud function for example? or I can do it directly in the google endpoints gateway?
Note that for cloud run the google endpoints is another cloud run container.
Thanks guys for some help, running pretty much out of option here.
As per my understanding, API Gateway should just work as a proxy, presenting all micro services as a single endpoint. To this scenarios I think you can have following 2 approaches :
1: Implement a new micro service (or on any of the existing one) which will do invocations and aggregation of responses.
2: Client(like UI) can invoke the services and do the aggregation on their side as well.
I feel, it is not a good idea to do it at api-gateway.
In my opinion, from an architectural point of view, the best option for you is to create a new microservice which will take the responses from the other two and then, it will aggregate them.
I understand that you want to aggregate the responses in a api-geteway and you are not able to find code examples for it. Here I was able to find a guide on what are you wanting to implement. The full code implementation can be found in this repository.
Keep in mind though, this idea of implementation is not a best practice.
This is ok, only if those two services that are going to be combined are independent. Meaning there is no functional/business relation between them and the concurrency or inconsistency problem will not occur in the process of aggregating.
Customers (around 1000) sign up to my service and receive a customer unique api key. They then use the key when calling a AWS lambda function through AWS api gateway in to access data in DynamoDb.
Requirement 1: The customers get billed by the number of api calls, so I have to be able to count those. AWS only provides metrics for total number of api calls per lambda so I have a few options:
At every api hit increment a counter in DynamoDB.
At every api hit enqueue a message in SQS, receive it in "hit
counter" lambda and increment a counter in DynamoDB.
Deploy a separate lambda for each customer. Use AWS built-in call
counter.
Requirement 2: The data that the lambda can access is unique for each customer and thus dependent on the api key provided.
To enable this I also have a number of options:
Store the required api key together with the data that the customer
has the right to access.
Deploy a separate lambda for each customer. Use api gateway to
protect it with a key.
Create a separate endpoint in api gateway for each customer, protect
it with the api key.
None of the options above seem like a good way to design the solution. Is there a canonical way of doing this? If not, which of the options above is the best? Have I missed an obvious solution due to my unfamiliarity with AWS?
I will try to break your problems down with my experience, but maybe Michael - Sqlbot or John Rotenstein may be able to give more appropriate answers.
Requirement 1
1) This sounds like a good approach. I don't see anything critical here.
2) This, IMHO, is the best out of the 3. It will decouple data access from the billing service, which is a great thing in a Microservices world.
3) This is not scalable. Imagine your system grows and you end up with 10K Lambda functions. Not only you'll have to build a very reliable mechanism to automate this process, but also you'll need to monitor 10K different things (imagine CloudWatch logs, API Gateway, etc), not to mention you'll have 10 thousand functions with exactly the same code (client specific parameters apart). I wouldn't even think about this one.
Requirement 2
1) It could work and it fits nicely in the DynamoDB model of doing things: store as much data as you can in a unique table, so you can fetch everything in one go. From what I see, you could even use this ApiKey as your partition key and, for the sake of simplicity for this answer, store the client's data as JSON in a column named data. Since your query only needs to query by the ApiKey, storing a JSON in DynamoDB won't hurt (do keep in mind, however, that if you need to query by any of its JSON attributes than you're in bad shoes, since DynamoDB's query capabilities are very limited)
2) No, because of Requirement 1.3
3) No, because of the above.
If you still need to store the ApiKey in a different table so you can run different analysis and keep a finer grained control over the client's calls, access, billing and etc., that's not a problem either, just make sure you duplicate your ApiKey on your ClientData table instead of creating a FK (DynamoDB doesn't support FKs, so you'd need to manage these constraints yourself). Duplication is just fine in a NoSQL world.
Your use case is clearly a Multi-Tenancy one, so I'd also recommend you to read Multi-Tenant Storage with Amazon DynamoDB which will give you some more insights and broaden your options a little bit. Multi-Tenancy is not an easy task and can give you lots of headaches if not implemented correctly. I think this is why AWS has also prepared this nice read for us :)
Happy to continue this on the comments section in case you have more info to share
Hope this helps!
This conceptual question has crept into my mind after becoming more familiar with AWS. In general, I’m curious if there is a best-practice and/or convention as to when an API provider should group endpoints into a new, separate API (vs. lumping the endpoints into an existing API).
To illustrate, let’s say a Service creates digital wallet coupons on behalf of Manufacturers, to be redeemed by Consumers at a bunch of Mom & pop stores — some of the activities the Service might engage in include:
Receiving data from the Manufacturers (in order to build the digital coupons)
Providing a mechanism for Consumers to find and download coupons
Providing a way for the Mom & pop stores’ payment terminals to validate the coupons
And, oh by the way, the Service might also be required to ...
Implement a variety of endpoints, based on technologies involved (e.g., PassKit with Apple Wallet)
So?
With AWS, it’s easy to modularize one’s backend (e.g., have an RDS instance for the database, run a few lambda functions for microservices, etc.) and load balance it all. API Gateway adds to this in that each endpoint can point to different things (lambda functions, EC2 instances via HTTP proxy, etc.).
Consequently, one approach might be to define one API in AWS API Gateway and have all the endpoints underneath it:
API: “Master”
/coupon
POST = create a new one (for Manufacturers)
PUT = update an existing one (for Manufacturers)
GET = retrieve one (for Consumers)
/coupon/validate
POST = verify it’s still valid (Mom & Pop store use-case)
/apple-wallet
/{version}
/passes
... per documentation
/devices
... per documentation
But would it make more sense for the Service to shave off the /apple-wallet endpoint and create an entirely new, separate API?
Alternatively, if the Service was going to publish documentation for public developers to use, would it make sense to move the Manufacturer-relevant endpoints into a separate API altogether?
Since AWS makes the effort of splitting endpoints so simple via API Gateway, are there any standard practices for when you should (or should not)?
Thank you for any insights / opinions!
My two cents. Think about your end-user for your APIs. You will have different developer end-users for each API set.
Your ideal situation will have each developer end-user only seeing the APIs that are relevant to them. So you should split your APIs into different Gateways according to the end-users
In the theoretical situation you describe:
Create an API for Manufacturers so they can integrate with you to create coupons. If you do the integration internally it will be the corporate sales and presales people who talk to the manufacturers
The users for the Service and End User coupons might end up being the
same app developers that create an interface for both stores and
users. So create a coupon API for them
Separating both should also give you security benefits as you will protect the knowledge of your Manufacturer API from the users who might try to hack it
I'm new to AWS, so apologies in advance if this question is missing some important considerations, or has incorrect assumptions.
But basically I want to implement a service on AWS to store and retrieve data from multiple clients, which may be Android apps, Windows applications, websites etc. The way I've considered doing this is via a RESTful service using API Gateway front end, with a Lambda back end and maybe an S3 bucket to hold the data.
The basic requirements are:
(1) Clients can publish data to the server, where it is stored, perhaps with some kind of key/value structure.
(2) Clients can retrieve said data by key.
(3) If it is possible, clients to be able to subscribe to events from the service, so that they are notified if the value of a piece of data changes. This would avoid the need to poll the service, which would presumably start racking up unnecessary charges if the data doesn't change often.
Any pointers on how to get started with this welcome!
Creating a RESTful API on top of Lambda and API Gateway is one of the main use cases for this architecture. You can think of Lambda functions as controllers with methods and API Gateway as a router that forwards requests to functions based on the URL pattern. There are many frameworks and approaches that can help out here if you don't want to write from scratch:
Lambdasync
https://medium.com/#fredrikanderzon/create-a-rest-api-on-aws-lambda-using-lambdasync-e46c68f8043f
Serverless
https://serverless.com/framework/docs/providers/aws/events/apigateway/
Swagger
https://cloudonaut.io/create-a-serverless-restful-api-with-api-gateway-swagger-lambda-and-dynamodb/
As far as event subscriptions go (requirement #3) you can model this in many datastores, certainly in a relational/SQL database, with a table like this:
Subscription (key_of_interest, user_id, events_of_interest)
I'm leaving out data types for you to figure out, but you get the idea hopefully. After each data modification on a particular key, see if that key is of interest in the subscription table, then wire up a response to the user's who indicated interest. The details of this of course depend on your particular requirements. A caution though: this approach will increase the cost of data modifications because of the additional overhead needed to process subscriptions.
EDIT: One other thing I forgot. S3 is better suited for non-structured data (think 'files'). For relational databases, checkout RDS. For a simple NoSQL database you might use DynamoDB, or host your own NoSQL database of choice on an EC2 instance.