My question is regarding the clean architecture. I am quite new to .NET Core, and my assignment is to integrate a third-party API into the console app so the data is stored in a SQL database. I have been researching for the past two days, and I found so many examples of how to create a repository service querying a local database, but I cannot see any decent example of how to integrate the data from external APIs to a local data storage solution, SQL database in my case.
Any information on how to structure services in a clean manner would help me a lot!
It basically boils down to the question whether you need to store locally data from a 3d party API. You are not the owner of that data.
In many cases it's better to query external API each time you need some data. You don't know if the data will change and when. E.g. you query each time weather data or currency convertion rate. External system knows how to produce such results, so ask the system each time. It's often ok to cache data from 3d party APIs, depending on the needs of your application (does your user needs weather updates with each refresh or once a day is enough, e.g. on a web site of a skiing area).
Now there can be cases when 3d party API is loosely coupled with your domain. E.g. you use a separate API to manage devices in a smart home solution. In this case we are talking about microservice architecture (Microservices by Fowler is a good start). It's OK in this case to build your local copy of the data from a 3d party API. Imaging your service is only interested in how often devices get replaced. So you may query device management API for broken devices and store this information locally in the form that is better for your application. You get data, reorganize it as it better fits you, store and use it.
So basically as always with such broad questions: it depends. On what are you building, on what external API provides, on what do you want to do with the data you query.
Going technical on your question: you query data e.g. via REST from a third party. This gets you TheirObjectDto. You process this data. Their object may change, you are not the owner of the contract. You create MyObject, which contains the data that you need. You save it to your DB (via repository, if you want). You build an Entity for it and DB table. When 3d party API changes and returns TheirObjectDto2 you still work with MyObject. Minimal change for you is to convert TheirObjectDto2 to MyObject. Your app continues working. In a second step if you want some new cool info of TheirObjectDto2 you modify your internal data structure.
Related
I've just started messing around with AWS DynamoDB in my iOS app and I have a few questions.
Currently, I have my app communicating directly to my DynamoDB database. I've been reading around lately and people are saying this isn't the proper way to go about getting data from my database.
By this I mean is I just have a function in my code querying my Dynamo database and returning the result.
How I do it works but is there a better way I should be going about this?
Amazon DynamoDB itself is a highly-scalable service and standing up another server in front of it requires scaling the service also in line with the RCU/WCU configured for your tables, which we can and should avoid.
If your mobile application doesn't need a backend server and you can perform all the business functions from the mobile device, then you should probably think about
Using the AWS DynamoDB SDK for iOS devices to write your client application that runs on the mobile device
Use AWS Token Vending Machine to authenticate your mobile users to grant them credentials to be used to run operations on DynamoDB tables.
Control access (i.e what operations should be allowed on tables etc.,) using IAM policies.
HTH.
From what you say, I can guess that you are talking about a way you can distribute data to many clients (ios apps).
There are few integration patterns (a very good book on this: Enterprise Integration Patterns), one of which is called shared database. It is essentially about using a common database for multiple clients to share the data. Main drawback for that pattern (in your case) is that you are doing assumption about how the database schema looks like. It can potentially bring you some headache supporting the schema in the future, if your business logic changes.
The more advanced approach would be sending events on every change in your data instead of directly writing changes to the database from client apps. This way you can add additional processing to the events before the data they carry is written to the database. For example, you may want to change the event format in the new version of your app, but still want to support legacy users, so you add translation procedure which transforms both types of events to the format which fits the database schema. It's basically a question of whether to work with diffs vs snapshots.
You should be aware of added complexity of working with events, and it can be an overkill if your app is simple and changes in schema are unlikely.
Also consider that you can do data preprocessing using DynamoDB Streams, which gives you some advantages of using events still keeping it simple to implement.
We have an internal application. As time went on and new applications were requested, that exchange data between eachother, the interaction became bound to the database schema. Meaning changes in the database require changes everywhere else. As we plan to build even more applications that will depend on the same data this quickly will become and unmanagable mess.
Now i'm looking to abstract that interaction behind an API. Currently i have trouble choosing the right tool.
Interaction at times could be complex, meaning data is posted to one service and if the action has been completed it should notify the sender of that.
Another example would be that some data does not have context without the data from other services. Lets say there is one service for [Schools] and one for [Students]. So if the [School] gets deleted or changed the [Student] needs to be informed about it immeadetly and not when he comes to [School].
Advice? Suggestions? SOAP/REST/?
I don't think you need an API. In my opinion you need an architecture which decouples your database from the domain logic and other parts of the application. Such an architecture is for example clean architecture, onion architecture and hexagonal architecture (ports&adapters by new name). They share the same concepts, you have a domain logic, which does not depend from any framework, external lib, delivery method, data storage solutions, etc... This domain logic communicates with the outside world through adapters having well defined interfaces. If you first design the inside of your domain logic, and the interfaces of the adapters, and just after the outside components, then it is called domain driven design (DDD).
So for example if you want to move from MySQL to MongoDB you already have a DataStorageInterface, and the only thing you need is writing a MongoDBAdapter which implements this interface, and ofc migrate the data...
To design the adapters you can use two additional concepts; command and query segregation (CQRS) and event sourcing (ES). CQRS is for connecting delivery methods like REST, SOAP, webapplications, etc... to the domain logic. For example you can raise a CreateUserCommand from your REST API. After that the proper listener in the domain logic processes that command, and by success it raises a domain event, like UserCreatedEvent. Your REST API can listen to that event and respond with a success message to the REST client. The UserCreatedEvent can be listened by one or more storage adapter too. So they can process that event and persist the new user. You don't necessary use only a single database. For example if a relational database is faster by a specific type of query, then you can use that, but if a noSQL database suites better to the job, then you can use that too. So you can use as many databases as you want for your queries, the only thing you need is writing a storage adapter for them. For example if your REST client wants to retrieve the profile of a specific user, then it can raise a GetUserProfileByIdQuery and the domain logic can ask the adapter of a database which can serve the query. After that the adapter can send for example an SQL query to a MySQL database and return the response. By ES you add EventStorage to your system, which stores the raised domain events. It can be very useful if you want to migrate your data from one query database to another. In that case you create a new storage adapter to your new database, and replay all of the domain events from the EventStorage in historical order to that adapter, so it can fill the new database with the relevant data. That's all, you don't have to write complicated migration scripts...
In your case I think your should create at least domain events, and use event sourcing. That will totally decouple your database from the other parts of your application. Adding a REST or SOAP API can have a similar effect, but building HTTP connections to access your database can slow down your application.
I have just started playing around with django. I can foresee one app generating data that another could use (bad example: a geomatics app crunching complex data to generate simple location data to pass to another app that uses the data to decide some sort of business logic). Having never done any web programming with frameworks, my first thought was ....globals... But thats obviously not a "good thing"!
Return a key to the client (either via a cookie or in the session) that can be used to retrieve the information from a datastore when needed.
Nowadays a lot of web applications are providing API for other applications to use.
I am new to the usage of API so I want to understand the use cases for it.
Lets take Basecamp as an example.
What are the use cases for using their API in my web application?
For inserting current data in my web application into a newly created Basecamp account instead of inserting everything manually which could take days or weeks if the data is huge?
For updating my application data when the user changes something in Basecamp. If so, how do I know for example when a user add/edit/remove a contact in Basecamp. Do I make a request and check every minute from the backend?
For making backup of the Basecamp data so I can move it to other applications if necessary?
Are all the above examples good use cases for the usage of API?
Are there more use cases?
I want to have a clear picture of why it's good to use another web service API and how I can leverage that on my application.
Thanks.
I've found the biggest reason to use and provide web services is to be able to programmatically drive the application with another process. This allows the coupling of different actions in different applications driven by one event/process/trigger.
For example I could create a use a webservice provided by Basecamp, my bug tracking database and the continuous integration server. I could tie all those things together and kick them off from a commit hook script.
I can have a monitor in production automatically open a ticket in our ticket tracker. This could trigger an autoremediation process from the ticket tracker which logs into the box remotely and restarts the service.
The other major reason I've seen to use and provide web service is to reduce double entry. If you do change management in your production environment that usually means you create Change tickets. The changes that occur may also need to be reflected in the Change Management Database which is usually a model of how production is suppose to look. Most of these systems don't automatically drive the update of your configuration item with the data from the change. Using web services you can stitch them together to eliminate the double (manual) entry that would normally occur.
APIs are used any time you want to get data to/from an application without using the default interface.
*I'd bet there's a mobile app would use the basecamp api.
*You could use the api to pull information from basecamp into another application (like project manager software or an individual's todo webpage)
*the geekiest of us may prefer to update basecamp from a script/command line rather than interrupting our work flow to open a web page and click around.
This'll be my first question on this platform. I've done lots of development using Flex, WebORB and ASP.NET. We have solved Concurrency problems with messaging (Pessimistic Concurrency Control). This works pretty good but it also makes the whole application dependent of the messaging. No messaging, no concurrency control.
I know that ASP.NET has version control in DataSets, but how would you go and use that if you are working on a RIA. It seems hard to go and store each dataset in the session of the client... So, if the Client would like need all products, I would need to store the dataset in the session of the client. When the client would change something to a product and save the product, I could then update the dataset (stored in the session) and try to save it...
Seems a lot of work and a lot of memory that will be used (because those products will be kept in the memory of the client, so the dataset needs to be kept on the server side session).
I think the most easy way would be to provide all DTO's with a version number. If the client would try to save a DTO, I could compare the version number with the one in the database.
Lieven Cardoen
This is something I've done before - as the original data was coming from an SQL Server database we just used a rowversion typed column in each DTO to determine if it had changed while the user was working on it.
At this point you can either barf on the error or try and figure out a way to merge the changes, but at least you can tell that it's changed underneath you :)