Currently we have a data service that is consumed by various clients inside our organization; most of them use the PDF report to view the data. The problem we are facing is that the PDF generation is obsolete and built over a technology that is becoming hard to maintain.
So what we want to achieve now is basically two goals:
Encapsulate all data access in a SOA manner, publishing services for RAW data, PDF report and maybe some others like "Excel RAW"
Give our business users the ability to load and change the templates for the PDF reports, without asking for a "development" (basically that they can be as independent as possible)
There are two main issues; the report has to be "pretty", and by that I mean that our users ask for details such as image resolution, almost pixel accuracy for text/image positioning, and that sort of stuff. The server/library that we choose has to be able to achieve this requirement.
The other is that our technology stack currently is limited to a JAVA/LINUX platform, and while we could evaluate other platforms (for instance a product developed in .NET), a solution in Java EE/LINUX would be preferable.
Any suggestions?
P.S.: The data is stored in an Oracle database.
Certainly one prickly problem you have here is providing the ability to allow your users provide "pretty"/pixel-perfect report. Depending on what type of people your users are (technical staff / developers, business staff, general users) it may not be possible to find a system that allows you to offload significant work-effort to your users. It is just a hard domain.
If your reports are spreadsheet style, you might wish to examine systems like Business Objects, Coognos or Yellow Fin. These systems require a significant set up in terms of creating models to report against, but they can provide tools for users to design their own reports through a web interface. These systems usually stand separate to your main application though there are certainly ways to integrate them (though for exposing services to your own customers it might be difficult to get it to work exactly as you would like).
If your reports are document-style (as opposed to spreadsheet-style), you could look at Docmosis which is intended for integration with applications (please note I work for the company that created Docmosis). Docmosis allows your application to produce PDF reports from DOC/ODT/DOCX documents which act as templates for population from databases / Java objects / text etc. The templates can be provided/modified/uploaded by your users. It integrates with Java and linux environments so your technology environment is well suited. For many applications it provides automatically the layout based on the template that is desired.
With regards to the provision of SOA services to your users, it sound like a fine approach depending on the users you have (does a service approach provide them something easy to use?). Because your customers are internal I'm sure you already have determined the suitability of services.
Hope that helps.
Related
I've read articles and posts about what a project and an app is for Django, and basically end up using the typical example of Pool and Users, however a real program generally use a complex relational database, therefore its design gravitates around this RDB; and the eternal conflict raises once again about: which ones to consider an application and which one to consider components of that application?
Let's take as an example this RDB (courtesy of Visual Paradigm):
I could consider the whole set as an application or to consider every entity as an application, the outlook looks gray. The only thing I'm sure is about this:
$ django-admin startproject movie_rental
So I wish to learn from the expertise of all of you: What approach (not necessarily those mentioned before) would you use to create applications based on this RDB for a Django project?
Thanks in advance.
PS1: MORE DETAILS RELATED ABOUT MY REQUEST
When programming something I follow this steps:
Understand the context what you are going to program about,
Identify the main actors and objects in this context,
If needed, make an UML diagram,
Design a solid-relational-database diagram, (solid=constraints, triggers, procedures, etc.)
Create the relational database,
Start coding... suffer and enjoy
When I learn something new I hope they follow these same steps to understand where they want to go with their actions.
When reading articles and posts (and viewing videos), almost all of them omit the steps 1 to 5 (because they choose simple demo apps), and when programming they take the easy route, and don't show other situations or the many supposed features that Django offers (reusability, pluggability, etc).
When doing this request, I wish to know what criteria is used for experienced programmers in Django to determine what applications to create based on this sample RDB diagram.
With the (2) answers obtained so far, "application" for...
brandonris1 is about features/services
Jeff Hui is about implementing entities of a DB
James Bennett is about every action on a object, he likes doing a lot of apps
Conclusion so far: Django application is a personal creed.
My initial request was about creating applications, but as models are mentioned, I have this another question: is with a legacy relational database (as showed in the picture) possible to create a Django project with multiple apps? this is because in every Django demo project showed, every app created has a model with their own tables, giving the impression that tables do not interact with those of other applications.
I hope my request is more clear. Thanks again for your help.
It seems you are trying to decide between building a single monolithic application vs microservices. Both approaches have their pros and cons.
For example, a single monolithic application is a good solution if you have a small amount of support resources and do not need to be able to develop new features in fast sprints across the different areas of the application (i.e. Film Management Features vs Staff Management Features)
One major downside to large monolithic applications is that eventually their feature sets grow too large and with each new feature, you have a significant amount of regression testing which will need to be done to ensure there aren't any negative repercussions in other areas of the application.
Your other option is to go with a microservice strategy. In this case, you would divide these entities amongst a series of smaller services and provide them each methods to integrate/communicate with each other (APIs).
Example:
- Film Service
- Customer Service
- Staff Service
The benefits of this approach is it allows you to separate capabilities and features by specific service areas thus reducing risk and regression testing across the application when new features are deployed or there is a catastrophic issue (i.e. DB goes down).
The downside to this approach is that under true microservice architecture, all resources are separated therefore you need to have unique resources (ie Databases, servers) for each service thus increasing your operating cost.
Either of these options is a good option but is totally dependent on your support model and expected volumes. Hope this helps.
ADDITIONAL DETAIL:
After reading through your additional details, since this DB already exists and my assumption is that you cannot migrate it, you still have the same choice as to whether or not you follow a monolithic application or a microservices architecture.
For both approaches, you would need to connect your django webapp the the specific DB you are already using. I can't speak for every connector out there but I know that the MySQL connector allows django to read from the pre-existing db to systematically generate the models.py file for the application. As a part of that connector, there is a model variable which allows you to define whether or not Django is responsible for actually managing the DB tables themselves.
The only thing this changes from an architecture perspective is how many times do you want to code this connection?
If you only want to do it once and completely comply with the DRY method, you can build a monolithic application knowing that as new features become required, application wide regression testing will be an absolute requirement.
If you want ultimate flexibility for future changes with this collection of features and don't mind recoding the migration across multiple apps while reducing the need for application wide regression testing as new features become required, a microservice architecture strategy is more appropriate.
I am new to micro services and I am keen to use this architecture. I am interested to know what architecture structure should be used for systems with multiple customer interfaces where customer systems may use one or many of the available services. Here is a simple illustration of a couple of ways I think it would be used:
An example of this type of system could be:
Company with multiple staff using system for quotes of products
using products, quotes and users mirco services
Company with website to display products
using products micro service
Company with multiple staff using system for own quotes
using quotes and users micro services
Each of these companies would have their own custom build interface only displaying relevant services.
As in the illustrations all quotes, products and users could be stored local to the mirco services, using unique references to identify records for each company. I dont know if this is advisable as it could make data difficult to manage and could grow fast making it difficult to manage.
Alternatively I could store such as users and quotes local to the client system and reference the micro services for data thats generic. Here mirco services could be used just to handle common logic and return results. This does feel someone illogical and problematic to me.
I've not been able to find anything online to explain the best course of action for this scenario and would be grateful for any experienced feedback on this.
I am afraid you will not find many useful recipes or patterns for microservice architectures yet. I think that the relative quiet on your question is that it doesn’t have enough detail for anybody to readily grasp. I will make a wag:
From first principles, you have the concept of a quote which would have to interrogate the product to get a price and other details. It might need to access users to produce commission information, and customers for things like discounts and lead times. Similar concepts may be used in different applications; for example inventory, catalog, ordering [ slightly different from quote ].
The idea in microservices is to reduce the overlap between these concepts by dispatching the common operations as their own (micro) services, and constructing the aggregate services in terms of them. Just because something exists as a service does not mean it has to be publicly available. It can be private to just these services.
When you have strained your system into these single function services, the resulting system will communicate more, but will be able to be deployed more flexibly. For example, more resources &| redundancy might be applied to the product service if it is overtaxed by requests from many services. In the end, infrastructure like service mesh help to isolate the implementation of these micro services from the sorts of deployment considerations.
Don’t be misled into thinking there is a free lunch. Micro service architectures require more upfront work in defining the service boundaries. A failure in this critical area can yield much worse problems than a poorly scaled monolithic app. Even when you have defined your services well, you might find they rely upon external services that are not as well considered. The only solace there is that it is much easier to insulate your self from these if you have already insulated the rest of your system from its parts.
After much research following various courses online, video tutorials and some documentation provided by Netflix, I have come to understand the first structure in the diagram in the best solution.
Each service should be able to effectively function independently, with the exception of referencing other services for additional information. Each service should in effect be able to be picked up and put into another system without any need to be aware of anything beyond the API layer of the architecture.
I hope this is of some use to someone trying to get to grips with this architecture.
I'm an architect in a large scale financial company and we are in the beginning of implementing a new business oriented infosystem across our different countries.
From the very early on the core idea has been to follow microservice oriented principles as much as possible (and making sure engineers have read Building Microservices book by Sam Newman).
By now I've come to crossroads. Our services are primarily JSON REST services using Swagger for automated documentation, but in order to use these services in our business processes and making sure not to write business logic into services outside the domain of those services, we've been using Camunda as an orchestration tool. And Camunda is fine (though some have considered Corezoid as an alternative), but somewhat clumsy in what is an otherwise an elegant set of services.
Now service orchestration is a concept pretty familiar to most engineers. But it is one that I am not entirely happy with due to still having a central engine that drives everything. It is incredibly expensive to replace later down the road (though still cheaper to replace than a monolith). And even if this central engine is split into multiple engines (which is actually the case today), it does not necessarily make it much better.
In recent years there has been a movement with microservices towards choreographed (close to event-driven) architecture. It is at this point where I am looking for advice from engineers and architects who have faced similar crossroad decision points.
I absolutely love the idea of decoupled architecture and despite feeling good about killing monoliths and having elegant independent services, I still detect a lot of dependencies in business process as a whole in current orchestrated solution in where it should not actually exist.
And it's not like we are avoiding events. We have actually implemented events on our architecture as well in order to decouple many processes with the core principle that if you don't need a synchronized response and just need to notify of something happening to initiate another process an event is put up that may be caught by another process that starts executing. And orchestration is easier to explain and visualize, it is easier to tweak and modify by more technical minded business users. And I think it is easier to test and validate from business perspective. Orchestrated architecture like this also (usually) expects a good service discovery and quality automated documentation and non-functional requirements which are all things I value greatly.
All of those things that are a question to me in choreographed approach since I don't have first-hand experience in running this in large scale - just some local test prototypes.
But I think you see where I am coming from. I'm trying to consider alternatives without having to regret driving the company all the other way in the end.
Perhaps you can share your own experience with a similar situation or share an interesting link or two? Or am I looking for a silver bullet that doesn't exist yet?
Services need to interact - services that don't interact are not part of the same system. The search needs access to the catalog, the cart doesn't get the price info from the page, the account needs the purchase history, the recommender needs purchase history, the cart needs to verify the currently available coupons, the inventory needs to know something was purchased etc.
Service boundaries are set to minimize the needed interactions. It can make sense to cut a service to smaller components but if they share a database (internal structure) they are different aspects of the service.
When services interact it creates a level of coupling - at the least, this coupling is some API (JSON or otherwise) that the service has to "maintain" for so other services can interact with it.
Another coupling type is temporal coupling - which is what you get in request-reply situations (and you can eliminate in event driven systems) However, Orchestration vs. Choreography is not about these differences (even though orchestration is mostly associated with request/reply) - it is about central control and governance vs.flexibility and serendipity.
Orchestration has risks like migrating business logic out of services into the orchestration while choreography runs the risk of chaos. By the way, direct request/reply integration has the worst of both worlds but wins on simplicity when systems are small enough.
Choosing between the two is a balancing act (like most architectural decisions) for instance, Netflix built on choreography for a lot of time but then found they need some control back and introduced an orchestration engine. Nothing is a silver bullet :)
Personally, I like choreography better because of the reduced coupling and flexibility and favor tools like open Zipkin to bring some order into the chaos.
You can see a partial example for an orchestration based arch in slides 10-22 of a presentation I did about microservices
I think I understand where you're coming from, having recently redesigned a system to a "microservices" architecture. I like (and use) the approach by these guys: http://scs-architecture.org/
The main point is, that you try to avoid cross-dependencies between you "services", which basically makes choreography obsolete. The hard part is decomposing your problem domain into chunks which do not need eachother for any of the executed business-cases. They may need different kinds of data that may or may not be "shared", as in present in multiple systems, but they don't need synchronous calls between them for any given business case.
This is quite different from what Netflix is doing for example. Those guys/gals are doing chain-calling through different layers of services, each adding its logic to the "process". This model might fit in some cases, and probably fits in Netflix's case. But it may not be necessary for you.
The ideal Self-Contained System would be completely independent of other Self-Contained Systems, would cover one or more highly cohesive business functions (in full depth from the UI to Persistence!), and would be not calling any other system synchronously. The ideal system would let the client "orchestrate", by just offering links through its Web (HTML) interface.
Think more like Amazon. The "Landing Page" is a different application than the "Search", which is still different from the "Checkout". They are completely different, sometimes even look a bit different! Integrated by links and forms in HTML, not explicitly orchestrated.
This might be what you are looking for.
Some warnings: First instinct of some people is to have "Customer" microservice, or "Product Repository" microservice, and similar. This will not lead to Self-Contained Systems, as you will need synchronous calls to these things, making them essentially "central" components. The key is to split the business domain, so bounded contexts a la Eric Evans.
For personal and university research reasons I am thinking of building a simple CRM using a service oriented architecture. Its meaning is just to explain the architecture itself, not commercial use.
I was thinking of implementing a CRM that offers a simple analytics service and customer care (user storing, personal comments, and few other things).
The architecture that I'm designing defines:
- WebGUI (a client of the other services)
- AnalyticsService (a service that receives data, analyzes and collect it)
- CustomerCareService (a service that uses RESTful APIs to apply CRUD operations).
Each service has it own database, being completely independent from others. They expose a public interface. The interface of course must provide some sort of authentication, to deny unautorized requests.
The advantages I'd like to explain in this kind of architecture is the possibility to have all things indepentent and the ability to combine them to offer new services (for example if there was an OrderService to handle orders it would be easy to combine it with Customer using the public APIs). The big advantage to me is that it'd be easy enough to build other clients that use these services.
I don't know what is some good Authentication method, that could be easy to implement, I'm also not sure about how to make this APIs (use XML or plain REST APIs with GET/POST data). I've worked with Amazon, PayPal and other company APIs, they seem to use REST services (paypal uses an ugly _cmd GET parameter while Amazon uses better URI) to know what to do, but reading something about SOAs it appears that people also use XML. Of course I also need to take into account that the web interface must be able to recognize the logged in user, get the permissions (token or whatever else) and use it with services to show information.
So I'm not sure SOA is the kind of architecture I'm really building up... is it SaaS instead of SOA?
I think it would be better to use RESTful applications, with JSON or something like that to implement it (I'm not a big fan of XML, I find it to be too verbose).
For clarity I'm listing here my questions:
Is this kind of architecture called SOA or SaaS (or both)?
What is a good implementation for what I want to obtain? (please explain it as more detailed as possible)
What sort of authentication is more suitable for a client (user token vs OAuth or similiar)
Do you have some suggestion for this kind of project?
I've about 3 months to do it, so I cannot do something real complex (beside the fact that it would not be realistic for a single programmer).
I know Python (WSGI frameworks), Ruby on Rails, C/C++ and other languages (.net excluded) and I'd like to develop it under a Linux environment (MySQL or Postgres, or even a NoSQL if you have any suggestion for the right choice), I could also combine several languages being these services independent programs.
What I'd like here is to have some good point of view and some good suggestion.
Thanks!
I would define SaaS as a Business model rather than an architecture; however like all business domain requirements it will influence systems architecture but it, itself is not. What you have defined is essential a Service Oriented Architecture.
Your statement "independent and the ability to combine them to offer new services" is the essential non-functional design requirement that suggests SOA.
Good implementation for SOA is about having well defined and flexible interfaces, with very clear delineation of responsibilities. However it is difficult subject to be prescriptive about. The proof is in the eating; does it provide that flexible reuse. My suggestion is spend time reading SOA design pattern resources, and understand the defining characteristics with regard to the appropriate context for use. Then apply the Single Responsibility principle appropriate level of abstraction. c.f. (Domain) Space Based Architecture is kind of SOA meta-pattern.
In regard to Authorisation, I recommend following the service approach, use a distribute directory services system like open LDAP, and note that is entirely reasonable for service provides and users to have their own credentials and you can use Public-Private keys for signing messages.
The main suggestion is study and learn from experience of others:
http://www.soapatterns.org/
http://martinfowler.com/eaaCatalog/
SOA doesn't forces to use XML.
Currently web technologies dominate, and define future.
So we in my company selected JSON RESTful services as foundation. And SOA as principles.
There is no sense to suggest languages, because the purpose of SOA and good implementation is
- to enable any language or framework to be used
(FYI we use Java with Spring MVC-based web-services, Node.js, PHP)
We have a website, where transactions are entered in and put through a workflow. We are going to follow the standard BLL(Business Logic Layer), DTO(Data Transfer Object), DAL(Data Access Layer) etc. for a tiered application. We have the need to separate everything out because some transactions will cross multiple applications with different business logic.
We also have a backend processor. It handles our transactions once the workflow has been completed. It works with various third party systems, some of which are unstable, or the interface to them is unstable, and then reports the status of the transaction. Each website will have its own version of the backend processor.
Now the question, with N-Tier, they suggest a new BLL for each application. With the layout of the application above, it can be argued that the backend processor and website is one application acting in unison, or two applications with different business logic. What would be the ideal way to handle this? Have it act like one system, or two?
One thing that I picked up on while learning MVC over the last couple years is the difference between what I call application logic and domain logic. I don't like the term business logic anymore, because it has too much baggage from all the conflicting theories and practices that have used that term too loosely.
Domain logic is the "traditional" business logic, how things are supposed to act, what they require (validation), etc. Application logic is anything that is specific to a given presentation of your domain, IE when the user clicks this submit button in your web app then they are directed to this web page over here (note that this has nothing to do with how a WinForms app or a background processor would work). Application logic should live in your application. Domain logic should live in your BLL and lower, and be reusable across the different applications that may use your common "business logic".
Kind of a general answer, but I hope that helps.
You might consider partitioning the functionality to reflect the organization of the stakeholders. Usually if you have two distinct organizational groups, then development and administration requirements are easier to manage if the functionality is similarly partioned. And vise versa.
Most of us don't spend that much time writing applications that explore the outer boundaries of hardware and software capabilities.
If you separate your concerns well then I think that you will be able to view them as the same application with a single business logic layer, there is no point writing the same code twice. The trick will be forcing the separation of concerns between the user interface portions of the website and the business logic in your BLL library.
Performance is going to be an issue as well, you have to ensure that your batch processing doesn't block your website from performing tasks that it needs to perform due to your resources. This may be an argument to keep them more separate, however as they're likely sharing a database anyway (or some other file based resource) then that may be an issue regardless.
I would keep a common business logic library programmed to interfaces and fully separated from your other concerns.
The "Ideal" way to do this depends on the project at hand and the various requirements of the system.
My default design is to have it act as one app. But if there are more heavyweight processes taking place, I like to create a batching process where the parameters of the requested job are stored and acted upon by a seperate process.