Our application exposes queries by way of web services, and what we've found is that our clients often want custom queries, either by way of further limiting the results returned by specifying additional criteria, or by asking for things that we don't already expose.
Now, we can take the approach of creating new methods for each of these new methods, but that's somewhat inconvenient; deployment of our application at a client site usually requires weeks of staged integration testing. We've proposed a named query mechanism, where the application administrator would define queries by name that are parameterized, and a corresponding web service that simply invokes these parameters. However, I can't help but think that someone has solved this problem before, so I'd like some input from the SO community on possible designs.
Thanks!
Updates
The specification pattern is a good one, but our application deals with enough data that we want to push as much of the querying work down into an RDBMS, which can do a better job of optimizing the query plan than we would ever want to. Moreover, we support three RDBMS backends, so we're stuck using a greatest-common-denominator approach: we use as much capability as the least functional database can provide.
I would also recommend to consider the "Specification Pattern" in this type of applications as a design decision for your backend. Check the following posts about "Specification Pattern":
http://www.mattberther.com/2005/03/25/the-specification-pattern-a-primer/
http://devlicio.us/blogs/jeff_perrin/archive/2006/12/13/the-specification-pattern.aspx
Take a look at Hibernates Criteria API and use it or build some similar
functionality for Your users.
If it's worth the effort, provide a tree-like interface for grouping criterias. ("all criteria of a group must match" / "one criteria must match" / "negate")
Advantages:
Easy to build.
User parameters are possible.
Powerful queries are possible.
You can apply restrictions like SELECT ... FROM table WHERE someRestriction AND (user-provided criteria)
Since we really don't know which how your users use your interface it seems a little premature to give a technical advice on something that feels a lot closer to "Inmates are running the Asylum" problem.
There are some very good advice and common ways to solve this i technical aspects but do they work for your users? Maybe the really don't give a crap about your problem but rather have a fine working one button solution? (Or more like google?)
Related
We're designing an app that will generate lots of different types of text output, eg email, html, sms, etc. The output will be generated using some kind of template, with data coming from a db. Our requirements include:
Basic logic / calculated fields within template. Eg "ifs" and "for" loops, plus some things like adding percentages for tax etc
Runtime editing. Our users need to be able to tweak the templates to their needs, such as change boilerplate text, add new logic, etc
Multi lingual. We need to choose the correct template for the current culture.
Culture sensitive. Eg dates and currencies will output according to current ui culture.
Flexibility. We need the templates to be able to handle multiple repeating groups, hierarchies, etc.
Cannot use commercial software as a solution (e.g. InfoPath). We need to be able to modify the source code at any time.
The app is c#.net. We are considering using T4, XML + XSLT or hosting the Razor engine. Given that the syntax cant be too overwhelming for non-techie users, we'd like to get your opinion on which you feel is the right templating engine for us. We're happy to consider ones notalready mentioned too.
Thanks.
I'm very hesitant to try and answer this question on a forum, because technology choices depend on far more factors than are conveyed in the question, including things such as attitude to risk, attitude to open source, previous good and bad experiences, politics and leadership on the project etc. The big advantage of XSLT over Razor is that it's a standard and has multiple implementations on multiple platforms (including at least three implementations on .NET!) so there's no lock-in; but that doesn't seem to be a factor in your statement of requirements. And the fact that you're using .NET suggests that supplier lock-in isn't something that worries you anyway.
One thing to bear in mind is that non-programmers often take to XSLT a lot more quickly than programmers do. Its rule-based declarative approach, and its XML syntax, sometimes make programmers uncomfortable (it's not like anything they have seen before) but end-users often take to it like ducks to water.
We've decided to go with Razor Hosting. The reason why I've posted this is an answer is that I thought it would help others if I include the following article link:
http://www.west-wind.com/weblog/posts/2010/Dec/27/Hosting-the-Razor-Engine-for-Templating-in-NonWeb-Applications
This excellent piece of work by Rick Strahl makes it really easy to host Razor.
I've been using some code that implements the phpBB DBAL for some time. Recently I had to implement a more full package around it and decided to use the DBAL throughout. In the main, it's been OK. But occassionally there are circumstances where I can't see the logic in using it. It seems to make the simple much more complicated.
What benefits does a DBAL offer rather then writing sql statements directly?
From wikipedia (http://en.wikipedia.org/wiki/Database_abstraction_layer) :
API level abstraction
Libraries like OpenDBX unify access to databases by providing a single low-level programming interface to the application developer. Their advantages are most often speed and flexibility because they are not tied to a specific query language (subset) and only have to implement a thin layer to reach their goal. The application developer can choose from all language features but has to provide configurable statements for querying or changing tables. Otherwise his application would also be tied to one database.
When cooking a dish, you do not want several chefs having access to the pot. They could all be adding spices unaware that another chef had already added a spice. Ideally, you want a single chef that would serve as a single point of access to avoid spoiling the soup.
The same with databases. A single point of access can avoid problems of multiple services accessing the data in different ways.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions concerning problems with code you've written must describe the specific problem — and include valid code to reproduce it — in the question itself. See SSCCE.org for guidance.
Closed 9 years ago.
Improve this question
When developing an application which mostly interacts with a database, what is a good way to start? The application requires a lot of filtering based on user input, sorting and structuring.
The best way to start is by figuring out "user stories" (or "use cases" -- but the "story" approach tends to really work great and start dragging shareholder into the shared storytelling...!-); on top of that, designing the database schema as the best-normalized idea you can find to satisfy all data layer needs of the user stories.
Thirdly, you may sketch layers such as views on top of the schema; fourthly, and optionally, triggers and stored procedures that might live in the DB to ensure consistency and ease of use for higher layers (but, no matter how strongly DBAs will push you towards those, don't accept their assurances that they're a MUST: they aren't -- if your storage layer is well designed in terms of normalization and maybe useful views on top, non-storage-layer functionality CAN always reside elsewhere, it's an issue of convenience and performance, NOT logical consistency, completeness, correctness).
I think the business layer and user-experience layers should come after. I realize that's a controversial position, but my point is that the user stories (and implied business-rules that come with them) have ALREADY told you a LOT about the business and user layers -- so, "nailing down" (relatively speaking -- agility and "embrace change!" should always rule;-) the data storage layer is the next order of business, and refining ("drilling down") the higher layers can and should come after.
When you get to the database layer you'll want to handle the database access via stored procedures. This will help give you additional protection against SQL Injection attacks, and make it much easier to push logic changes to the database layer.
If it's mostly users interacting with data, you can design using a form perspective.
What forms are needed for user input?
What forms are needed for output reports?
Once you've determined that, the use of the forms will dictate the business logic needed to be coded behind the scenes. You'll take the inputs, create the set of procedures or methods to deal with them, and output what is necessary. Once you know the inputs and outputs, you will be able to easily design the necessary functions.
The scope of the question is very broad. You are expecting me to tell what to do. I can only do a good job of telling how to do things. Do investigate upon using Hibernate/Spring. Since most of your operations looks like querying db, hibernate should help. Make sure the tables are sufficiently indexed so your queries can run faster if filtered based on index fields. The challenging task is design your DB layer which will be the glue between your application and db. Design your db layer generic enough so that it can build queries based on the params that you pass to it. Then move on to develop the above presentation layer. Developing your application layer by layer helps since it will force you to decouple the db logic from the presentation logic. When you develop the db layer, assume that not just your presentation layer but any client can call it. This will help you to design applications that can be scalable and adaptable to new requirements.
So bottom line : Start with DB, DB integeration layer, Controller and last Presentation Layer.
For the purpose of discussion, I'm going to assume that you are working with a starting application that doesn't have a pre-existing database. If this is false, I'd probably move the order of steps around quite a bit.
1 - Understand the Universe
First, you've got to get a sense of what's around you so you can really understand the problem that you are trying to solve.
User stories or use cases are often a good starting point. Starting with what tasks the user will try to do, and evaluating how frequently they are likely to be is a great starting point. I like to start with screen mockups as well, with or without lots of hands on time with users, I find that having a screen gives our team something really finite to argue about.
What other tools exist in this sphere? These days, it seems to me that users never use just one tool, they swap around alot. You need to know two main things about the other tools you users use:
(1) - what will they be using as part of the process, along side your tool? Consider direct input/output needs - what might they want to cut/copy/paste from or to? What tools might you want to offer file upload/download for with specific formats, what tools are they using alongside your tool that you might want to share terminology, layout, color coding, icons or other GUI elements with. Focus especially on the edges of the tools - a real gotcha I hit in a recent project was emulating the databases of previous tools. It turned out that we had massive database shift, and we would likely have been better starting fresh.
(2) What (if anything) are you replacing or competing with? Steal the good stuff, dump and improve the bad stuff. Asking users is always best. If you can't at least understanding the management initiative is important - is this tool replacing a horrible legacy tool? It may be legacy, but there may be the One True Feature that has kept the tool in business all these years...
At this stage, I find that things are really mushy - there's some screen shots, some writing, some schemas or ICDs - but not a really gelled clue.
2 - Logical Entities
Or at least that's what the OO books call it.
I don't care much for all the writing I see on this task - but I find that any any given system, I have one true diagram that I draw over and over. It's usually about 3-10 boxes, and hopefully less than an exponentially large number of lines connecting them. W
The earlier you can get that diagram the better.
It doesn't matter to me if it's in UML, a database logical model, something older, or on the back of a napkin (as long as the napkin is shrouded in plastic and hung where everyone can see it).
The earlier you can make this diagram correctly, the better.
After the diagram is made, you can start working on the follow on work that may be more official.
I think it's a chicken and egg question on whether you start with your data or you start with your screens and business logic. I know that you certianly want to optimize for database sizing and searchability... but how do you know exactly what your database needs are without screens and interfaces giving you a sense for the data?
In practice, I think this is an ever-churning cycle. You do a little bit everywhere, and then you change it all.
Even if you don't get to do a formal agile lifecycle, I think you're best bet is to view design as agile -- it will take many repetitions and arguments before you really feel it's "right".
The most important thing to keep in mind is that your first, and most likely 2nd 3rd attempt at designing the database will be wrong in some way. That might sound negative, maybe even a little rash, (it's certainly more towards the 'agile' software design philosophy) but it's important thing to keep in mind.
You still need to do your analysis thoroughly of course, try to implement one feature at a time, but try to get all layers working first. That way you won't have to do to much rework when the specs change and you understand the issues better. One you have a lot of data loaded into a system, changing things becomes increasingly difficult.
The main benefit of this approach is you find out quickly where you design is broken, where you haven't separated you design layers correctly. One trick I find extremely useful is to do both a sqllite and a mysql version, so seamless switching between the two is possible. Because the two use a different accent of SQL it highlights where you have too tight a coupling between the layers.
A good start would be to get familiar with Multitier architecture
Then you design your presentation layer.
In your business logic layer implement all logic
And finally you implement your data access layer.
Try to setup a prototype with something that is more productive then C++ for example Ruby, Python and well maybe even PHP.
When the prototype works and you see your data model is okay and your queries are too slow then you can start using C++.
But as your questions suggests you have more options then data and in this case the speed of a scripting langauge should be enough.
I feel the need to refactor my old CF5 based code into CFC's. We already have some code in ColdSpring and Transfer but feel a large rewrite to ColdSpring and Transfer is pointless.
What tips, approaches and gotchas will I hit.
How can I make this easy?
I don't mind keeping ColdSpring in the mix but Transfer is the bit I'm scared of with the size of the project.
edit: my code base has been going for 7-8 years and is vast. To describe it would be difficult, however I'm looking for generic suggestions on approaches
Changing the whole code base just for the sake of it if it basically works would be introducing a lot of potential bugs into your system. I don’t think there is an easy way to do it.
If you look at the areas of your site which are 1: most likely to change and 2: executed the most you may be able to target some areas which could benefit from change and see how easily they would fit into a CFC based framework, and what benefits. But for most of the code if it is working OK, there may be no pressing need to change.
However whenever you need to do a major alteration to part of the system it may be worth looking at that from an OO perspective and moving the existing code over, where applicable.
In one of my ongoing projects (almost same situation, even more -- most of code is really bad) I am using technique I'd called "wave-style". General ideas I use are following:
Splitting processing from output. I can not implement true MVC here, but at least I can move view into separate templates (sometimes re-use them) and prepare all data in basic (model) templates.
Move all repeating code into components -- this is one of most important tips.
Group related functions into components. Say, all customer-related info grouped into CustomerManager.cfc, invoices into InvoiceManager.cfc etc.
Why "wave"? In a big project I can't just sit and rewrite all customer-related code. So I have make it step by step. For example, I have to work on customer signup, extend it with few attributes. I've created basic component, moved there methods to validate form (check login, email etc.) and add customer - so this page works in new style. Lated I will need to improve invoice page, where I need to get invoice owner details: I just add method into customer manager and get rid of direct queries. Later edit customer page... Also it can be called "on demand refactoring" or smth.
There can be additional stuff relying on your current project state. But it helped me a lot. Hope you'll find these tips useful.
Before you change anything: create a full set of regression tests!
When refactoring, the goal has to be preserve functionality first, so that you don't directly affect your clients.
I agree with Sergii's wave-style refactoring also - this allows you to break things into manageable chunks rather than doing everything in one go.
But whatever method you have, the more regression tests you can create, the better - it's really the only way you can confirm you haven't unintentionally changed something.
This is extremely hard (bordering on impossible) to answer without knowing any of your code.
The question is a bit like "I want to disassemble my old Volkswagen and build a new one from the parts, what should I consider?" :-)
My advice would be to start off by encapsulating your business logic into CFCs instead of worrying about the whole presentation layer of your site.
By just concentrating on the business logic, you'll be able to get the most important functionality into CFCs and ease the maintenance nightmare. It also won't be too hard to just "drop-in" these CFCs into your existing site.
After getting as much business logic into the CFCs as you can, you'll notice that the enormous monster has been cut down to size. At that point you can now decide on what you want to do with the presentation layer of your site. You're now free to pick from a multitude of frameworks available to use (CFWheels, FuseBox, ColdBox, Mode-Glue) to port over the presentation layer.
Or you could just say "the heck with it" and rewrite the whole thing in CFWheels from the start :)
If you are not using version control get that set up before you do anything else. Being able to back out of broken refactoring is a serious life saver. After that I agree with what has been posted. You will want to take on small chunks at a time - divide and conquer.
I have a website (ASP.NET) and some winforms(.Net 2.0) for a project (written in C#). I use the webservice (IIS6) for task that both require like sending email inside the business.
I think Webservice is nice but I would like from your experience what should and what should not be in a webservice?
In My Opinion:
Web services should be reserved for code that
You either can't or don't want to distribute; or,
code that needs to seriously scale up.
One example is custom business logic that multiple applications need access to.
Code you don't want to put into web services include:
code that is performance based;
code that applies only to the application in question.
Well it sounds like you have a limited Service Oriented Architecture (at least, that's what I think you're getting at), which according to Gartner means you'll be rich soon. :)
I find that the benefit of SOA for me really comes down to the heterogeneity of the systems involved (sounds like yours doesn't qualify there because it's all .NET), and the negative of SOA is primarily because of the verbose nature of XML. True, you don't need XML for SOA, but it's the current majority, IMHO.
But if you're not concerned about the bandwidth/parsing penalties, who cares? Maybe you're not piping through 10,000 service calls a minute. With this style of implementation, you're following DRY, just with a WS instead of a sub, and you're adhering to a standard that is by nature compatible with multiple systems.
There's worse approaches.
It seems like the new trend for web services/SOA is to more or less expose a light-weight middle tier that your host application can use. Instead of having individual method calls exposed through a service (as in your example), SOA-oriented applications have extensive Data/Operation contracts that act as the "traditional" middle tier assembly.
As little as possible, while still being useful.
By default, DON'T put every field of the return objects in the return data, and DON'T expose every method of an existing class.
read this too...