I'm currently working with a team of developers on a company project to create a centralized repository of product and pricing information. This will be built for both internal company use and external client use. On top of the basic features of storing product and pricing information we also need to build up an infrastructure to accommodate:
REST API endpoints
Dev/Staging/Deployment workflows (particularly for performing updates on records in a live environment)
Logging
Analytics
Reporting
Security (authentication and authorization).
Going over the list, it reads like a very common set of requirements for a web application and I doubt my company is breaking any sort of new ground. SO, is there any particular resources (frameworks, technology stacks, articles, books) that can help me understand how other web applications are tackling these problems?
A bit of background on the team. The team has worked on a handful of small-to-medium sized web applications using PHP, Mongo and MySQL for the backend, and basic HTML, CSS, JQuery on the frontend. The team is familiar with design patterns (i.e.Gang of Four) but to date have not worked on anything requiring all of the elements listed above
It's probably worth playing around with a solid Web development framework like Zend, Yii or even Ruby on Rails or Django, which are not PHP frameworks, but are fairly mature and well structured. Even if you do not plan to use that framework for development you'll get some great ideas for how to structure your web applications, how to implement logging and common web security features.
As far as deployment and workflows go, you may want to give Extreme Programming a read if you haven't already. It describes what many developers today considered to be a fairly classic agile project management methodology, but it also gets into important practices such as testing and continuous integration, which in my opinion are incredibly important components of the development workflow. If you're starting from scratch as a team you'll benefit enormously from implementing a solid agile methodology -- or at the very least from a solid foundation in testing and continuous integration.
For examples of REST style applications you may want to see how popular open source implementations work. Some of these frameworks will have a REST structure built in, but there are many open source options, some of which are discussed here.
For analytics, Google has quite a bit of documentation here.
As far as reporting goes, I'm not clear on what you need, but if you're talking about log parsers and bug or downtime reporters there are some excellent tools out there, including continuous integration automation tools such as Atlassian's Bamboo that will provide some reporting assistance. These can help you with part of the reporting process, but from my experience a large, complex web application can benefit from custom reporting elements, considered as part of the development process from the beginning. It's not that difficult to parse logs programmatically, I don't think there's a one size fits all implementation.
As a side note, Atlassian has some excellent development tools if you're willing to pay for them, but open source alternatives shouldn't be difficult to find, such as the ubiquitous Trac for ticket tracking, and basic project management with an integrated wiki.
I can't say I know of a single, comprehensive location that provides you all the information you need (at least not yet!), but hopefully you'll glean something interesting from this answer. Starting on some serious web development projects with a fresh team (if I interpreted your situation correctly) can be a really enjoyable challenge. Good luck!
Related
We currently have a monolithic web application built with Scala (scalatra for the Rest APIs) for the backend and AngularJS for the front end. The application is deployed at AWS. We are going to build a new component, which we would like to build it as an independent microservice. And this component will have its own data repository which may not be the same type of DB. It will also be built with Scala as well, but Akka for the Rest APIs. The current application is built with DB module, domain module, and web service API module and front end/client module.
What is a good approach of a smooth journey? We possibly need to set up a micro service architecture first, such as an API gateway service along with others.
Too many ways, too many approaches, too many best practices. It really all depends on the analysis of your application, trying to figure out where the natural breaks are.
One place I start is looking at the data model. Lots of people advocate each microservice having its own database. Well, that's fine and dandy, but that can really be difficult to achieve without breaking things all over the place. But if you get lucky and there's a place where the data segregates nicely, than see what services would go with it and try breaking it out.
If you do not adhere to the separate database mentality, then I start with the low-hanging fruit, often times nothing more than simple CRUD operations with just a little business logic mixed in, providing some of the basic support for other larger-grained services to come. Of course, this becomes more iterative, not sure your organization will like it.
Which brings me to methodology. Organizations who've created monolithic applications often have methodologies that support them, whereas microservices require a much different approach to application development. Is your organization ready for that?
Needless to say, there's no right answer. I've gone to many conferences where these concepts are high on the interest list and the fact is there's no silver bullet, everyone has different ideas of what is right, and there's exceptions galore. You're just going to have to bite the bullet and cross your fingers, unfortunately.
We have a (AJAX heavy) web application hosted in cloud across servers and we need to monitor the availability of this service. Requires logging in to the application with a username-password, perform some searches as that user etc.
Since we plan to use Nagios for some other monitoring tasks, we decided to use Nagios for web application monitoring too.
I came across three such solutions:
Webinject: I don't feel like using this. Project not under active development. It was last released in Jan 2006. I can't see any support/help available. Also I suspect how will it behave with Ajax.
Cucumber-Nagios:
I tried using this. It involves many Ruby components and found that you have to have in-depth knowledge of Ruby platform to make all these components work together. I am not a Ruby guy and having tough time making all these components work together. Also even this project is not under active development and I don't see support/help options available. I posted a bug 4 days back and don't see any response yet.
Selenium plugin for Nagios: Haven't tried it yet. Will try now.
Any more solutions available?
Also, since I don't see any good actively developed solutions for monitoring web applications using Nagios, I suspect if it's really a good approach to use Nagios for this? If not, what alternatives do I have? In short what is the best approach to monitor web applications availability?
Edit 1: We can't afford the Nagios XI paid version and will prefer open source solutions.
If not, what alternatives do I have?
Although Nagios was one of options that we've considered, we've chosen OpenNMS for monitoring purposes. Rationale for our decision is that OpenNMS is highly reliable and configurable free open-source tool and additionally, most of our applications are Java-based; OpenNMS offers integration with JMX. However, bear in mind that if you're demanding very complex tests for your Web site maybe it's better to look elsewhere. OpenNMS can be set to check for HTTP status codes etc., but if you're looking for complex scenarios take a look at:
Apache JMeter (we're using it mainly during the testing phase)
Selenium (can be well used even in production phase)
For personal and university research reasons I am thinking of building a simple CRM using a service oriented architecture. Its meaning is just to explain the architecture itself, not commercial use.
I was thinking of implementing a CRM that offers a simple analytics service and customer care (user storing, personal comments, and few other things).
The architecture that I'm designing defines:
- WebGUI (a client of the other services)
- AnalyticsService (a service that receives data, analyzes and collect it)
- CustomerCareService (a service that uses RESTful APIs to apply CRUD operations).
Each service has it own database, being completely independent from others. They expose a public interface. The interface of course must provide some sort of authentication, to deny unautorized requests.
The advantages I'd like to explain in this kind of architecture is the possibility to have all things indepentent and the ability to combine them to offer new services (for example if there was an OrderService to handle orders it would be easy to combine it with Customer using the public APIs). The big advantage to me is that it'd be easy enough to build other clients that use these services.
I don't know what is some good Authentication method, that could be easy to implement, I'm also not sure about how to make this APIs (use XML or plain REST APIs with GET/POST data). I've worked with Amazon, PayPal and other company APIs, they seem to use REST services (paypal uses an ugly _cmd GET parameter while Amazon uses better URI) to know what to do, but reading something about SOAs it appears that people also use XML. Of course I also need to take into account that the web interface must be able to recognize the logged in user, get the permissions (token or whatever else) and use it with services to show information.
So I'm not sure SOA is the kind of architecture I'm really building up... is it SaaS instead of SOA?
I think it would be better to use RESTful applications, with JSON or something like that to implement it (I'm not a big fan of XML, I find it to be too verbose).
For clarity I'm listing here my questions:
Is this kind of architecture called SOA or SaaS (or both)?
What is a good implementation for what I want to obtain? (please explain it as more detailed as possible)
What sort of authentication is more suitable for a client (user token vs OAuth or similiar)
Do you have some suggestion for this kind of project?
I've about 3 months to do it, so I cannot do something real complex (beside the fact that it would not be realistic for a single programmer).
I know Python (WSGI frameworks), Ruby on Rails, C/C++ and other languages (.net excluded) and I'd like to develop it under a Linux environment (MySQL or Postgres, or even a NoSQL if you have any suggestion for the right choice), I could also combine several languages being these services independent programs.
What I'd like here is to have some good point of view and some good suggestion.
Thanks!
I would define SaaS as a Business model rather than an architecture; however like all business domain requirements it will influence systems architecture but it, itself is not. What you have defined is essential a Service Oriented Architecture.
Your statement "independent and the ability to combine them to offer new services" is the essential non-functional design requirement that suggests SOA.
Good implementation for SOA is about having well defined and flexible interfaces, with very clear delineation of responsibilities. However it is difficult subject to be prescriptive about. The proof is in the eating; does it provide that flexible reuse. My suggestion is spend time reading SOA design pattern resources, and understand the defining characteristics with regard to the appropriate context for use. Then apply the Single Responsibility principle appropriate level of abstraction. c.f. (Domain) Space Based Architecture is kind of SOA meta-pattern.
In regard to Authorisation, I recommend following the service approach, use a distribute directory services system like open LDAP, and note that is entirely reasonable for service provides and users to have their own credentials and you can use Public-Private keys for signing messages.
The main suggestion is study and learn from experience of others:
http://www.soapatterns.org/
http://martinfowler.com/eaaCatalog/
SOA doesn't forces to use XML.
Currently web technologies dominate, and define future.
So we in my company selected JSON RESTful services as foundation. And SOA as principles.
There is no sense to suggest languages, because the purpose of SOA and good implementation is
- to enable any language or framework to be used
(FYI we use Java with Spring MVC-based web-services, Node.js, PHP)
I've got an intermediate amount of experience programming in Java and a fairly advanced amount of experience programming web applications in other languages. I'm currently in the process of learning some of the Java web frameworks (Spring MVC, Struts.)
Where I work, we have a number of individual Java web applications, deployed in separate contexts and as separate wars. A few of the oldest are servlets, many are Struts 1, and the new ones will be Spring 3.
Most of the existing applications have inconsistent or no branding (graphics/decoration), and one of the things I am looking into is applying a consistent header and footer across multiple sites that we can maintain in one place. Ideally it wouldn't even have to be on the same server as the web applications. To be clear-- whereever I configure my template, I want to be able to configure http://some.other.server/template instead of WEB-INF/template.
I would like to use SiteMesh or something with the same "decorating" approach but I researched it, and apparently the only way this has been done in SiteMesh 2 was done by someone who forked the code and customized it to allow remote templates to be imported. (I found this information on the Sitemesh3 Google group.)
I'm at the beginning of designing this solution so I suppose we could use anything like Velocity, Freemarker, Tiles. However, where I work we are slow to update old apps and conservative in adopting new technologies, so I have to prove a solution has good community support, works with Struts 1 and Spring 3 MVC, and is fairly trustworthy and stable.
Velocity has a concept of "resource loader", i.e. an interface for finding template files which can be configured. There is already a loader for URL-based resources. See the Velocity Documentation for Resource Loaders. It also has basic support for caching.
If you need something a bit more sophisticated it should also be fairly easy to provide your own implementation of a resource loader, I did that for one of my projects and it works fine.
I've been looking at Rietveld as a solution for the lack of code reviews at my company. Can it be set up on a server in-house without using App Engine? It seems to have a bit of App Engine specific code, and I'm not sure it could be set up on a plain old Django/Apache install. I've looked around, but haven't found any information about this.
Check out http://django-gae2django.googlecode.com/svn/trunk/examples/rietveld/README
The gae2django project lets GAE apps run against django instead of the GAE development environment.
That means you can run rietveld under django directly, using (by default) an SQLite backend. You can also use mysql or any other DB backend django supports.
That, plus a web server (e.g. Apache) with WSGI integration, makes a local rietveld install run nicely.
What about using one of these projects that provide the same backend services as GAE?
Typhoon AE
Appscale
There may be more, these are just the ones I know about off the top of my head.
A bit of App Engine specific code? It's supposed to be an example App Engine app, so yeah it's pretty well tied to it. But, you're right, it does use Django which could make it somewhat more feasible to port. I'll second #cope360 recommendation, but from the sounds of your question, it doesn't sound like you've done much with App Engine. If it's only used by a few people, try running it on the GAE SDK itself.
Beyond that, I'd think you could take most of the code in the "codereview" directory and build you're own Django/apache app from that.
Rather than fussing around with a port or other GAE emulation, I would consider using ReviewBoard.
Review Board is a powerful web-based
code review tool that offers
developers an easy way to handle code
reviews. It scales well from small
projects to large companies and offers
a variety of tools to take much of the
stress and time out of the code review
process.
For too long, code reviews have been
too much of a chore. This is largely
due to the lack of quality tools
available, leaving developers to
resort to e-mail and bug tracker-based
solutions.
We've seen a lot of time and energy
wasted doing code reviews both in open
source projects and at companies. In
both cases, code reviews were
typically done over e-mail. A
significant amount of time was spent
in forming review requests, switching
between the diff and the e-mail, and
trying to understand what parts of the
code the reviewer was referring to.
So in an effort to keep our sanity and
improve the process both in our open
source projects and at companies, we
wrote Review Board. We hope it will be
useful to your team too so you can
focus on what's important: writing
great products.