We have a (AJAX heavy) web application hosted in cloud across servers and we need to monitor the availability of this service. Requires logging in to the application with a username-password, perform some searches as that user etc.
Since we plan to use Nagios for some other monitoring tasks, we decided to use Nagios for web application monitoring too.
I came across three such solutions:
Webinject: I don't feel like using this. Project not under active development. It was last released in Jan 2006. I can't see any support/help available. Also I suspect how will it behave with Ajax.
Cucumber-Nagios:
I tried using this. It involves many Ruby components and found that you have to have in-depth knowledge of Ruby platform to make all these components work together. I am not a Ruby guy and having tough time making all these components work together. Also even this project is not under active development and I don't see support/help options available. I posted a bug 4 days back and don't see any response yet.
Selenium plugin for Nagios: Haven't tried it yet. Will try now.
Any more solutions available?
Also, since I don't see any good actively developed solutions for monitoring web applications using Nagios, I suspect if it's really a good approach to use Nagios for this? If not, what alternatives do I have? In short what is the best approach to monitor web applications availability?
Edit 1: We can't afford the Nagios XI paid version and will prefer open source solutions.
If not, what alternatives do I have?
Although Nagios was one of options that we've considered, we've chosen OpenNMS for monitoring purposes. Rationale for our decision is that OpenNMS is highly reliable and configurable free open-source tool and additionally, most of our applications are Java-based; OpenNMS offers integration with JMX. However, bear in mind that if you're demanding very complex tests for your Web site maybe it's better to look elsewhere. OpenNMS can be set to check for HTTP status codes etc., but if you're looking for complex scenarios take a look at:
Apache JMeter (we're using it mainly during the testing phase)
Selenium (can be well used even in production phase)
Related
I have created a django app that contains c++ for some of the views as well as a java library. How would I deploy this app? What kind of hosting service allows for multiple languages? I have looked at EC2, GAE, and several platforms (like heroku) but I can't seem to find a definitive solution.
I have never deployed anything to the web so a simple explanation would be much appreciated.
PaaS stuff is probably not your best bet. If you want the scalability and associated buzzwords(muh 99.9999999999% availability because my servers are hosted in a parallel dimension without electrical storms, power outages, hurricanes, earthquakes, or nuclear holocausts) that comes with hosting your application on a huge web company's platform, check out IaaS(Infrastructure as a service) systems like Google's Compute Engine or AWS. With these you just get a virtual server (or servers), running your Linux distro of choice, and you can install and run whatever you please on them without being constrained to a specific platform like App Engine or Heroku(where you have to basically write your app to specifically run on that platform). If you plan on consuming a ton of bandwidth/resources from the get-go, you will almost certainly get a better deal using a dedicated server(s) from a small company.
Interested in what specifically you are executing C++ for in a Django view. Image/video processing?
Well. Deployment is not really something where a simple explanation helps much.
First I would check what the requirements to the operating system are (compilers, dependencies,…). That will maybe reduce the options quickly.
I guess that with a setup containing C++ & Java artifacts, the usual PaaS (GaE, Heroku,…) offerings will not be sufficient because they define the stack. And a mixture of Python/C++/Java is rather uncommon I'd say.
Choosing an IaaS offering (EC2, …) may be an option. There you can run your whole self-defined stack and have the possibility of easier scaling.
Hosting the application on your own server(s) is also always possible. Check your data protection regulations to find out if it's not even a requirement.
There are a lot of ways to get the Django application to run. The Django documentation has some information about deployment. If you have certain special requirements, uwsgi may be a good application server.
You may also want a web server in front of the application. Possibilities range from using uwsgi's built-in http server or using e.g. Nginx with uwsgi.
All in all every component of the whole "deployment" has hundereds of bells and whistels and it's not easy to give advice without knowing specific requirements and properties of the system itself. You'll also probably need a database you have to deploy.
But before deploying it to the web, it's also important to have a solid build process to assemble all the parts. And not only on the development machine. With three languages involved this should be the first step solve. If it easily and automagically deploys in a development environment, moving it to a server is easier.
I'm currently working with a team of developers on a company project to create a centralized repository of product and pricing information. This will be built for both internal company use and external client use. On top of the basic features of storing product and pricing information we also need to build up an infrastructure to accommodate:
REST API endpoints
Dev/Staging/Deployment workflows (particularly for performing updates on records in a live environment)
Logging
Analytics
Reporting
Security (authentication and authorization).
Going over the list, it reads like a very common set of requirements for a web application and I doubt my company is breaking any sort of new ground. SO, is there any particular resources (frameworks, technology stacks, articles, books) that can help me understand how other web applications are tackling these problems?
A bit of background on the team. The team has worked on a handful of small-to-medium sized web applications using PHP, Mongo and MySQL for the backend, and basic HTML, CSS, JQuery on the frontend. The team is familiar with design patterns (i.e.Gang of Four) but to date have not worked on anything requiring all of the elements listed above
It's probably worth playing around with a solid Web development framework like Zend, Yii or even Ruby on Rails or Django, which are not PHP frameworks, but are fairly mature and well structured. Even if you do not plan to use that framework for development you'll get some great ideas for how to structure your web applications, how to implement logging and common web security features.
As far as deployment and workflows go, you may want to give Extreme Programming a read if you haven't already. It describes what many developers today considered to be a fairly classic agile project management methodology, but it also gets into important practices such as testing and continuous integration, which in my opinion are incredibly important components of the development workflow. If you're starting from scratch as a team you'll benefit enormously from implementing a solid agile methodology -- or at the very least from a solid foundation in testing and continuous integration.
For examples of REST style applications you may want to see how popular open source implementations work. Some of these frameworks will have a REST structure built in, but there are many open source options, some of which are discussed here.
For analytics, Google has quite a bit of documentation here.
As far as reporting goes, I'm not clear on what you need, but if you're talking about log parsers and bug or downtime reporters there are some excellent tools out there, including continuous integration automation tools such as Atlassian's Bamboo that will provide some reporting assistance. These can help you with part of the reporting process, but from my experience a large, complex web application can benefit from custom reporting elements, considered as part of the development process from the beginning. It's not that difficult to parse logs programmatically, I don't think there's a one size fits all implementation.
As a side note, Atlassian has some excellent development tools if you're willing to pay for them, but open source alternatives shouldn't be difficult to find, such as the ubiquitous Trac for ticket tracking, and basic project management with an integrated wiki.
I can't say I know of a single, comprehensive location that provides you all the information you need (at least not yet!), but hopefully you'll glean something interesting from this answer. Starting on some serious web development projects with a fresh team (if I interpreted your situation correctly) can be a really enjoyable challenge. Good luck!
Sorry for the very involved question, but this is something I've been researching for a while now and it is really frustrating me. I feel like in today's age we have a million and one ways to implement services tat are cross-platform (SOAP) and easy to build (thanks to .NET, java, and other frameworks). However, these technologies have been in the community for 5-10 years, but we are (or at least I am) constantly plagued with the same issues:
Identification (Tracking services) - UDDI; e.g., had to remind a co-worker the 3 times this month where a service is at, despite the fact there is a wiki that discusses the service and a PDF version of the same documentation that lives in a repository where we keep our service docs.
Scalability - Out of the box clustering; As organizations, we spend a lot of money on paying our admins just to watch the utilization of our services and make decisions like, does this service need more RAM, more CPU, more interfaces? How do I load balance this?
Monitoring - error logging, etc; I can't count how many times I have to set up tracing on services in order to see why a bug is happening that only seems to affect one customer, or have to code logic into the service to serialize exceptions, log exceptions to dbs, fail gracefully, etc.
Deployment - easy to deploy; none of this deploying DLLs to 5 load balanced servers
Each one of these problems requires some type of custom solution implemented by the organization. Documentation and UDDIs for #1. Virtualization and load balancing hardware / software for #2. Tracing, writing exceptions to databases / logs, etc for #3. Custom deployment software for #4. I work for a mid-sized organization. I can't even imagine how a company the size of Sun, Google, or Microsoft would tackle these dilemmas.
Maybe my vision is unrealistic, but I dream of having a Framework per se that lives on top of a server cluster that manages all of the above. I was ecstatic to read about Microsoft's AppFabric since it really seems to extend some of the functionality of BizTalk to WCF service implementors: Caching, Hosting, Monitoring, etc. However, from what I've seen, I still don't feel it lives up to my dream for an all-in-one solution that assists the developer and organization in writing services that are scaled across clusters easily, deployed into the cluster easily, and identifiable, possibly even version-able.
So, I don't mean this post to be about my dream. I do actually have a question. For starters, is my dream / want completely unrealistic? Furthermore, what solutions are there available that attempt to solve these problems without confining us to a new and more proprietary way (BizTalk) of developing services? An lastly, in concern to a complete SOA / ESB solution, where do we see the most potential in the market right now or in the future?
I think that you are talking about different kinds of problems here.
1). Developers who don't read documentation. This is an endemic problem, not limited to SOA - just look at some questions on StackOverflow. At least the developer is asking you whether there is a service, rather then just duplicating logic in their own code. I don't see any technical solution to these kinds of problems, you've already provided good registries and documentation, but some developers prefer to talk to people. Maybe, even, this is actually a good thing - human interaction has value above the technical content of the interaction. Or maybe, you're too nice: "No, I won't answer that question, look it up."
2). Scaling. There are technologies addressing this issue. (Disclaimer I work for IBM, who sell some, so I'll reference these - I'm not intending to imply that IBM are the only vendor with solutions in this space.) There are products such as this that can provision a new machine, install a software stack and add it to a cluster to address workload changes. Then at a finer grained level of control in the Java EE world the Application Server can dynamically shape traffic and adjust clusters. See WebSphere Virtual Enterprise
3). Monitoring. I don't "get" what you expect here. In all likelyhood such tricky bugs will require application level trace. For some problems such as finding memory leaks and performance bottlenecks there are very good tools, at least in the Java EE world.
4). I can't speak to the .Net world, but I'd say that Java EE app servers do a reasonable job of deploying the apps across clusters smoothly, and in the cases where we use JNI and need DLLs deploying then we can use products such as the Tivoli stack I mention to manage this.
So, in summary, I do think that vendors are trying to address these issues. And I don't think your life would be simpler without SOA. Imagine instead the same problems applied to myriad separate, independent applications.
Here's my two cents.
I've been a developer at a company that used SOA incorrectly. The worst solution they implemented was field level validation of form elements on a desktop app using SOA. To perform acceptably these require very low latency. A 2-4 second wait to change to a new field gets old fast. The service ran over the network on a biztalk server. Everyone hated it.
If you're going to do this you really need to spend a lot of time dealing with network latency, service failure, timing, and timeout issues.
Don't get carried away and think SOA is the solution to every problem. Used at a high level it's great, used at a low level it makes your applications fragile, slow, and impossible to debug.
If you talk to IBM or one of the big SOA vendors, they got a products that cover each scenario.
Identification (Tracking services) - UDDI; e.g., had to remind a co-worker the 3 times this month where a service is at, despite the fact there is a wiki that discusses the service and a PDF version of the same documentation that lives in a repository where we keep our service docs.
Registry and Repository server. Nice thing is that it does governance (promotion, demotion, versioning, approval) and your ESB typically does a "lookup" for the latest and greatest against the register server.
Scalability - Out of the box clustering; As organizations, we spend a lot of money on paying our admins just to watch the utilization of our services and make decisions like, does this service need more RAM, more CPU, more interfaces? How do I load balance this?
Transaction monitoring software like IBM Tivoli Composite Application Manager for SOA. Basically, it tracks things from a horizontal point of view and to see if there is a service disruption from a end user/end app point of view.
As far as your clustering.... you have to pick good middleware and architecture. Personally speaking, get stuff that is "cloud" ready. App Servers with NoSQL connected by MOM.
Monitoring - error logging, etc; I can't count how many times I have to set up tracing on services in order to see why a bug is happening that only seems to affect one customer, or have to code logic into the service to serialize exceptions, log exceptions to dbs, fail gracefully, etc.
Enterprise standards for your developers and for your vendors. Integration of all business and system events into a single dashboard. (Most companies spilt them). This is done already at most enterprise shops.
Deployment - easy to deploy; none of this deploying DLLs to 5 load balanced servers
Ahh.. Microsoft IIS Web Deployment Tool 2.0. You can sync 100s of MS servers by just updating the master. It's really easy.
I've been looking at Rietveld as a solution for the lack of code reviews at my company. Can it be set up on a server in-house without using App Engine? It seems to have a bit of App Engine specific code, and I'm not sure it could be set up on a plain old Django/Apache install. I've looked around, but haven't found any information about this.
Check out http://django-gae2django.googlecode.com/svn/trunk/examples/rietveld/README
The gae2django project lets GAE apps run against django instead of the GAE development environment.
That means you can run rietveld under django directly, using (by default) an SQLite backend. You can also use mysql or any other DB backend django supports.
That, plus a web server (e.g. Apache) with WSGI integration, makes a local rietveld install run nicely.
What about using one of these projects that provide the same backend services as GAE?
Typhoon AE
Appscale
There may be more, these are just the ones I know about off the top of my head.
A bit of App Engine specific code? It's supposed to be an example App Engine app, so yeah it's pretty well tied to it. But, you're right, it does use Django which could make it somewhat more feasible to port. I'll second #cope360 recommendation, but from the sounds of your question, it doesn't sound like you've done much with App Engine. If it's only used by a few people, try running it on the GAE SDK itself.
Beyond that, I'd think you could take most of the code in the "codereview" directory and build you're own Django/apache app from that.
Rather than fussing around with a port or other GAE emulation, I would consider using ReviewBoard.
Review Board is a powerful web-based
code review tool that offers
developers an easy way to handle code
reviews. It scales well from small
projects to large companies and offers
a variety of tools to take much of the
stress and time out of the code review
process.
For too long, code reviews have been
too much of a chore. This is largely
due to the lack of quality tools
available, leaving developers to
resort to e-mail and bug tracker-based
solutions.
We've seen a lot of time and energy
wasted doing code reviews both in open
source projects and at companies. In
both cases, code reviews were
typically done over e-mail. A
significant amount of time was spent
in forming review requests, switching
between the diff and the e-mail, and
trying to understand what parts of the
code the reviewer was referring to.
So in an effort to keep our sanity and
improve the process both in our open
source projects and at companies, we
wrote Review Board. We hope it will be
useful to your team too so you can
focus on what's important: writing
great products.
trying to pick version control, continuous integration, and host for Flex + Ruby or Django smallish project. Question:
version control: I've used SVN and CVS in the past. I hear great things about git. Not sure what to pick.
continuous integration: I've heard good things about hudson and cruiseControl. Not sure what to pick
hosting: is my own server the only way to go? Are the decent cloud options that are not too expensive? or should I look for some free hosting service?
thank you for your help!
f
Use Git.
Git is a great tool that allows a very flexible workflow. It has lots of benefits over subversion/cvs, the biggest of which is the ability to branch and merge seamlessly. This can't be overstated. The merge-hell that ensues when attempting to use svn's branching and merging is a thing of the past. For a better case on why to use git, check out http://whygitisbetterthanx.com/
Use Hudson.
Hudson is the easily the best CI tool in the game. The reason Hudson is the best is that its easy to configure (for one or multiple nodes), it has a ton of plugins, and handles the 90% use case extremely well. You are in the 90% use case. People like Mozilla aren't. Check out C. Titus Brown's talk at Pycon for more info. http://pycon.blip.tv/file/3259794/ (If you decide that Hudson isn't what you should use, check out buildbot)
Use Webfaction (or Rackspace Cloud).
Webfaction is a great starter ground. If your needs are low, check them out. Beyond that, I'd suggest taking a hard look at Rackspace Cloud (RSC). RSC makes scaling out much easier and their pricing model is very palatable for things that aren't bandwidth intensive (ie: most things that don't require tons of uploads/downloads). It starts at $10/mo. Their management console is good (save the DNS administration interface, but even that is more than bearable). If your needs expand beyond RSC (doubtful), you would do well to check out Amazon's EC2. Companies like RightScale can help when it comes to scaling out.