I'm the only programmer of a pretty small ISP in a rural area with just around 2000 customers. Now I have finished a couple of semesters in university but I only have a couple of years of experience in the field so I'm uncertain of the architectural decisions that I'm making and was hoping somebody could help me pick the right path.
Most of our internal apps were created 8-10 years ago and are severely outdated and I have been given the job to replace those systems. Most of the basic underlying systems are solid but the apps that we use to manage our customers and connecting those to our internal systems are...lacking to say the least.
Most of these applications were created in PHP back in the day and are using mysql databases. I decided that i was going to create a couple of rest APIs using NodeJS on top of these databases and then create a central app that will take care of connecting all those systems together and making sure they stay up to date with one another.
Now for the question. I've been looking a bit into enterprise architecture and from what I've gathered going with this sort of micro service architecture seems to be a solid plan. However I've also seen a couple articles talking about message buses and my question is if i should instead set up a message bus, for example apache activemq so these services can talk together amongst themselves instead of using a central app that would handle managing all of them.
Are there any specific patterns that i should be reading up on or does what I've come up with look solid enough?
An enterprise service bus will add a lot of complexity to your design, so you need to look at the pro/con to see if it's really necessary. Here is an article you can always upgrade your architecture in the future and migrate the services.
I run some complex services on Apache Tomcat and they work great. Supports a user pool of 70,000. If you build in connection pooling and redundancy you should be fine.
Related
I would be starting ft in one company, where i was been told that the application is developed using 'Sas' and 'salesforce'. What is the difference between two?
And which are recommended online resource which I can use to learn more about it.
SAS is software for statistical analysis. If your company/job description doesn't look like working with large sets of data & complex reporting that's probably not it.
They probably mean SaaS (Software as a Service) model, also known as "the cloud", cloud computing etc. You write the program (or use / modify existing one) but you don't buy servers, worry about network connection, electricity costs, load balancing (spikes in traffic will not cause your website to go down). Many apps operate in this model. Microsoft's Azure cloud (or even online wersions of MS Office). There's Siebel Oracle on Demand CRM, Microsoft Dynamics, SAP I think also has SaaS offering...
It's a big topic, I'm simplifying a lot here. And then there are Platform as a Service things too (PaaS) where they give you "just" the hosting etc but no base application to build on top of. You write everything you need from scratch and upload it. Think Heroku or Amazon Web Services (AWS).
Salesforce is "just" one more SaaS application. You start with base application & database, similar to all other clients in the world. You can install plugins to it (some free, some paid), configure it yourself, write custom code if your functionality is too complex... You can do a lot with just clicks & drag & drop but if you need to code stuff then JavaScript (for client-side) and Apex (for server-side) will be your friend. Apex is bit similar to Java.
Where to start... Trailhead is good source of self-paced trainings. You can sign up for a free Salesforce Developer Edition (has almost all features as the paid one but limited storage space), try to pass some courses... Or in SF help&training there should be tons of videos (actually in that link whole left menu "getting started with salesforce" might be good).
We currently have a monolithic web application built with Scala (scalatra for the Rest APIs) for the backend and AngularJS for the front end. The application is deployed at AWS. We are going to build a new component, which we would like to build it as an independent microservice. And this component will have its own data repository which may not be the same type of DB. It will also be built with Scala as well, but Akka for the Rest APIs. The current application is built with DB module, domain module, and web service API module and front end/client module.
What is a good approach of a smooth journey? We possibly need to set up a micro service architecture first, such as an API gateway service along with others.
Too many ways, too many approaches, too many best practices. It really all depends on the analysis of your application, trying to figure out where the natural breaks are.
One place I start is looking at the data model. Lots of people advocate each microservice having its own database. Well, that's fine and dandy, but that can really be difficult to achieve without breaking things all over the place. But if you get lucky and there's a place where the data segregates nicely, than see what services would go with it and try breaking it out.
If you do not adhere to the separate database mentality, then I start with the low-hanging fruit, often times nothing more than simple CRUD operations with just a little business logic mixed in, providing some of the basic support for other larger-grained services to come. Of course, this becomes more iterative, not sure your organization will like it.
Which brings me to methodology. Organizations who've created monolithic applications often have methodologies that support them, whereas microservices require a much different approach to application development. Is your organization ready for that?
Needless to say, there's no right answer. I've gone to many conferences where these concepts are high on the interest list and the fact is there's no silver bullet, everyone has different ideas of what is right, and there's exceptions galore. You're just going to have to bite the bullet and cross your fingers, unfortunately.
I have a web app running on php, mysql, apache on a virtual windows server. I want to redesign it so it is scalable (for fun so I can learn new things) on AWS.
I can see how to setup an EC2 and dump it all in there but I want to make it scalable and take advantage of all the cool features on AWS.
I've tried googling but just can't find a simple guide (note - I have no command line experience of Linux)
Can anyone direct me to detailed resources that can lead me through the steps and teach me? Or alternatively, summarise the steps in an answer so I can research based on what you say.
Thanks
AWS is growing and changing all the time, so there aren't a lot of books to help. Amazon offers training that's excellent. I took their three day class on Architecting with AWS that seems to be just what you're looking for.
Of course, not everyone can afford to spend the travel time and money to attend a class. The AWS re:Invent conference in November 2012 had a lot of sessions related to what you want, and most (maybe all) of the sessions have videos available online for free. Building Web Scale Applications With AWS is probably relevant (slides and video available), as is Dissecting an Internet-Scale Application (slides and video available).
A great way to understand these options better is by fiddling with your existing application on AWS. It will be easy to just move it to an EC2 instance in AWS, then start taking more advantage of what's available. The first thing I'd do is get rid of the MySql server on your own machine and use one offered with RDS. Once that's stable, create one or more read replicas in RDS, and change your application to read from them for most operations, reading from the main (writable) database only when you need completely current results.
Does your application keep any data on the web server, other than in the database? If so, get rid of all local storage by moving that data off the EC2 instance. Some of it might go to the database, some (like big files) might be suitable for S3. DynamoDB is a good place for things like session data.
All of the above reduces the load on the web server to just your application code, which helps with scalability. And now that you keep no state on the web server, you can use ELB and Auto-scaling to automatically run multiple web servers (and even automatically launch more as needed) to handle greater load.
Does the application have any long running, intensive operations that you now perform on demand from a web request? Consider not performing the operation when asked, but instead queueing the request using SQS, and just telling the user you'll get to it. Now have long running processes (or cron jobs or scheduled tasks) check the queue regularly, run the requested operation, and email the result (using SES) back to the user. To really scale up, you can move those jobs off your web server to dedicated machines, and again use auto-scaling if needed.
Do you need bigger machines, or perhaps can live with smaller ones? CloudWatch metrics can show you how much IO, memory, and CPU are used over time. You can use provisioned IOPS with EC2 or RDS instances to improve performance (at a cost) as needed, and use difference size instances for more memory or CPU.
All this AWS setup and configuration can be done with the AWS web console, or command-line tools, or SDKs available in many languages (Python's boto library is great). After learning the basics, look into CloudFormation to automate it better (I've written a couple of posts about that so far).
That's a bit of the 10,000 foot high view of one approach. You'll need to discover the details of each AWS service when you try to use them. AWS has good documentation about all of them.
Depending on how you look at it, this is more of a comment than it is an answer, but it was too long to write as a comment.
What you're asking for really can't be answered on SO--it's a huge, complex question. You're basically asking is "How to I design a highly-scalable, durable application that can be deployed on a cloud-based platform?" The answer depends largely on:
The specifics of your application--what does it do and how does it work?
Your tolerance for downtime balanced against your budget
Your present development and deployment workflow
The resources/skill sets you have on-staff to support the application
What your launch time frame looks like.
I run a software consulting company that specializes in consulting on Amazon Web Services architecture. About 80% of our business is investigating and answering these questions for our clients. It's a multi-week long project each time.
However, to get you pointed in the right direction, I'd recommend that you look at Elastic Beanstalk. It's a PaaS-like service that abstracts away the underlying AWS resources, making AWS easier to use for developers who don't have a lot of sysadmin experience. Think of it as "training wheels" for designing an autoscaling application on AWS.
Sorry for the very involved question, but this is something I've been researching for a while now and it is really frustrating me. I feel like in today's age we have a million and one ways to implement services tat are cross-platform (SOAP) and easy to build (thanks to .NET, java, and other frameworks). However, these technologies have been in the community for 5-10 years, but we are (or at least I am) constantly plagued with the same issues:
Identification (Tracking services) - UDDI; e.g., had to remind a co-worker the 3 times this month where a service is at, despite the fact there is a wiki that discusses the service and a PDF version of the same documentation that lives in a repository where we keep our service docs.
Scalability - Out of the box clustering; As organizations, we spend a lot of money on paying our admins just to watch the utilization of our services and make decisions like, does this service need more RAM, more CPU, more interfaces? How do I load balance this?
Monitoring - error logging, etc; I can't count how many times I have to set up tracing on services in order to see why a bug is happening that only seems to affect one customer, or have to code logic into the service to serialize exceptions, log exceptions to dbs, fail gracefully, etc.
Deployment - easy to deploy; none of this deploying DLLs to 5 load balanced servers
Each one of these problems requires some type of custom solution implemented by the organization. Documentation and UDDIs for #1. Virtualization and load balancing hardware / software for #2. Tracing, writing exceptions to databases / logs, etc for #3. Custom deployment software for #4. I work for a mid-sized organization. I can't even imagine how a company the size of Sun, Google, or Microsoft would tackle these dilemmas.
Maybe my vision is unrealistic, but I dream of having a Framework per se that lives on top of a server cluster that manages all of the above. I was ecstatic to read about Microsoft's AppFabric since it really seems to extend some of the functionality of BizTalk to WCF service implementors: Caching, Hosting, Monitoring, etc. However, from what I've seen, I still don't feel it lives up to my dream for an all-in-one solution that assists the developer and organization in writing services that are scaled across clusters easily, deployed into the cluster easily, and identifiable, possibly even version-able.
So, I don't mean this post to be about my dream. I do actually have a question. For starters, is my dream / want completely unrealistic? Furthermore, what solutions are there available that attempt to solve these problems without confining us to a new and more proprietary way (BizTalk) of developing services? An lastly, in concern to a complete SOA / ESB solution, where do we see the most potential in the market right now or in the future?
I think that you are talking about different kinds of problems here.
1). Developers who don't read documentation. This is an endemic problem, not limited to SOA - just look at some questions on StackOverflow. At least the developer is asking you whether there is a service, rather then just duplicating logic in their own code. I don't see any technical solution to these kinds of problems, you've already provided good registries and documentation, but some developers prefer to talk to people. Maybe, even, this is actually a good thing - human interaction has value above the technical content of the interaction. Or maybe, you're too nice: "No, I won't answer that question, look it up."
2). Scaling. There are technologies addressing this issue. (Disclaimer I work for IBM, who sell some, so I'll reference these - I'm not intending to imply that IBM are the only vendor with solutions in this space.) There are products such as this that can provision a new machine, install a software stack and add it to a cluster to address workload changes. Then at a finer grained level of control in the Java EE world the Application Server can dynamically shape traffic and adjust clusters. See WebSphere Virtual Enterprise
3). Monitoring. I don't "get" what you expect here. In all likelyhood such tricky bugs will require application level trace. For some problems such as finding memory leaks and performance bottlenecks there are very good tools, at least in the Java EE world.
4). I can't speak to the .Net world, but I'd say that Java EE app servers do a reasonable job of deploying the apps across clusters smoothly, and in the cases where we use JNI and need DLLs deploying then we can use products such as the Tivoli stack I mention to manage this.
So, in summary, I do think that vendors are trying to address these issues. And I don't think your life would be simpler without SOA. Imagine instead the same problems applied to myriad separate, independent applications.
Here's my two cents.
I've been a developer at a company that used SOA incorrectly. The worst solution they implemented was field level validation of form elements on a desktop app using SOA. To perform acceptably these require very low latency. A 2-4 second wait to change to a new field gets old fast. The service ran over the network on a biztalk server. Everyone hated it.
If you're going to do this you really need to spend a lot of time dealing with network latency, service failure, timing, and timeout issues.
Don't get carried away and think SOA is the solution to every problem. Used at a high level it's great, used at a low level it makes your applications fragile, slow, and impossible to debug.
If you talk to IBM or one of the big SOA vendors, they got a products that cover each scenario.
Identification (Tracking services) - UDDI; e.g., had to remind a co-worker the 3 times this month where a service is at, despite the fact there is a wiki that discusses the service and a PDF version of the same documentation that lives in a repository where we keep our service docs.
Registry and Repository server. Nice thing is that it does governance (promotion, demotion, versioning, approval) and your ESB typically does a "lookup" for the latest and greatest against the register server.
Scalability - Out of the box clustering; As organizations, we spend a lot of money on paying our admins just to watch the utilization of our services and make decisions like, does this service need more RAM, more CPU, more interfaces? How do I load balance this?
Transaction monitoring software like IBM Tivoli Composite Application Manager for SOA. Basically, it tracks things from a horizontal point of view and to see if there is a service disruption from a end user/end app point of view.
As far as your clustering.... you have to pick good middleware and architecture. Personally speaking, get stuff that is "cloud" ready. App Servers with NoSQL connected by MOM.
Monitoring - error logging, etc; I can't count how many times I have to set up tracing on services in order to see why a bug is happening that only seems to affect one customer, or have to code logic into the service to serialize exceptions, log exceptions to dbs, fail gracefully, etc.
Enterprise standards for your developers and for your vendors. Integration of all business and system events into a single dashboard. (Most companies spilt them). This is done already at most enterprise shops.
Deployment - easy to deploy; none of this deploying DLLs to 5 load balanced servers
Ahh.. Microsoft IIS Web Deployment Tool 2.0. You can sync 100s of MS servers by just updating the master. It's really easy.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I am a programmer who writes a lot of code for desktop applications, now started considering cross-platform apps as an issue but at work I write C# apps and I come from C++ and CS background and of course, I wrote several things in QT/C++. But now I am kinda confused about web applications, I have done some work on PHP and I know how things go there, I was a gmail and google docs user for a lot of time and I have seen how much web applications were improved with new web 2.0 technology including Ajax, XML so on. And my confusion is that should I start looking forward for web application development? and continue exploring the power of web 2.0 or I have to just stick with my old world where I feel very comfortable on parallelism and other stuff? Because believe me I had too many offers to work as a web application developer but I didn't realize this opportunity and now I am kinda confused whether I must start writing web apps. Have you been writing desktop applications and switched to web? or have somebody experience in this scenario?
Thank you.
The boundaries between desktop and web applications have really blurred. Whilst once upon a time the nature of developing for the web was totally different to developing for the desktop, nowadays you find the same concepts (such as parallelism which you referred to) cropping up in both. Don't think of developing web applications as taking a huge step away from traditional software development as you'll employ just as many skills and concepts as you already use. You wouldn't need to learn a whole lot more to get involved in web development if you have C# experience, as you could code backends to web applications in a very similar way to how you currently work. If you wanted/needed to get involved in the UI side of things, there are new technologies you'd need to pick up, but they're not essential to get a job in web development (as long as you weren't looking for a frontend role obviously).
To follow up Dustman's comments about companies wanting to keep tight control of their data etc; bear in mind that not all "web applications" involve the use of the internet. Really all the term means is "applications developed on web-based technologies" and as well as being deployed publically on the web, they're commonly deployed on intranets and other closed-access environments. I work for a software company which develops "web applications" but a large number of systems are hosted by clients for use on their internal networks for the very reasons Dustman refers to - they want to keep tight control of their data. The beauty of web based technologies is that you can achieve this whilst still reaping the benefits of a centralised system, meaning there is no need to manage deployment across 100s of workstations, no need to worry too much about the specifications of client devices, the ability to access the system across different types of device (mobile etc), regular and easily deployed updates, and so I could continue.
It's all about what kind of programs you want to be writing. End-user apps already have already started a significant move to being web-oriented, because of the advantages that some companies find in outsourcing their data handling and IT infrastructure. Because this area of development is a new and growing sector, I have no doubt that you will be getting all kinds of offers, and hearing all about new startups and so forth that are centered on developing these kinds of applications.
That doesn't mean that desktop apps are going to go away. Some companies, and lots of private individuals like to have a sense of being in physical possession of their data, and see no monetary benefit in "renting" an online app or in outsourcing their data handling. These people are going to keep the desktop app market open in the foreseeable future, although perhaps not to the extent that we have seen previously.
So at this point, you needn't feel forced to make a move into the web game, but there are certainly opportunities there if you want them.
In the near future, the boundary between the web development and the desktop application development will go on erasing. For a professional programmer, learning new things is the real growth. learning web development for an experienced programmer is not a difficult task. you can obviously go ahead and learn the web development. You should recognize web well as it will certainly come to meet the desktop apps in the near future.