We have developed a custom JAX-WS application that essentially achieves two things.
Exposes a few web service methods to perform some functionality.
Utilizes org.quartz.Scheduler to schedule and execute some polling tasks that monitors and processes data on a few database tables. (The logic here is slightly complex, hence a custom application was chosen over the use of WSO2 DSS)
This application is uploaded on WSO2 AS 5.2.1 and runs quite seamlessly. However, I'm unsure what will happen if we have to cluster the AS application server. Logically, I would think that each node will have its own instance of the custom application running within it, and hence its own scheduler. Would this not increase the risk of processing the same record, across both instances. Is my interpretation of the above scenario correct, from a clustering perspective?
Yes.You are correct.In cluster of app server nodes each nodes will have its own instance of the application.In your case each node will have seperate scheduler.You may consider using tasks from ESB 4.9.0. there WSO2 has added coordination support to work in cluster environment.
Related
As I understand with akka-cluster, frontend node receives the request and sends the job to one of the backend nodes for executing it. I want to find out for debugging purpose how to know which of the backend node is executing the job?
Does akka also provides some UI where one can look for current job executions happening on different backends?
There is nothing in Akka Cluster that is specifically about work scheduling or frontend and backend nodes, that would just be one application out of many that you can possibly build on top of Akka Cluster and as such, if you would want an UI of some kind you would build that as well for your application.
I want to start several web-server, and every server has a quartz instance for avoiding the job being interrupted by restarting the server.
I found that immutant can config the single job .But when i run the server i found that the scheme use the not-cluster config.And i do not know how to config it.
Immutant has built-in support for singelton jobs, but it requires running your application in a WildFly cluster, and does not use Quartz's clustering functionality.
Quartz clustering requires a JDBC JobStore, and Immutant does not currently expose a way to set a JobStore for the scheduler instance. The clustering works by using the database to lock the job - it would not be difficult to implement something similar yourself, by scheduling the same job on every node in the cluster, and using an external store as a synchronization mechanism, allowing the job to run on only one node at a time.
If you truly need the clustering inplementation in Quartz, or need more control over scheduler creation than Immutant provides, please file an issue against Immutant to have those options exposed. In the interim, you could take a look at Quartzite, I believe it exposes more options for scheduler creation.
Following is a point mentioned in a presentation slide related to SOA, and it confuses me with the concepts of service orchestration and service choreography. To enable service choreography, shouldn't a web service be able to call another web service?
SOA builds applications out of software services. Services comprise intrinsically
unassociated, loosely coupled units of functionality that have no calls to
each other embedded in them.
In theory, a service can do anything it needs to do to accomplish its job. So there doesn't seem to be a good reason to forbid using a second service to do your work. Why reinvent the wheel?
In practice, the issue is more complicated. If you start calling other services on your own web server, then you'll eventually starve it of resources. At best, "real" clients will have to wait a bit longer for their answers while your web service server plays with itself.
Another issue is recursive loops: Service A calls B calls C calls A calls B ... you get the idea. A small change in one service can introduce such a loop without anyone noticing and it can sit there for a long time until it suddenly kills your server.
That is why you should build micro services in a hierarchy inside the server (i.e. below the web service layer - this is not exposed to clients). Those micro services can use each other in a top-down manner (to avoid the loops). Unit tests then make sure they behave properly.
Lastly, such reuse is very slow. Each HTTP request takes a lot of resources to create, send, parse and process. Calling an internal method directly can be 10 - 10000 times faster.
These are the main reasons why the services exposed by a single server shouldn't reuse each other via the "public client API".
Note: There are web services which build new services by using existing ones. IFTTT - "If This Then That" is one such beast.
You could adopt every concept according to your needs. In my current project we have a separate module that is responsible for the Orchestration. This is required since in real life usage, scenarios can be very complicated. So in order to be close to the actual management of your system, you need to have such one.
Another advantage of this approach is that the Separation_of_concerns is kept. Also aligns the business request with the applications, data, and infrastructure that you have. It defines policies and service levels through automated workflows, provisioning etc.
Orchestration is critical in the delivery of Cloud services too. As they are networked to allow sharing of data-processing tasks, centralized data storage, and online access to services or resources.
I have some code that needs to run as a result of a call to a service bus. This particular code is CPU intensive and it is possible that 100s of these will need to run at the same time. Does Azure Web Jobs use computing resources from one machine, or does it use any available computing resources from several machines?
Web Jobs uses your web app resources, why don't you try the Azure Functions which can be scaled and their pricing is almost zero. They are in Technical preview i have tried using it, Azure functions is very cheap. If your service bus call can be out of process service meaning it does not need a instance results from your application you can try azure functions. Azure functions are mostly used for maintenance and night time running jobs. I have used it to minify my images to thumbnail. It worked perfectly fine
Azure Webjobs is designed to run on as many servers as you've scaled up the website to run on. By default it will run up to 16 tasks from a queue concurrently but this is configurable as shown here.
public class Program
{
static void Main()
{
JobHostConfiguration config = new JobHostConfiguration();
config.Queues.BatchSize= 1;
JobHost host = new JobHost(config);
host.RunAndBlock();
}
}
A web job deployed as part of an Azure WebSite will share the resources with the web application. You have the option to scale the website up to 10 (I think?) instances if you need parallelism.
As I mentioned in the comment to Matthew's post, if you use the Azure WebJobs SDK to have functions triggered by ServiceBus queue messages, we don't parallelize as part of the same host. This means that messages will be processed sequentially as long as you have a single host.
I am new to WSO2 ESB clustering, actually I'm still learning about it. I still don't understand the concept here.
In my case, I installed WSO2 ESB on 2 servers. My questions are:
1. Were both of the WSO2 ESB working as one application or as two separate applications?
2. If I configure one WSO2 ESB, will the other ESB have the same configuration?
3. If I configure a VFS proxy service the ESB to poll a file from specific directory, will it create a conflict when I poll a file? I think that both of the ESB will poll the same file.
Please enlighten me :) Thanks...
Check my answers inline.
Were both of the WSO2 ESB working as one application or as two separate applications?
In any cluster, applications or servers are working together to provide a high availability to the end users. It will work as one single server (application).
If I configure one WSO2 ESB, will the other ESB have the same configuration?
Yes. You can achieve this with the deployment synchronization. It will make sure all your changes are evenly distributed among other nodes of the cluster.
If I configure a VFS proxy service the ESB to poll a file from specific directory, will it create a conflict when I poll a file? I think that both of the ESB will poll the same file.
No. Since only one server is active at once. This should not be a problem.
You can learn more from the following link:
http://docs.wso2.org/display/CLUSTER420/Clustering+WSO2+Products
Were both of the WSO2 ESB working as one application or as two separate applications?
No, both ESBs are separate applications. The clustering is done mainly to ensure availability and scalability. So even a member in a cluster fails, others continue to operate.
If I configure one WSO2 ESB, will the other ESB have the same configuration?
Each ESB can get the same cluster configuration, but each will be separately identified by the LB that is fronting the cluster. Therefore, each member will get different IP addresses and even they can use different member-port to create cluster.
If I configure a VFS proxy service the ESB to poll a file from specific directory, will it create a conflict when I poll a file? I think that both of the ESB will poll the same file.
Each request is independently handled by separate ESB depending on the load balancer algorithm of the fronting LB. When two threads poll the same file, there can be conflicts. Since VFS transport deals with file operations, there are certain times that these can fail due to unavailability of some resources. In such a case, VFS transport is equipped with the following fault handling mechanism.
If you point the same directory in a clustered environment, both proxy services will try to poll files and cause issues. Therefore if you want to poll files in a clustered environment, best practice is to use inbound endpoints [1]. But if it is necessary to use proxy services you can apply the following property in your proxy, so that the proxy service will only operate in one server. Thus there will not be any conflicts between the two proxy services. Please refer [2] to further clarify this.
<parameter name="transport.vfs.ClusterAware">true</parameter>
[1]-https://docs.wso2.com/display/EI620/File+Inbound+Protocol
[2]-https://docs.wso2.com/display/ESB500/VFS+Transport