I'm trying to configure one Siddhi application on WSO2 Stream Processor with two sources (both are files) but that doesn't work (one source works fine). However, even when I split the application in to Siddhi application that doesn't work too. There logs in both situations are the same as below:
[2018-01-25 08:51:20,583] INFO {org.quartz.impl.StdSchedulerFactory} - Using default implementation for ThreadExecutor
[2018-01-25 08:51:20,586] INFO {org.quartz.simpl.SimpleThreadPool} - Job execution threads will use class loader of thread: Timer-0
[2018-01-25 08:51:20,599] INFO {org.quartz.core.SchedulerSignalerImpl} - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
[2018-01-25 08:51:20,599] INFO {org.quartz.core.QuartzScheduler} - Quartz Scheduler v.2.3.0 created.
[2018-01-25 08:51:20,600] INFO {org.quartz.simpl.RAMJobStore} - RAMJobStore initialized.
[2018-01-25 08:51:20,601] INFO {org.quartz.core.QuartzScheduler} - Scheduler meta-data: Quartz Scheduler (v2.3.0) 'polling-task-runner' with instanceId 'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 1 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.
[2018-01-25 08:51:20,601] INFO {org.quartz.impl.StdSchedulerFactory} - Quartz scheduler 'polling-task-runner' initialized from an externally provided properties instance.
[2018-01-25 08:51:20,601] INFO {org.quartz.impl.StdSchedulerFactory} - Quartz scheduler version: 2.3.0
[2018-01-25 08:51:20,601] INFO {org.quartz.core.QuartzScheduler} - Scheduler polling-task-runner_$_NON_CLUSTERED started.
[2018-01-25 08:51:20,604] INFO {org.quartz.core.QuartzScheduler} - Scheduler polling-task-runner_$_NON_CLUSTERED started.
[2018-01-25 08:51:20,605] ERROR {org.wso2.carbon.connector.framework.server.polling.PollingTaskRunner} - Exception occurred while scheduling job org.quartz.ObjectAlreadyExistsException: Unable to store Trigger with name: 'scheduledPoll' and group: 'group1', because one already exists with this identification.
at org.quartz.simpl.RAMJobStore.storeTrigger(RAMJobStore.java:415)
at org.quartz.simpl.RAMJobStore.storeJobAndTrigger(RAMJobStore.java:252)
at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:855)
at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:249)
at org.wso2.carbon.connector.framework.server.polling.PollingTaskRunner.start(PollingTaskRunner.java:74)
at org.wso2.carbon.connector.framework.server.polling.PollingServerConnector.start(PollingServerConnector.java:57)
at org.wso2.carbon.transport.remotefilesystem.server.connector.contractimpl.RemoteFileSystemServerConnectorImpl.start(RemoteFileSystemServerConnectorImpl.java:75)
at org.wso2.extension.siddhi.io.file.FileSource.deployServers(FileSource.java:537)
at org.wso2.extension.siddhi.io.file.FileSource.connect(FileSource.java:370)
at org.wso2.siddhi.core.stream.input.source.Source.connectWithRetry(Source.java:130)
at org.wso2.siddhi.core.SiddhiAppRuntime.start(SiddhiAppRuntime.java:335)
at org.wso2.carbon.stream.processor.core.internal.StreamProcessorService.deploySiddhiApp(StreamProcessorService.java:280)
at org.wso2.carbon.stream.processor.core.internal.StreamProcessorDeployer.deploySiddhiQLFile(StreamProcessorDeployer.java:81)
at org.wso2.carbon.stream.processor.core.internal.StreamProcessorDeployer.deploy(StreamProcessorDeployer.java:170)
at org.wso2.carbon.deployment.engine.internal.DeploymentEngine.lambda$deployArtifacts$0(DeploymentEngine.java:291)
at java.util.ArrayList.forEach(ArrayList.java:1249)
at org.wso2.carbon.deployment.engine.internal.DeploymentEngine.deployArtifacts(DeploymentEngine.java:282)
at org.wso2.carbon.deployment.engine.internal.RepositoryScanner.sweep(RepositoryScanner.java:112)
at org.wso2.carbon.deployment.engine.internal.RepositoryScanner.scan(RepositoryScanner.java:68)
at org.wso2.carbon.deployment.engine.internal.DeploymentEngine.start(DeploymentEngine.java:121)
at org.wso2.carbon.deployment.engine.internal.DeploymentEngineListenerComponent.onAllRequiredCapabilitiesAvailable(DeploymentEngineListenerComponent.java:216)
at org.wso2.carbon.kernel.internal.startupresolver.StartupComponentManager.lambda$notifySatisfiableComponents$7(StartupComponentManager.java:266)
at java.util.ArrayList.forEach(ArrayList.java:1249)
at org.wso2.carbon.kernel.internal.startupresolver.StartupComponentManager.notifySatisfiableComponents(StartupComponentManager.java:252)
at org.wso2.carbon.kernel.internal.startupresolver.StartupOrderResolver$1.run(StartupOrderResolver.java:204)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Anybody can throw an idea on how to overcome this?
Thanks.
Thanks for pointing this issue out.
Seems like this is occurred due to scheduling two polling tasks with the same id.
I have created an issue for this in git repository[1]. The fix will be shipped with an update soon.
[1] https://github.com/wso2/product-sp/issues/463
Best Regards!
Related
We are using Google Cloud Build as CI/CD tool and we use private pools to be able to connect to our database using private IPs.
Since 08/27 our builds using private pools are stuck in Queued and are never executed ou fail due to timeout, they just hang there until we cancel them.
We have already tried without success:
Change the worker pool to another region (from southamerica-east1 to us-central1);
Recreate the worker pool with different configurations;
Recreate all triggers and connections.
Removing the worker pool configuration (running the build in global) executed the build.
cloudbuild.yaml:
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
id: Backup database
args: ['gcloud', 'sql', 'backups', 'create', '--instance=${_DATABASE_INSTANCE_NAME}']
- name: 'node:14.17.4-slim'
id: Migrate database
entrypoint: npm
dir: 'build'
args: ['...']
secretEnv: ['DATABASE_URL']
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
id: Migrate traffic to new version
dir: 'build'
entrypoint: bash
args: ['-c', 'gcloud app services set-traffic ${_SERVICE_NAME} --splits ${_VERSION_NAME}=1']
availableSecrets:
secretManager:
- versionName: '${_DATABASE_URL_SECRET}'
env: 'DATABASE_URL'
options:
pool:
name: 'projects/$PROJECT_ID/locations/southamerica-east1/workerPools/<project-id>'
our worker pool configuration:
$ gcloud builds worker-pools describe <worker-pool-id> --region=southamerica-east1 --project=<project-id>
createTime: '2021-08-30T19:35:57.833710523Z'
etag: W/"..."
name: <worker-pool-id>
privatePoolV1Config:
networkConfig:
egressOption: PUBLIC_EGRESS
peeredNetwork: projects/<project-id>/global/networks/default
workerConfig:
diskSizeGb: '1000'
machineType: e2-medium
state: RUNNING
uid: ...
updateTime: '2021-08-30T20:14:13.918712802Z'
It was my last week discussion with the Cloud Build PM... TL;DR: if you haven't support subscription, or a corporate account, you can't (for now)
In detail, you can check the 1. link of RJC, you will get that
If you have a closer look, you can see (with my personal account, even if I have an Organization structure) the Concurrent Builds per worker pool is set to 0. That is the reason of your infinite queue of your build job.
The most annoying part is this one. Click on a Concurrent build per worker pool line checkbox and then click on edit, to change the limit. Here what you get
Read carefully: set a limit between 0 and 0.
Therefore, if you haven't support subscription (like me) you can't use the feature with your personal account. I was able to use it with my corporate account, even if I shouldn't...
For now, I haven't a solution, only this latest message from the PM
The behaviour around quota restrictions in private pools is a recent change that we're still iterating on and appreciate the feedback to make it easier for personal accounts to try out the feature.
The build in queue state can have the following possible reasons:
Concurrency limits. Cloud Build enforces quotas on running builds for various reasons. As a default, Cloud Build has only 10 concurrent build limit, whilst as per Worker Pool, it has a 30 concurrent build limit. You can also further check in this link for the quotas limit.
Using a custom machine size. In addition to the standard machine type, Cloud Build provides four high-CPU virtual machine types to run your builds.
You are using worker pools alpha and has too few nodes available.
Additionally, if the issue still persist, you can submit a bug under Google Cloud. I see that your colleague already submitted a public issue tracker in this link. In addition, if you have a free trial or paid support plan, it would be better to use it to file an issue.
I have more than 50 datafusion pipelines running concurrently in an Enterprise istance of DataFusion.
About 4 of them randomly fail at each concurrent run, showing in the logs only the operation of provision followed by the deprovision of the Dataproc cluster, as in this log:
2021-04-29 12:52:49,936 - INFO [provisioning-service-4:i.c.c.r.s.p.d.DataprocProvisioner#203] - Creating Dataproc cluster cdap-fm-smartd-cc94285f-a8e9-11eb-9891-6ea1fb306892 in project project-test, in region europe-west2, with image 1.3, with system labels {goog-datafusion-version=6_1, cdap-version=6_1_4-1598048594947, goog-datafusion-edition=enterprise}
2021-04-29 12:56:08,527 - DEBUG [provisioning-service-1:i.c.c.i.p.t.ProvisioningTask#116] - Completed PROVISION task for program run program_run:default.[pipeline_name].-SNAPSHOT.workflow.DataPipelineWorkflow.cc94285f-a8e9-11eb-9891-6ea1fb306892.
2021-04-29 13:04:01,678 - DEBUG [provisioning-service-7:i.c.c.i.p.t.ProvisioningTask#116] - Completed DEPROVISION task for program run program_run:default.[pipeline_name].-SNAPSHOT.workflow.DataPipelineWorkflow.cc94285f-a8e9-11eb-9891-6ea1fb306892.
When a failed pipeline is restarted it completes the execution with success.
All the pipeline are started and monitored via Composer using async start and custom wait SensorOperator.
There is no warning of quota exceeded.
Additional info:
Data Fusion 6.1.4
with Dataporc ephemeral cluster with 1 master 2 workers. Image version 1.3.89
EDIT
The appfabric log releted to each failed pipeline are:
WARN [program.status:i.c.c.i.a.r.d.DistributedProgramRuntimeService#172] - Twill RunId does not exist for the program program:default.[pipeline_name].-SNAPSHOT.workflow.DataPipelineWorkflow, runId f34a6fb4-acb2-11eb-bbb2-26edc49aada0
WARN [pool-11-thread-1:i.c.c.i.a.s.RunRecordCorrectorService#141] - Fixed RunRecord for program run program_run:default.[piepleine_name].-SNAPSHOT.workflow.DataPipelineWorkflow.fdc22f56-acb2-11eb-bbcf-26edc49aada0 in STARTING state because it is actually not running
Further research connected somehow the problem to an inconsistent state in the CDAP run records, when many concurrent requests (via REST API) are made.
I started with Amazon's Java-based HelloWorldWorkflowDistributed example and I'm adding to it little by little to achieve what we want. I have added a second activity worker, but the two activities are receiving each other's tasks and no tasks are getting accomplished. Can anyone point me to a COMPLETE, WORKING example of a workflow that calls out to two or more distinct workers?
E.g. the following error appears in the console where BarActivities.getName is running, and vice versa:
Aug 26, 2016 2:15:24 PM com.amazonaws.services.simpleworkflow.flow.worker.SynchronousActivityTaskPoller execute
SEVERE: Failure processing activity task with taskId=10, workflowGenerationId=id_for_107, activity={Name: FooActivities.getAddress,Version: 1.0.7}, activityInstanceId=1
com.amazonaws.services.simpleworkflow.flow.ActivityFailureException: Unknown activity type: {Name: FooActivities.getAddress,Version: 1.0.7} : null
at com.amazonaws.services.simpleworkflow.flow.worker.SynchronousActivityTaskPoller.execute(SynchronousActivityTaskPoller.java:194)
at com.amazonaws.services.simpleworkflow.flow.worker.ActivityTaskPoller$2.run(ActivityTaskPoller.java:92)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Activity workers poll for activity tasks using task lists. I believe you added a new worker without using a separate task list for its activities. As both workers share the same task list they end up sometimes receiving tasks for activities that they don't support which results in the "Unknown activity type" exception. The solution is to use a different task list for each worker.
Update #2 to reflect the comments....
I have tried correcting for this issue launching the services varous ways...
#./wso2server.sh start
#./wso2server.sh -Dprofile=api-key-manager
#nohup ./wso2server.sh -Dprofile=api-key-manager &
Unfortuneately the errors have started occurring regardless.
Earlier post below...
Please note this is an edit...didn't want to post yet another question...
So now I'm really confused. After upgrading WSO2 AM (API Manager) 1.6.0 and the problem persisting I decided to split up the services into Key Manager, Gateway, Store, and Publisher.
So now I have:
I have one server running the store and publisher on different ports.
I have another server running the gateway and keymanager on different ports.
The first service I started was Key Manager... which failed to start with:
2014-01-22 18:09:41,786 [-] [Timer-2] WARN CarbonServerManager Carbon initialization is delayed due to the following unsatisfied items:
2014-01-22 18:09:41,788 [-] [Timer-2] WARN CarbonServerManager Waiting for required OSGiAxis2Service: org.wso2.carbon.mediation.configadmin-4.2.0
2014-01-22 18:09:41,789 [-] [Timer-2] WARN CarbonServerManager Waiting for required OSGiAxis2Service: org.wso2.carbon.apimgt.gateway-1.2.1
2014-01-22 18:09:41,789 [-] [Timer-2] WARN CarbonServerManager Waiting for required OSGiAxis2Service: org.wso2.carbon.proxyadmin-4.2.1
2014-01-22 18:09:41,790 [-] [Timer-2] WARN CarbonServerManager Waiting for required OSGiAxis2Service: org.wso2.carbon.localentry-4.2.0
Which is confusing to me since the Key Manager should not rely on the other components in AM. It only connects to the APIMGT_DB which I've added in api-manager.xml
jdbc/WSO2AM_DB
Any advice?
Here is a wild suggestion that you can tryout. I re-produced your issue frequently in APIM 1.6 server start-up, and after this modification I haven't seen this error anymore. So, may be this would work for you too.
Open the jar file, synapse-core_2.1.2.wso2v3.jar, at APIM_HOME/repository/components/plugins/. You can use a archive openers like file-roller.
Open the file at META-INF/MANIFEST.MF.
Search for the text, javax.xml.soap;version="0.0.0" . Replace it with - javax.xml.soap;version="1.0.0" . (The difference is in the version)
Save the file, and restart the server.
I had the same issue, after we moved from AM 1.5.0 to 1.6.0, the issue disappears.
Problem
I have a very basic configuration for a Spring integration mail adapter setup (below is the relevant sample):
<int:channel id="emailChannel">
<int:interceptors>
<int:wire-tap channel="logger"/>
</int:interceptors>
</int:channel>
<mail:inbound-channel-adapter id="popChannel"
store-uri="pop3://user:password#domain.net/INBOX"
channel="emailChannel"
should-delete-messages="true"
auto-startup="true">
<int:poller max-messages-per-poll="1" fixed-rate="30000"/>
</mail:inbound-channel-adapter>
<int:logging-channel-adapter id="logger" level="DEBUG"/>
<int:service-activator input-channel="emailChannel" ref="mailResultsProcessor" method="onMessage" />
This is working fine the majority of the time and I can see the logs showing the polling (and it works fine hooking into my mailResultsProcessor when a mail is there):
2013-08-13 08:19:29,748 [task-scheduler-3] DEBUG org.springframework.integration.mail.Pop3MailReceiver - opening folder [pop3://user:password#fomain.net/INBOX]
2013-08-13 08:19:29,796 [task-scheduler-3] INFO org.springframework.integration.mail.Pop3MailReceiver - attempting to receive mail from folder [INBOX]
2013-08-13 08:19:29,796 [task-scheduler-3] DEBUG org.springframework.integration.mail.Pop3MailReceiver - found 0 new messages
2013-08-13 08:19:29,796 [task-scheduler-3] DEBUG org.springframework.integration.mail.Pop3MailReceiver - Received 0 messages
2013-08-13 08:19:29,893 [task-scheduler-3] DEBUG org.springframework.integration.endpoint.SourcePollingChannelAdapter - Received no Message during the poll, returning 'false'
The problem I have is that the polling stops during the day, with no indication in the logs why it has stopped working. The only reason I can tell is the debug above is not present in the logs and E-Mails build up on the E-Mail account.
Questions
Has anyone seen this before and know how to resolve it?
Is there a change that I can make in my configuration to capture the issue into the log? I thought the logging channel adapter set to debug would have this covered.
Using version 2.2.3.RELEASE of Spring Integration on Tomcat 7, logs output default to catalina.out. Deployed on AWS standard tomcat 7 instance.
Most likely the poller thread is hung someplace upstream. With your configuration, the next poll won't happen until the current poll completes.
You can use jstack or VisualVM to get a thread dump to find out what the thread is doing.
Another possibility is you are suffering from poller thread starvation - if you have a lot of other polled elements in your application, and depending on their configuration. The default taskScheduler bean has only 10 threads.
You can add a task executor to the <poller/> so each poll is handed off to another thread, but be aware that that can result in concurrent polls if a polled task takes longer to execute than the polling rate.
To resolve this problem specifically I used the configuration below:
<mail:inbound-channel-adapter id="popChannel"
store-uri="pop3://***/INBOX"
channel="emailChannel"
should-delete-messages="true"
auto-startup="true">
<int:poller max-messages-per-poll="5" fixed-rate="60000" task-executor="pool"/>
</mail:inbound-channel-adapter>
<task:executor id="pool" pool-size="10" keep-alive="50"/>
Once moving to this approach we saw no further problems, and is with any use of pool the advantage is any Threads that become a problem are cleaned up and recreated.