Thanks advance for your help.
I deploy spring-cloud-dataflow-server-yarn with ambari. But when I start a stream jdbc-source-kafka and hdfs-sink-kafka. The stream deploy not run.
The streams config
jdbc-source-kafka --max-rows-per-poll=10
--query='select t.* from clear_order_report t'
--password=******
--driver-class-name=oracle.jdbc.OracleDriver
--username=****** --url=jdbc:oracle:thin:#10.48.171.21:1521:******
| hdfs-sink-kafka
--fs-uri=hdfs://master.99wuxian.com:8020
--file-name=clear_order_report
--directory=/dataflows/apps/top
I aslo repackage the jdbc-source-kafka-10-1.1.1.RELEASE.jar and add oracle jdbc driver.
And the yarn log is below
Application Overview
User: scdf
Name: scdstream:app:oracle2hdfs
Application Type: DATAFLOW
Application Tags:
Application Priority: 0 (Higher Integer value indicates higher priority)
YarnApplicationState: ACCEPTED: waiting for AM container to be allocated, launched and register with RM.
Queue: default
FinalStatus Reported by AM: Application has not completed yet.
Started: 星期四 二月 09 17:38:33 +0800 2017
Elapsed: 21hrs, 18mins, 27sec
Tracking URL: ApplicationMaster
Log Aggregation Status NOT_START
Diagnostics: [星期四 二月 09 17:38:34 +0800 2017] Application is added to the scheduler and is not yet activated. Queue's AM resource limit exceeded. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:1024, vCores:1>; Queue Resource Limit for AM = <memory:1280, vCores:1>; User AM Resource Limit of the queue = <memory:1280, vCores:1>; Queue AM Resource Usage = <memory:1024, vCores:1>;
Unmanaged Application: false
Application Node Label expression: <Not set>
AM container Node Label expression: <DEFAULT_PARTITION>
Related
I need to find strings in a log file with regex and later send output to Zabbix monitoring server to fire triggers if needed.
For example here is a part of the log file:
===== Backup Failures =====
Description: Checks number of studies that their backup failed
Status: OK , Check Time: Sun Oct 30 07:31:13 2022
Details: [OK] 0 total backup commands failed during the last day.
===== Oracle queues =====
Description: Count Oracle queues sizes. The queues are used to pass information between the applications
Status: OK , Check Time: Sun Oct 30 07:31:04 2022
Details: [OK] All queues have less than 15 elements.
===== Zombie Services =====
Description: Checks for zombie services
Status: Error , Check Time: Sun Oct 30 07:31:30 2022, Script: <check_mvs_services.pl>
Details: [CRITICAL] 1 missing process(es) found. Failed killing 1 process(es)
===== IIS Application Pools Memory Usage =====
Description: Checks the memory usage of the application pools that run under IIS (w3wp.exe)
Status: OK , Check Time: Sun Oct 30 07:32:30 2022
Details: [OK] All processes of type w3wp.exe don't exceed memory limits
===== IIS Web Response =====
Description: Checks that the web site responds properly
Status: OK , Check Time: Sun Oct 30 07:32:34 2022
Details: [OK] All addresses returned 200
I need to find all items for monitoring and it's results.
If results not OK then Zabbix triggers should send alarm.
I found Zabbix can handle log file monitoring with similar command here but first need to find strings in the log file:
log[/path/to/the/file,"regex expression",,,,]
In this example I believe these items should find for Zabbix:
===== Backup Failures =====
Details: [OK] 0 total backup commands failed during the last day.
===== Oracle queues =====
Details: [OK] All queues have less than 15 elements.
===== Zombie Services =====
Details: [CRITICAL] 1 missing process(es) found. Failed killing 1 process(es)
===== IIS Application Pools Memory Usage =====
Details: [OK] All processes of type w3wp.exe don't exceed memory limits
===== IIS Web Response =====
Details: [OK] All addresses returned 200
Can you advise how possible to achieve this solution?
For any help I would be really appreciated.
Thanks in advance.
I am seeing something similar to this post. It looked like additional detail was needed to answer that question, so I'm re-asking with my details since those details weren't provided.
I am running a modified version of the Google Cloud Run image processing tutorial example.
I am inserting tasks into a task queue using this create tasks snippet. The tasks from the queue get pushed to my cloud run instance.
The problem is it isn't scaling up and making it through my tasks in a timely manner.
My cloud run service configuration:
I have tried setting a minimum of both 0 and 50 instances
I have tried a maximum of 100 and 1000 instances
I have tried --concurrency=1 and 2, and 8
I have tried with --async and without --async
With 50 instances pre-allocated even with concurrency set to 1, I am typically seeing ~10 active container instances and ~40 idle container instances. I have ~30,000 tasks in the queue and it is getting through ~5 jobs/minute.
My tasks queue has the default settings. My containers aren't using a lot of cpu, but they are using a lot of memory.
A process takes about a minute to complete. I'm only running one process per container instance. What additional parameters should be set to get higher throughput?
Edit - adding additional logs
I enabled the logs for the queue, I'm seeing some errors for some of the jobs. The errors look like this:
{
insertId: "<my_id>"
jsonPayload: {
#type: "type.googleapis.com/google.cloud.tasks.logging.v1.TaskActivityLog"
attemptResponseLog: {
attemptDuration: "19.453155s"
dispatchCount: "1"
maxAttempts: 0
responseCount: "0"
retryTime: "2021-10-20T22:45:51.559121Z"
scheduleTime: "2021-10-20T16:42:20.848145Z"
status: "UNAVAILABLE"
targetAddress: "POST <my_url>"
targetType: "HTTP"
}
task: "<my_task>"
}
logName: "<my_log_name>"
receiveTimestamp: "2021-10-20T22:45:52.418715942Z"
resource: {
labels: {
location: "us-central1"
project_id: "<my_project>"
queue_id: "<my-queue>"
target_type: "HTTP"
}
type: "cloud_tasks_queue"
}
severity: "ERROR"
timestamp: "2021-10-20T22:45:51.459232147Z"
}
I don't see errors in the cloud run logs.
Edit - Additional Debug Information
I tried to take the queue out of the equation to determine if it is cloud run or the queue. Instead I directly used curl to post to the url. Some of the tasks ran successfully, for others I received an error. In the below logs empty lines are successful:
upstream connect error or disconnect/reset before headers. reset reason: connection termination
upstream connect error or disconnect/reset before headers. reset reason: connection termination
upstream connect error or disconnect/reset before headers. reset reason: connection termination
upstream connect error or disconnect/reset before headers. reset reason: connection termination
upstream connect error or disconnect/reset before headers. reset reason: connection termination
This makes me think cloud run isn't handling all of the incoming requests.
Edit - task completion time test
I wanted to test if the time it takes to complete a task causes any issues with CloudRun and the Queue scaling up and keeping up with the tasks.
In place of the task I actually want completed I put a dummy task that just sleeps for n seconds and prints the task details to stdout (which I can read in the cloud run logs).
With n set to 0, 5, 10 seconds I see the number of instances scale up and it keeps up with the tasks being added to the queue. With n set to 20 seconds or more I see that less CloudRun instances are instantiated and items accumulate in the task queue. I see more errors with the Unavailable status in my logs.
According to this post:
Cloud Run offers a longer request timeout duration of up to 60 minutes
So it seems that long running tasks are expected. Is this a Google bug or am I missing setting some parameter?
I do not think this is a Cloud Run Service problem. I think this is an issue with how you have Tasks setup.
The dates in the log entry look odd. Take a look at the receiveTimestamp and the scheduleTime. The task is scheduled for six hours before the receive time. Do you have a timezone problem?
According to the documentation, if the response_time is not set then the task was not attempted. It looks like you are scheduling tasks incorrectly and the tasks never run.
Search for the text The status of a task attempt. in this link:
Types for Google Cloud Tasks
I start make image mikrotik-chr from my bucket but always error. I dontt know how to fix it
[inflate.import-virtual-disk]: 2021-08-16T05:39:39Z CreateInstances: Creating instance "inst-importer-inflate-6t2qt".
[inflate]: 2021-08-16T05:39:46Z Error running workflow: step "import-virtual-disk" run error: operation failed &{ClientOperationId: CreationTimestamp: Description: EndTime:2021-08-15T22:39:46.802-07:00 Error:0xc00007b770 HttpErrorMessage:SERVICE UNAVAILABLE HttpErrorStatusCode:503 Id:1873370325760361715 InsertTime:2021-08-15T22:39:40.692-07:00 Kind:compute#operation Name:operation-1629092379433-5c9a6a095186f-620afe4b-ba26ba50 OperationGroupId: OperationType:insert Progress:100 Region: SelfLink:https://www.googleapis.com/compute/v1/projects/circular-jet-322614/zones/asia-southeast2-a/operations/operation-1629092379433-5c9a6a095186f-620afe4b-ba26ba50 StartTime:2021-08-15T22:39:40.692-07:00 Status:DONE StatusMessage: TargetId:6947401086746772724 TargetLink:https://www.googleapis.com/compute/v1/projects/circular-jet-322614/zones/asia-southeast2-a/instances/inst-importer-inflate-6t2qt User:606260965808#cloudbuild.gserviceaccount.com Warnings:[] Zone:https://www.googleapis.com/compute/v1/projects/circular-jet-322614/zones/asia-southeast2-a ServerResponse:{HTTPStatusCode:200 Header:map[Cache-Control:[private] Content-Type:[application/json; charset=UTF-8] Date:[Mon, 16 Aug 2021 05:39:46 GMT] Server:[ESF] Vary:[Origin X-Origin Referer] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]]} ForceSendFields:[] NullFields:[]}:
Code: ZONE_RESOURCE_POOL_EXHAUSTED
Message: The zone 'projects/circular-jet-322614/zones/asia-southeast2-a' does not have enough resources available to fulfill the request. Try a different zone, or try again later.
[inflate]: 2021-08-16T05:39:46Z Workflow "inflate" cleaning up (this may take up to 2 minutes).
[inflate]: 2021-08-16T05:39:48Z Workflow "inflate" finished cleanup.
[import-image]: 2021-08-16T05:39:48Z Finished creating Google Compute Engine disk
[import-image]: 2021-08-16T05:39:49Z step "import-virtual-disk" run error: operation failed &{ClientOperationId: CreationTimestamp: Description: EndTime:2021-08-15T22:39:46.802-07:00 Error:0xc00007b770 HttpErrorMessage:SERVICE UNAVAILABLE HttpErrorStatusCode:503 Id:1873370325760361715 InsertTime:2021-08-15T22:39:40.692-07:00 Kind:compute#operation Name:operation-1629092379433-5c9a6a095186f-620afe4b-ba26ba50 OperationGroupId: OperationType:insert Progress:100 Region: SelfLink:https://www.googleapis.com/compute/v1/projects/circular-jet-322614/zones/asia-southeast2-a/operations/operation-1629092379433-5c9a6a095186f-620afe4b-ba26ba50 StartTime:2021-08-15T22:39:40.692-07:00 Status:DONE StatusMessage: TargetId:6947401086746772724 TargetLink:https://www.googleapis.com/compute/v1/projects/circular-jet-322614/zones/asia-southeast2-a/instances/inst-importer-inflate-6t2qt User:606260965808#cloudbuild.gserviceaccount.com Warnings:[] Zone:https://www.googleapis.com/compute/v1/projects/circular-jet-322614/zones/asia-southeast2-a ServerResponse:{HTTPStatusCode:200 Header:map[Cache-Control:[private] Content-Type:[application/json; charset=UTF-8] Date:[Mon, 16 Aug 2021 05:39:46 GMT] Server:[ESF] Vary:[Origin X-Origin Referer] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]]} ForceSendFields:[] NullFields:[]}: Code: ZONE_RESOURCE_POOL_EXHAUSTED; Message: The zone 'projects/circular-jet-322614/zones/asia-southeast2-a' does not have enough resources available to fulfill the request. Try a different zone, or try again later.
ERROR
ERROR: build step 0 "gcr.io/compute-image-tools/gce_vm_image_import:release" failed: step exited with non-zero status: 1
You will need to check if you have enough CPUs and other resources quota in 'projects/circular-jet-322614/zones/asia-southeast2-a'. Resource requirement can be found by looking at the deployment specs of the workload.
I am running corda 4.5. My flows work perfectly when run using gradle task, deployNodes. But when I run the flow for the nodes created using the network bootstrapper, I run to the below exception.
Mon Jul 26 12:43:10 GMT 2021>>> start CreateAccount name: accountOne
▶︎ Starting
✘ Requesting signature by notary service
✘ Requesting signature by Notary service
✘ Validating response from Notary service
✘ Broadcasting transaction to participants
✘ Done
☂ java.lang.IllegalArgumentException: Flow sessions were not provided for the following transaction participants: [O=PartyA, L=New York, C=US]
From the logs:
inMemoryConfigSelectionLogger. - Did not detect a configuration for InMemory selection - enabling memory usage for token indexing. Please set stateSelection.inMemory.enabled to "false" to disable this
inMemoryConfigSelectionLogger. - No indexing method specified. Indexes will be created at run-time for each invocation of selectTokens - Cordapp configuration is incorrect due to exception - empty config: No configuration setting found for key 'stateSelection'
like Sneha said in the comments it's impossible to be confident about the source of the issues here without more context about the code here.
Remember that you want to be sure flow sessions are provided in a way similar to this where you specify the identities in a list and submit them to finality flow.
Party me = getOurIdentity();
Party notary = getServiceHub().getNetworkMapCache().getNotaryIdentities().get(0);
Command<YoContract.Commands.Send> command = new Command<YoContract.Commands.Send>(new YoContract.Commands.Send(), Arrays.asList(me.getOwningKey()));
YoState state = new YoState(me, this.target);
StateAndContract stateAndContract = new StateAndContract(state, YoContract.ID);
TransactionBuilder utx = new TransactionBuilder(notary).withItems(stateAndContract, command);
this.progressTracker.setCurrentStep(VERIFYING);
utx.verify(getServiceHub());
this.progressTracker.setCurrentStep(SIGNING);
SignedTransaction stx = getServiceHub().signInitialTransaction(utx);
this.progressTracker.setCurrentStep(FINALISING);
FlowSession targetSession = initiateFlow(this.target);
return subFlow(new FinalityFlow(stx, Collections.singletonList(targetSession), Objects.requireNonNull(FINALISING.childProgressTracker())));
}
source: https://github.com/corda/samples-java/blob/master/Features/dockerform-yocordapp/workflows/src/main/java/net/corda/samples/dockerform/flows/YoFlow.java
I followed the official guide to set up a cluster (Clustering AS 5.3.0) (https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster).
But eventually, I could not reach the management page with https://localhost:9443/carbon, and the
Manager Node (10.13.46.34): (with some Error when passing date, i still dont know how to fix)
wso2server -Dsetup
[05-10 11:58:29]ERROR {org.wso2.carbon.registry.indexing.solr.SolrClient}-Error when passing date to create solr date format.java.text.ParseException: Unparseable date: "Tue May 03 17:35:
14 CST 2016"
[05-10 12:01:04]INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme}-Member joined [a9402117-a832-4eb6-b563-a58949ff784e]: /10.0.34.41:4200
[05-10 12:01:06]INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils}-Added member: Host:10.0.34.41, Remote Host:null, Port: 4200, HTTP:9763, HTTPS:9443, Domain: wso2.as.doma
in, Sub-domain:worker, Active:true
[05-10 12:03:31]INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}-'admin#carbon.super [-1234]' logged in at [2016-05-10 12:03:31,999+0800]
Worker node(10.0.34.44):
wso2server.bat -DworkerNode=true
......
......
[05-10 12:01:25]INFO {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent}-Server :Application Server-5.3.0
[05-10 12:01:25]INFO {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent}-WSO2 Carbon started in 88 sec
[05-10 12:01:26]INFO {org.wso2.carbon.ui.internal.CarbonUIServiceComponent} - Mgt Console URL : https://10.0.34.44:9443/carbon/
[05-10 12:02:20]INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2016-05-10 12:02:20,817+0800]
i can successfully login Manager-node's mgt console ('https://10.13.46.34:9443/carbon/')
but fail to login Worker-node's mgt console (https://10.0.34.44:9443/carbon/)
So, anyone can tell me how Manager-node's console page to list out a set of application servers? because I want to manage all node together.
And how to deploy a web-application to all nodes in this AS cluster environment.
thanks!
When you start the WSO2 AS node with -DworkerNode=true then you can't access the UI. Because normally worker nodes are use to serve the requests. therefore worker profile doesnt cantains ui features
According to your comment you are having a one manager node and one worker node. You can use deployment synchronizer to deploy webapps in worker nodes. Basically what is happening through that is when you deploy a webapp in the management node it will be commit to a svn location and worker node will checkout that. So worker node also get a copy of the app.
You can refer to https://docs.wso2.com/display/CLUSTER44x/Configuring+SVN-Based+Deployment+Synchronizer for more details and setup
Or simply you can copy the war file to the repository/deployment/server/webapp folder manually in the worker node.