I am trying to integrate external cassandra to BAM. I have changed cassandra-component.xml.
1) I want to know how keyspace are created on external cassandra because when I am running BAM,
I am getting the error Unknown keyspace EVENT_KS.
2) I am having the following error in my wso2 logs
TID: [0] [BAM] [2014-02-11 15:28:30,905] WARN {org.apache.hadoop.mapred.JobClient} - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. {org.apache.hadoop.mapred.JobClient}
TID: [0] [BAM] [2014-02-11 15:37:04,393] ERROR {org.apache.hadoop.hive.ql.exec.ExecDriver} - Job Submission failed with exception 'java.lang.RuntimeException(org.apache.thrift.transport.TTransportException)'
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getRangeMap(ColumnFamilyInputFormat.java:297)
at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:105)
at org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat.getSplits(HiveCassandraStandardColumnInputFormat.java:291)
at org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat.getSplits(HiveCassandraStandardColumnInputFormat.java:216)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:302)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:292)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:933)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:925)
at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:839)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:792)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1123)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:792)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:766)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:460)
at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:733)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.transport.TTransportException
EVENT_KS is created only after the very first event is published to BAM as I remember. If you try to access it before getting created errors may arise.
In BAM 2.4.0, EVENT_KS is getting created when you run the BAM for the first time. (But in previous versions EVENT_KS will be created when the very first event is published to BAM). Please make sure your cassandra-component.xml looks similar to something like below. Also tell us about the cassandra version you are using.
<Cassandra><Cluster>
<Name>Test Cluster</Name>
<DefaultPort>9160</DefaultPort>
<Nodes>localhost:9160</Nodes>
<AutoDiscovery disable="false" delay="1000"/>
</Cluster></Cassandra>
First You need to check the following:
Have you pointed the cassandra-component.xml correctly to the external cassandra. With this your published data will be stored in the intended external cassandra database.
Have you installed a toolbox with intended stream definition inside? Or Else have you triggered to publish the data to BAM? In both cases the EVENT_KS will be created with the column family with the name of stream.
Have you modified the $BAM_HOME/repository/conf/datasource/master-datasource.xml to point to external cassandra databse? You need to validate the cassandra database configuration provided in WSO2BAM_CASSANDRA_DATASOURCE datasource. For the default toolboxes, this is the default cassandra datasource being used and by default it points to localhost. If you are using this in your hive script you need to change this configuration.
After putting many efforts i figure that after changing data directory of cassendra.yaml of external cassandra to repository/database/cassandra/data everything works fine with external cassandra.Not to mention with version 1.1.3. I want to know is there any other work around for this external cassandra configuration.
Related
I am setting up WSO2 APIM with Analytics running on docker. I am getting the following error in the worker & no data is being published to the console. I'm using a MySQL database.
I am using the docker images # https://github.com/wso2/docker-apim/tree/v2.6.0.3/dockerfiles/centos
OpenJDK8U-jdk_x64_linux_hotspot_8u222b10
mysql-connector-java-5.1.47-bin.jar
[2019-10-05 04:58:27,208] ERROR {org.wso2.carbon.databridge.core.internal.queue.QueueWorker} - Dropping wrongly formatted event sent org.wso2.carbon.databridge.core.exception.EventConversionException: Error when converting org.wso2.metrics.stream.Gauge:1.0.0 of event bundle with events 4
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.createEventList(ThriftEventConverter.java:188)
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.toEventList(ThriftEventConverter.java:90)
at org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:72)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.wso2.carbon.databridge.core.exception.EventConversionException: No StreamDefinition for streamId org.wso2.metrics.stream.Gauge:1.0.0 present in cache
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.createEventList(ThriftEventConverter.java:171)
... 7 more
This issue occurs when the particular stream, "org.wso2.metrics.stream.Gauge:1.0.0", hasn't deployed properly from the capp. Below steps can be followed to resolve this.
Remove the /tmp directory which is available in the APIM_ANALYTICS_HOME directory.
Restart the server.
If there is any issue observed while deploying the capp, please backup the capp and remove it from the carbon console. (You can find it under the fault apps section. This is to remove any cache data related to the capp).
Then follow steps 1 and 2 again.
The WSO2 Business Rule Manager fails when deploy. I'm using Docker to comunicate WSO2-dashboard and WSO2-worker.
The error logs shows the following:
ERROR {org.wso2.carbon.business.rules.core.services.TemplateManagerService} - Failed to update the deployed artifact for business rule myRule org.wso2.carbon.business.rules.core.exceptions.SiddhiAppsApiHelperException: Failed to update the siddhi app '#App:name('MyApp')
#App:description('MyDescription')
.
.
.
Siddi Template Code
.
.
.'
on node 'wso2sp-worker:9443' due to a validation error occurred when updating the siddhi app
at org.wso2.carbon.business.rules.core.deployer.SiddhiAppApiHelper.update(SiddhiAppApiHelper.java:139)
at org.wso2.carbon.business.rules.core.services.TemplateManagerService.updateDeployedSiddhiApp(TemplateManagerService.java:1400)
at org.wso2.carbon.business.rules.core.services.TemplateManagerService.updateDeployedArtifacts(TemplateManagerService.java:1388)
at org.wso2.carbon.business.rules.core.services.TemplateManagerService.redeployBusinessRule(TemplateManagerService.java:663)
at org.wso2.carbon.business.rules.core.api.impl.BusinessRulesApiServiceImpl.redeployBusinessRule(BusinessRulesApiServiceImpl.java:412)
at org.wso2.carbon.business.rules.core.api.BusinessRulesApi.redeployBusinessRule(BusinessRulesApi.java:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.wso2.msf4j.internal.router.HttpMethodInfo.invokeResource(HttpMethodInfo.java:187)
at org.wso2.msf4j.internal.router.HttpMethodInfo.invoke(HttpMethodInfo.java:143)
at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.dispatchMethod(MSF4JHttpConnectorListener.java:218)
at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.lambda$onMessage$57(MSF4JHttpConnectorListener.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
This can occur if the siddhi app created by the business rules manager is incorrect.
One possible reason for this is using an invalid siddhi app template to create business rules.
Therefore, can you check the following?
Create a siddhi app by filling up the templated fields in your templated siddhi app.
Copy that siddhi file to $SP_HOME/wso2/worker/deployment/siddhi-files directory.
Start the worker runtime.
If there is any issue with the template, it worker will fail to deploy that siddhi app and it will log the relevant error.
Test cluster of two brokers, WKA membership scheme, PostgreSQL message store, working fine for a couple of days, then throwing following errors:
TID: [] [] [2016-07-19 12:09:24,738] ERROR {org.wso2.andes.server.protocol.MultiVersionProtocolEngine} - Error establishing session {org.wso2.andes.server.protocol.MultiVersionProtocolEngine}
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.java:218)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcessor.java:198)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProcessor.java:45)
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProcessor.java:485)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Thread.java:745)
Startup of Message Broker looks fine, no errors, JDBC connection to PostgreSQL DB is ok, Registry mount looks ok. Then after that error appears in wso2carbon.log several times/minute.
Anyone any ideas? As far as I know nothing's changed and I don't know what it's trying to connect to.
This usually happens when client's whom connected to MB tries to create connections per message. jms is heavy connection and not recommended to create connections per each message. Therefore, please go through client implementation and verify connections are not created per message.
If by any chance you are using wso2 esb to publish/subscribe queues/topics to mb there is a property "transport.jms.CacheLevel" connection caching in esb axis2.xml.Read the documentation and use appropriate caching level for your usecase.
There was bug in connection caching property to be ignored in esb 4.8.1 which is currently fixed in 4.9.0 as well.
These are the possible cases I can think of with the given information. If you need more info please provide a detailed usecase.
While trying to follow the instructions from the wso2am (1.10.0) manual, regarding working with statistics with the wso2das (3.0.1) server i have encountered a problem.
If i choose to let the wso2am server define the stream while making the first call of the api, the wso2das server refuses to post statistics to the WSO2_STATS_DB.
If on the other hand i choose to import the analytics.car file in wso2das (as stated here ) i get an exception (AsyncDataPublisher Stream definition already exist) because the org.wso2.apimgt.statistics.request defined in the latest Analytics.car is different to the one being send from wso2am.
I pinpointed the problem in the definition of the Eventstream_request_1.0 in files
org.wso2.apimgt.statistics.request_1.0.0.json ,
throttledOutORG_WSO2_APIMGT_STATISTICS_REQUEST.xml
where the definition of the throttledOut option is missing
Is there a way to solve this issue?
Thank you.
I think your DAS is in some kind of a corrupted state. Can you first delete the car application (/repository/deployment/server/carbonapps) and then log in to DAS and go to Manage > Event > Streams and delete any existing streams. Then try again to deploy the car app in the /repository/deployment/server/carbonapps location.
If everything goes well you would see two scripts in Manage > Batch Analytics > Scripts section. Try to execute each script and see if there is any error. If not then you can point the API manager to DAS
I haven't been able to find much on the web about this problem but...
I am setting up a fresh BigTable cluster on Googles Cloud services. I've gone through the whole process you do with most Google APIs (create service account, know your project ID, authing with the gcloud tool, Google enviornment variable set, etc.).
I am having a problem though after going through the setup. I get this error I can't find anything on web on that says:
Caused by: com.google.bigtable.repackaged.com.google.common.util.concurrent.UncheckedExecutionException: io.grpc.StatusRuntimeException:
NOT_FOUND: Error listing tables for cluster projects/bigtable-1127/zones/us-central1-c/clusters/bigdatastats : Failed to read Tables in cluster: bigdatastats
Here is the complete print that includes the error..note that I get the same error when trying to create a table as well:
./bin/hbase com.google.cloud.bigtable.hbase.CheckConfig
User Agent: bigtable-hbase-1.0-0.2.1
Project ID: bigtable-1127
Cluster Id: bigdatastats
ZoneId: us-central1-c
Cluster admin host: bigtableclusteradmin.googleapis.com
Table admin host: bigtabletableadmin.googleapis.com
Data host: bigtable.googleapis.com
Attempting credential refresh...
HBase Connection Class = com.google.cloud.bigtable.hbase1_0.BigtableConnection (OK)
Opening table admin connection...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/Michael/bigtable/hbase-1.0.1.1/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.7.1/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2015-11-12 01:30:31,552 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-11-12 01:30:32,619 INFO [main] grpc.BigtableSession: Opening connection for projectId bigtable-1127, zoneId us-central1-c, clusterId bigdatastats, on data host bigtable.googleapis.com, table admin host bigtabletableadmin.googleapis.com.
Tables in cluster bigdatastats:
Exception in thread "main" java.io.IOException: Failed to listTables
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.requestTableList(AbstractBigtableAdmin.java:221)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.listTableNames(AbstractBigtableAdmin.java:208)
at com.google.cloud.bigtable.hbase.CheckConfig.main(CheckConfig.java:99)
Caused by: com.google.bigtable.repackaged.com.google.common.util.concurrent.UncheckedExecutionException: io.grpc.StatusRuntimeException: NOT_FOUND: Error listing tables for cluster projects/bigtable-1127/zones/us-central1-c/clusters/bigdatastats : Failed to read Tables in cluster: bigdatastats
at io.grpc.stub.Calls.getUnchecked(Calls.java:117)
at io.grpc.stub.Calls.blockingUnaryCall(Calls.java:129)
at com.google.bigtable.admin.table.v1.BigtableTableServiceGrpc$BigtableTableServiceBlockingStub.listTables(BigtableTableServiceGrpc.java:338)
at com.google.cloud.bigtable.grpc.BigtableTableAdminGrpcClient.listTables(BigtableTableAdminGrpcClient.java:44)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.requestTableList(AbstractBigtableAdmin.java:219)
... 2 more
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Error listing tables for cluster projects/bigtable-1127/zones/us-central1-c/clusters/bigdatastats : Failed to read Tables in cluster: bigdatastats
at io.grpc.Status.asRuntimeException(Status.java:428)
at io.grpc.stub.Calls$UnaryStreamToFuture.onClose(Calls.java:324)
at io.grpc.ChannelImpl$CallImpl$ClientStreamListenerImpl$3.run(ChannelImpl.java:402)
at io.grpc.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:154)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
It would be amazing if someone could help with this. I am not sure what to do and I can't find anything out there. Obviously its around authentication, my key file is fresh and in the right place, ive ran the gcloud auth and Im not sure what else to check.
Please let me know if I can provide anymore information to help answer.
As noted in the comments, this was unlikely to have been an authentication issue.
You would receive NOT_FOUND as an error if the resource you are trying to query does not exist in your project. So it's likely that you wanted to switch your default project using gcloud config set project as recommended by Les.