I am using Janusgraph 0.2.0 and Hbase with lucene as index backend. I am facing issue while Reindexing using MapReduceIndexManagement
Below is Error trace -
Error=java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0
at org.janusgraph.hadoop.MapReduceIndexManagement$FailedJobFuture.get(MapReduceIndexManagement.java:298)
at org.janusgraph.hadoop.MapReduceIndexManagement$FailedJobFuture.get(MapReduceIndexManagement.java:268)
at com.inn.foresight.core.generic.utils.CustomDataSource.addMapreduceReIndex(CustomDataSource.java:188)
at com.inn.foresight.core.generic.utils.CustomDataSource.main(CustomDataSource.java:163)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:354)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:159)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:211)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:799)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:193)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:324)
at org.apache.hadoop.hbase.client.HRegionLocator.getAllRegionLocations(HRegionLocator.java:90)
at org.apache.hadoop.hbase.util.RegionSizeCalculator.init(RegionSizeCalculator.java:94)
at org.apache.hadoop.hbase.util.RegionSizeCalculator.<init>(RegionSizeCalculator.java:81)
at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:256)
at org.apache.hadoop.hbase.mapreduce.TableInputFormat.getSplits(TableInputFormat.java:237)
at org.janusgraph.hadoop.formats.hbase.HBaseBinaryInputFormat.getSplits(HBaseBinaryInputFormat.java:58)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
at org.janusgraph.hadoop.scan.HadoopScanRunner.runJob(HadoopScanRunner.java:138)
at org.janusgraph.hadoop.MapReduceIndexManagement.updateIndex(MapReduceIndexManagement.java:187)
... 2 more
And My janusgraph properties are below
storage.backend = hbase
storage.hostname = localhost
storage.port = 2181
storage.hbase.ext.hbase.zookeeper.property.clientPort = 2181
storage.hbase.ext.zookeeper.znode.parent = /hbase-unsecure
query.fast-property = true
storage.hbase.table = Master9
storage.read-time = 200000
cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000
cache.db-cache-size = 0.5
index.search.backend=lucene
index.search.directory=/home/ist/jIndexFolder
and Here is my code
JGraphUtil jGraphUtil = new JGraphUtil();
JanusGraph jGraph = jGraphUtil.openGraphInstance();
JanusGraphManagement management;
JanusGraphTransaction newTransaction;
jGraphUtil.closeOtherInstances();
newTransaction = jGraph.newTransaction();
management = jGraph.openManagement();
JanusGraphIndex graphIndex = management.getGraphIndex("bySrcEName");
MapReduceIndexManagement mrim = new MapReduceIndexManagement(jGraph);
try {
mrim.updateIndex(graphIndex, SchemaAction.REINDEX).get();
} catch (InterruptedException | ExecutionException | BackendException e) {
System.out.println("Error=" + Utils.getStackTrace(e));
}
Please any one check and let me know how can I perform reindex in JanusGraph. Any example of reindexing using MapReduceIndexManagement will be appreciated
Related
I have google app engine project taht i finish upgrade from java 8 to java 11.
I also upgraded TaskQueque to cloudTask and every working but how i can run cloudTask on localHost?
this is my old code
Queue queue = QueueFactory.getDefaultQueue();
TaskOptions taskOptions = TaskOptions.Builder.withUrl("/" + backgroundTaskParams.getUrl());
taskOptions.retryOptions(RetryOptions.Builder.withTaskRetryLimit(backgroundTaskParams.getRetryLimit()));
taskOptions.countdownMillis(backgroundTaskParams.getStartDelay());
Map<String, String> params = backgroundTaskParams.getParams();
if(params != null){
for (String key : params.keySet()) {
taskOptions.param(key, params.get(key));
}
}
TaskHandle taskHandle = queue.add(taskOptions);
is not working on java 11 standard gcp
so I upgrade to this code
try (CloudTasksClient client = CloudTasksClient.create()) {
RetryConfig retyConfig = RetryConfig.builder()
.setMaxRetries(backgroundTaskParams.getRetryLimit())
.build();
String queueName = QueueName.of(EMF.getProjectId(), "us-central1", "default").toString();
String payload = Utilities.buildUrlParams(backgroundTaskParams.getParams());
long startTime = (ApplicationServicesManager.getInstance().getTimeManager().getCurrentTime() + backgroundTaskParams.getStartDelay()) / 1000;
Task.Builder taskBuilder = Task.newBuilder();
taskBuilder.setAppEngineHttpRequest(
AppEngineHttpRequest.newBuilder()
.setRelativeUri("/" + backgroundTaskParams.getUrl())
.setHttpMethod(HttpMethod.POST)
.setBody(ByteString.copyFrom(payload, Charset.defaultCharset()))
.build());
taskBuilder.setScheduleTime(
Timestamp.newBuilder()
.setSeconds(startTime));
taskBuilder.setDispatchCount(1);
Task task = taskBuilder.build();
Task taskResponse = client.createTask(queueName, task);
}
//
catch (Exception e) {
NotificationsManager.sendLogException(e);
}
the new code working fine in production server but when I run in localhost
the code create task in production server so i cannot debugging and is not good :)
so how I can to start task in localHost
thank you
while producing message to msk(kafka 2.1.0) I am getting
"Exception thrown when sending a message with key='null' and payload='Message->0' to topic AWSKafkaTopic"
I am trying to produce it from a springboot app deployed on EC2 using docker.
But the producer is working fine when I am trying to produce the message from same EC2 client using kafka-console-producer.sh.
bin/kafka-console-producer.sh --broker-list "XXBootstrapBrokerStringTlsXX" --producer.config client.properties --topic AWSKafkaTopic
I have tried the same program on my local with kafka 2.3.0 and zookeeper, it is working fine there(running springboot app on docker).
Config->
#Value("${spring.kafka.producer.bootstrap-servers}")
private String bootstrapServers;
#Bean
public Map<String, Object> producerConfigs() {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapServers); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,StringSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class);
return props;
}
#Bean
public ProducerFactory<String, String> producerFactory() {
return new DefaultKafkaProducerFactory<>(producerConfigs());
}
#Bean
public KafkaTemplate<String, String> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
#Bean
public Sender sender() {
return new Sender();
}
Client->
#Autowired
private KafkaTemplate<String,String> kafkaTemplate;
public void sendMessage(String message){
this.kafkaTemplate.send("AWSKafkaTopic",message);
}
Actual result->
ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [XXBootstrapBrokerStringTlsXX]
buffer.memory = 33554432
client.id =
compression.type = none
connections.max.idle.ms = 540000
enable.idempotence = false
interceptor.classes = []
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class
value.serializer = class org.apache.kafka.common.serialization.StringSerializer
Log:
2019-07-24 07:40:43.305 INFO 1 --- [nio-9000-exec-1] o.a.kafka.common.utils.AppInfoParser : Kafka version : 2.0.1
2019-07-24 07:40:43.305 INFO 1 --- [nio-9000-exec-1] o.a.kafka.common.utils.AppInfoParser : Kafka commitId : fa14705e51bd2ce5
2019-07-24 07:41:43.313 ERROR 1 --- [nio-9000-exec-1] o.s.k.support.LoggingProducerListener : Exception thrown when sending a message with key='null' and payload='Message->0' to topic AWSKafkaTopic:
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
In my case i tried to produce the message in new topic but auto.create.topics.enable was false in the aws broker so better to create a message on existing topic or auto.create.topics.enable set this property as true and try.
I am using OpenDaylight and trying to replace the default distributed database with Apache Ignite.
I am using the jar obtained by the source code here.
https://github.com/Romeh/akka-persistance-ignite
However, the class IgniteWriteJournal does not seem to load which i have checked by putting some print statements in its constuructor.
Is there any issue with the .conf file?
The following is a portion of the akka.conf file i am using in OpenDaylight.
odl-cluster-data {
akka {
remote {
artery {
enabled = off
canonical.hostname = "10.145.59.38"
canonical.port = 2550
}
netty.tcp {
hostname = "10.145.59.38"
port = 2550
}
# when under load we might trip a false positive on the failure detector
# transport-failure-detector {
# heartbeat-interval = 4 s
# acceptable-heartbeat-pause = 16s
# }
}
cluster {
# Remove ".tcp" when using artery.
seed-nodes = ["akka.tcp://opendaylight-cluster-data#10.145.59.38:2550"]
roles = ["member-1"]
}
extensions = ["akka.persistence.ignite.extension.IgniteExtensionProvider"]
akka.persistence.journal.plugin = "akka.persistence.journal.ignite"
akka.persistence.snapshot-store.plugin = "akka.persistence.snapshot.ignite"
persistence {
# Ignite journal plugin
journal {
ignite {
# Class name of the plugin
class = "akka.persistence.ignite.journal.IgniteWriteJournal"
cache-prefix = "akka-journal"
// Should be based into the the dara grid topology
cache-backups = 1
// if ignite is already started in a separate standalone grid where journal cache is already created
cachesAlreadyCreated = false
}
}
# Ignite snapshot plugin
snapshot {
ignite {
# Class name of the plugin
class = "akka.persistence.ignite.snapshot.IgniteSnapshotStore"
cache-prefix = "akka-snapshot"
// Should be based into the the dara grid topology
cache-backups = 1
// if ignite is already started in a separate standalone grid where snapshot cache is already created
cachesAlreadyCreated = false
}
}
}
}
ignite {
//to start client or server node to connect to Ignite data cluster
isClientNode = false
// for ONLY testing we use localhost
// used for grid cluster connectivity
tcpDiscoveryAddresses = "localhost"
metricsLogFrequency = 0
// thread pools used by Ignite , should based into target machine specs
queryThreadPoolSize = 4
dataStreamerThreadPoolSize = 1
managementThreadPoolSize = 2
publicThreadPoolSize = 4
systemThreadPoolSize = 2
rebalanceThreadPoolSize = 1
asyncCallbackPoolSize = 4
peerClassLoadingEnabled = false
// to enable or disable durable memory persistance
enableFilePersistence = true
// used for grid cluster connectivity, change it to suit your configuration
igniteConnectorPort = 11211
// used for grid cluster connectivity , change it to suit your configuration
igniteServerPortRange = "47500..47509"
//durable memory persistance storage file system path , change it to suit your configuration
ignitePersistenceFilePath = "./data"
}
}
I assume you modified the configuration/initial/akka.conf. First those sections need to be inside the odl-cluster-data section (can't tell from just your snippet). Also it looks like the following should be:
akka.persistence.journal.plugin = "akka.persistence.journal.ignite"
akka.persistence.snapshot-store.plugin = "akka.persistence.snapshot.ignite"
I'm trying to run an akka stream application, but I am getting an exception:
No configuration setting found for key 'akka.stream'
the relevant code snippet is:
ConfigFactory.load()
implicit val system = ActorSystem("svc")
implicit val mat = ActorMaterializer()
I try both command lines:
java -jar ./myService.jar -Dconfig.resource=/opt/myservice/conf/application.conf
java -jar ./myService.jar -Dconfig.file=/opt/myService/conf/application.conf
my application.conf file:
akka {
event-handlers = ["akka.event.slf4j.Slf4jEventHandler"]
loglevel = "DEBUG"
actor {
}
stream {
# Default materializer settings
materializer {
max-input-buffer-size = 16
dispatcher = ""
subscription-timeout {
mode = cancel
timeout = 5s
}
output-burst-limit = 1000
auto-fusing = on
max-fixed-buffer-size = 1000000000
sync-processing-limit = 1000
}
blocking-io-dispatcher = "akka.stream.default-blocking-io-dispatcher"
default-blocking-io-dispatcher {
type = "Dispatcher"
executor = "thread-pool-executor"
throughput = 1
thread-pool-executor {
fixed-pool-size = 16
}
}
}
}
exception details:
No configuration setting found for key 'akka.stream'
at
com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:152)
at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:145)
at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:172)
at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:184)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:189)
at com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:258)
at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:264)
at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:37)
at akka.stream.ActorMaterializerSettings$.apply(ActorMaterializer.scala:248)
at akka.stream.ActorMaterializer$$anonfun$1.apply(ActorMaterializer.scala:41)
at akka.stream.ActorMaterializer$$anonfun$1.apply(ActorMaterializer.scala:41)
at scala.Option.getOrElse(Option.scala:121)
at akka.stream.ActorMaterializer$.apply(ActorMaterializer.scala:41)
at com.Listener$.main(Listener.scala:41)
at com.Listener.main(Listener.scala)
can you assist?
thanks
To load config from a file, you should use:
-Dconfig.file=/opt/myService/conf/application.conf
Doc link: https://github.com/typesafehub/config#standard-behavior
I'm trying the following code
UserDetail ud = UserDetail.SingleOrDefault(u => u.UserName == CurrentUserName);
if (ud == null)
ud = new UserDetail();
Address uAddress = ud.AddressId.HasValue
? Address.SingleOrNew(a => a.Id == ud.AddressId)
: new Address();
using (TransactionScope tc = new TransactionScope())
{
uAddress.Save();
ud.AddressId = uAddress.Id;
ud.Save(); // error is here
tc.Complete();
}
when i reach ud.save() i get the error 'The operation is not valid for the state of the transaction. ---> System.Transactions.TransactionPromotionException: Failure while attempting to promote transaction'
if i comment out the transaction part it works fine, isn't .SingleOrDefault disconnecting from the db ?
thanks
You need to wrap your TransactionScope in a SharedDbConnectionScope, see here for details. The following should work for your example
using (SharedDbConnectionScope sharedConnectionScope = new SharedDbConnectionScope()){
{
using (TransactionScope tc = new TransactionScope())
{
uAddress.Save();
ud.AddressId = uAddress.Id;
ud.Save(); // error is here
tc.Complete();
}
}
it's a bug with subsonic 3.0.0.3
the fix can be found here issue 69