I use KafkaEmbedded for my consumer unit tests, where I have two create two different producers with two different serialisers, and two different topics:
// first producer factory
Map<String, Object> firstProducerProperties = KafkaTestUtils.producerProps(kafkaEmbedded);
firstProducerProperties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
firstProducerProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
firstProducerProperties.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, properties.getKafka().getSchemaRegistryUrl());
ProducerFactory<String, TicketChange> firstProducerFactory =
new DefaultKafkaProducerFactory<>(firstProducerProperties);
firstTemplate = new KafkaTemplate<>(firstProducerFactory);
firstTemplate.setDefaultTopic("topic-one");
...
// second producer factory
Map<String, Object> secondProducerProperties = KafkaTestUtils.producerProps(kafkaEmbedded);
secondProducerProperties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
secondProducerProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
ProducerFactory<String, TicketChange> secondProducerFactory =
new DefaultKafkaProducerFactory<>(secondProducerProperties);
secondTemplate = new KafkaTemplate<>(secondProducerFactory);
secondTemplate.setDefaultTopic("topic-two")
Then, as described here, I assign partitions to producers:
for (MessageListenerContainer messageListenerContainer : kafkaListenerEndpointRegistry
.getListenerContainers()) {
ContainerTestUtils.waitForAssignment(messageListenerContainer,
kafkaEmbedded.getPartitionsPerTopic() * 2);
}
The last line the number of partitions is multiplied by the number of topics. If I run the tests I get the following error:
org.junit.ComparisonFailure:
Expected :2
Actual :1
<Click to see difference>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at org.springframework.kafka.test.utils.ContainerTestUtils.waitForAssignment(ContainerTestUtils.java:74)
at com.ks.cfc.service.kafka.KafkaEmbeddedBase.setUp(KafkaEmbeddedBase.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
It is like I have to add somehow additional listeners? message handlers for each of the producers?
Related
I am following the Corda tutorial to build Corda app. Instead of building it in Kotlin, I built it in Java.
I followed the steps provided in the doc and below is my CarIssueInitiator class.
package com.template.flows;
import co.paralleluniverse.fibers.Suspendable;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import com.template.contracts.CarContract;
import com.template.states.CarState;
import net.corda.core.contracts.Command;
import net.corda.core.contracts.UniqueIdentifier;
import net.corda.core.flows.*;
import net.corda.core.identity.Party;
import net.corda.core.transactions.SignedTransaction;
import net.corda.core.transactions.TransactionBuilder;
import net.corda.core.utilities.ProgressTracker;
import net.corda.core.utilities.ProgressTracker.Step;
import java.util.List;
import java.util.stream.Collectors;
// ******************
// * Initiator flow *
// ******************
#InitiatingFlow
#StartableByRPC
public class CarIssueInitiator extends FlowLogic<SignedTransaction> {
private final Party owningBank;
private final Party holdingDealer;
private final Party manufacturer;
private final String vin;
private final String licensePlateNumber;
private final String make;
private final String model;
private final String dealershipLocation;
private final Step GENERATING_TRANSACTION = new Step("Generating transaction based on new IOU.");
private final Step VERIFYING_TRANSACTION = new Step("Verifying contract constraints.");
private final Step SIGNING_TRANSACTION = new Step("Signing transaction with our private key.");
private final Step GATHERING_SIGS = new Step("Gathering the counterparty's signature.") {
#Override
public ProgressTracker childProgressTracker() {
return CollectSignaturesFlow.Companion.tracker();
}
};
private final Step FINALISING_TRANSACTION = new Step("Obtaining notary signature and recording transaction.") {
#Override
public ProgressTracker childProgressTracker() {
return FinalityFlow.Companion.tracker();
}
};
private final ProgressTracker progressTracker = new ProgressTracker(
GENERATING_TRANSACTION,
VERIFYING_TRANSACTION,
SIGNING_TRANSACTION,
GATHERING_SIGS,
FINALISING_TRANSACTION
);
public CarIssueInitiator(Party owningBank,
Party holdingDealer,
Party manufacturer,
String vin,
String licensePlateNumber,
String make,
String model,
String dealershipLocation){
this.owningBank = owningBank;
this.holdingDealer = holdingDealer;
this.manufacturer = manufacturer;
this.vin = vin;
this.licensePlateNumber = licensePlateNumber;
this.make = make;
this.model = model;
this.dealershipLocation = dealershipLocation;
}
#Override
public ProgressTracker getProgressTracker() {
return progressTracker;
}
#Suspendable
#Override
public SignedTransaction call() throws FlowException {
// Initiator flow logic goes here.
final Party notary = getServiceHub().getNetworkMapCache().getNotaryIdentities().get(0);
// Stage 1.
progressTracker.setCurrentStep(GENERATING_TRANSACTION);
// Generate an unsigned transaction.
Party me = getOurIdentity();
CarState carState = new CarState(this.owningBank,
this.holdingDealer,
this.manufacturer,
this.vin,
this.licensePlateNumber,
this.make,
this.model,
this.dealershipLocation,
new UniqueIdentifier());
final Command<CarContract.Commands.Issue> txCommand = new Command<CarContract.Commands.Issue>(new CarContract.Commands.Issue(),
ImmutableList.of(carState.getOwningBank().getOwningKey(), carState.getHoldingDealer().getOwningKey(), carState.getManufacturer().getOwningKey()));
final TransactionBuilder txBuilder = new TransactionBuilder(notary)
.addOutputState(carState, CarContract.ID)
.addCommand(txCommand);
// Stage 2.
progressTracker.setCurrentStep(VERIFYING_TRANSACTION);
// Verify that the transaction is valid.
txBuilder.verify(getServiceHub());
//Stage 3
progressTracker.setCurrentStep(SIGNING_TRANSACTION);
// Sign the transaction.
final SignedTransaction partSignedTx = getServiceHub().signInitialTransaction(txBuilder);
// Stage 4.
progressTracker.setCurrentStep(GATHERING_SIGS);
// Send the state to the counterparty, and receive it back with their signature.
List<FlowSession> sessions = carState.getParticipants().stream().map(a -> initiateFlow((Destination) a)).collect(Collectors.toList());
//FlowSession session = initiateFlow(me);
final SignedTransaction fullySignedTx = subFlow(
new CollectSignaturesFlow(partSignedTx, sessions, ImmutableList.of(me.getOwningKey()), CollectSignaturesFlow.Companion.tracker()));
// Stage 5.
progressTracker.setCurrentStep(FINALISING_TRANSACTION);
return subFlow(new FinalityFlow(fullySignedTx, sessions));
}
}
I deployed the CordApp and ran the flow. But I ended up with below error
java.lang.IllegalArgumentException: The Initiator of CollectSignaturesFlow must pass in exactly the sessions required to sign the transaction.
at net.corda.core.flows.CollectSignaturesFlow.call(CollectSignaturesFlow.kt:164) ~[corda-core-4.3.jar:?]
at net.corda.core.flows.CollectSignaturesFlow.call(CollectSignaturesFlow.kt:67) ~[corda-core-4.3.jar:?]
at net.corda.node.services.statemachine.FlowStateMachineImpl.subFlow(FlowStateMachineImpl.kt:330) ~[corda-node-4.3.jar:?]
at net.corda.core.flows.FlowLogic.subFlow(FlowLogic.kt:326) ~[corda-core-4.3.jar:?]
at com.template.flows.CarIssueInitiator.call(CarIssueInitiator.java:129) ~[?:?]
at com.template.flows.CarIssueInitiator.call(CarIssueInitiator.java:23) ~[?:?]
at net.corda.node.services.statemachine.FlowStateMachineImpl.run(FlowStateMachineImpl.kt:270) ~[corda-node-4.3.jar:?]
at net.corda.node.services.statemachine.FlowStateMachineImpl.run(FlowStateMachineImpl.kt:46) ~[corda-node-4.3.jar:?]
at co.paralleluniverse.fibers.Fiber.run1(Fiber.java:1092) ~[quasar-core-0.7.10-jdk8.jar:0.7.10]
at co.paralleluniverse.fibers.Fiber.exec(Fiber.java:788) ~[quasar-core-0.7.10-jdk8.jar:0.7.10]
at co.paralleluniverse.fibers.RunnableFiberTask.doExec(RunnableFiberTask.java:100) ~[quasar-core-0.7.10-jdk8.jar:0.7.10]
at co.paralleluniverse.fibers.RunnableFiberTask.run(RunnableFiberTask.java:91) ~[quasar-core-0.7.10-jdk8.jar:0.7.10]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_192]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_192]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) ~[?:1.8.0_192]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:1.8.0_192]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:1.8.0_192]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:1.8.0_192]
at net.corda.node.utilities.AffinityExecutor$ServiceAffinityExecutor$1$thread$1.run(AffinityExecutor.kt:63) ~[corda-node-4.3.jar:?]
I am not getting what is wrong with the code. Request you to please help me out with this.
UPDATE: With below updated code, it works for me.
//Stage 3
progressTracker.setCurrentStep(SIGNING_TRANSACTION);
// Sign the transaction.
final PublicKey ourSigningKey = getServiceHub().getMyInfo().getLegalIdentities().get(0).getOwningKey();
final SignedTransaction partSignedTx = getServiceHub().signInitialTransaction(txBuilder, ourSigningKey);
// Stage 4.
progressTracker.setCurrentStep(GATHERING_SIGS);
// Send the state to the counterparty, and receive it back with their signature.
FlowSession HoldingDealerPartyFlow = initiateFlow(carState.getHoldingDealer());
FlowSession ManufacturerPartyFlow = initiateFlow(carState.getManufacturer());
final SignedTransaction fullySignedTx = subFlow(new CollectSignaturesFlow(
partSignedTx,
ImmutableSet.of(HoldingDealerPartyFlow, ManufacturerPartyFlow),
ImmutableList.of(ourSigningKey))
);
// Stage 5.
progressTracker.setCurrentStep(FINALISING_TRANSACTION);
return subFlow(new FinalityFlow(fullySignedTx, ImmutableSet.of(HoldingDealerPartyFlow, ManufacturerPartyFlow)));
The list of signers in carState.getParticipants()(see Code A) does not match with the list of signers in ImmutableList of CollectSignaturesFlow (see Code B).
Code A:
List<FlowSession> sessions = carState.getParticipants().stream().map(a -> initiateFlow((Destination) a)).collect(Collectors.toList());
Code B: final SignedTransaction fullySignedTx = subFlow(
new CollectSignaturesFlow(partSignedTx, sessions, ImmutableList.of(me.getOwningKey()), CollectSignaturesFlow.Companion.tracker()));
In the ImmutableList you have to add carState.getOwningBank().getOwningKey(), carState.getHoldingDealer().getOwningKey(), carState.getManufacturer().getOwningKey()
EDITED
Try the following code from "Stage 3" onwards
//Stage 3
progressTracker.setCurrentStep(SIGNING_TRANSACTION);
// Sign the transaction.
//final SignedTransaction partSignedTx = getServiceHub().signInitialTransaction(txBuilder);
val ourSigningKey = serviceHub.myInfo.legalIdentities.first().owningKey
val partSignedTx = serviceHub.signInitialTransaction(tx, ourSigningKey)
// Stage 4.
progressTracker.setCurrentStep(GATHERING_SIGS);
// Send the state to the counterparty, and receive it back with their signature.
//List<FlowSession> sessions = carState.getParticipants().stream().map(a -> initiateFlow((Destination) a)).collect(Collectors.toList());
//FlowSession session = initiateFlow(me);
//final SignedTransaction fullySignedTx = subFlow(
// new CollectSignaturesFlow(partSignedTx, sessions, ImmutableList.of(me.getOwningKey()), CollectSignaturesFlow.Companion.tracker()));
val HoldingDealerPartyFlow = initiateFlow(carState.getHoldingDealer())
val ManufacturerPartyFlow = initiateFlow(carState.getManufacturer())
val fullySignedTx = subFlow(CollectSignaturesFlow(
partSignedTx,
setOf(HoldingDealerPartyFlow ManufacturerPartyFlow),
listOf(ourSigningKey))
)
// Stage 5.
progressTracker.setCurrentStep(FINALISING_TRANSACTION);
//return subFlow(new FinalityFlow(fullySignedTx, sessions));
subFlow(FinalityFlow(fullySignedTx, setOf(HoldingDealerPartyFlow ManufacturerPartyFlow)))
return fullySignedTx.tx.outRef<State>(0)
You should only call initiateFlow() for all participants except for yourself, so your code could be like (the grammar might be wrong, but shows the point):
List<FlowSession> sessions = (carState.getParticipants() - me).stream().map(a -> initiateFlow((Destination) a)).collect(Collectors.toList());
I made a unit test using KafkaEmbedded (and KafkaTemplate), but the message order is random. Does anyone know if it is logical, and if it is possible guaranty order?
here is my code:
public class KafkaTest {
private static String TOPIC = "test.topic";
#ClassRule
public static KafkaEmbedded embeddedKafka = new KafkaEmbedded(1, true, TOPIC);
#Test
public void testEmbeddedKafkaSendOrder() throws Exception {
Map<String, Object> producerConfig = new HashMap<>();
producerConfig.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, embeddedKafka.getBrokersAsString());
producerConfig.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
producerConfig.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class);
KafkaTemplate<String, byte[]> kafkaTemplate = new KafkaTemplate<>(new DefaultKafkaProducerFactory<>(producerConfig));
kafkaTemplate.send(TOPIC, "TEST1".getBytes()).get();
kafkaTemplate.send(TOPIC, "TEST2".getBytes()).get();
kafkaTemplate.send(TOPIC, "TEST3".getBytes()).get();
kafkaTemplate.send(TOPIC, "TEST4".getBytes()).get();
kafkaTemplate.send(TOPIC, "TEST5".getBytes()).get();
Map<String, Object> consumerConfig = new HashMap<>();
consumerConfig.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, embeddedKafka.getBrokersAsString());
consumerConfig.put(ConsumerConfig.GROUP_ID_CONFIG, "consumer-test-group");
consumerConfig.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
consumerConfig.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class);
consumerConfig.put("auto.offset.reset", "earliest");
final Consumer<String, byte[]> consumer = new KafkaConsumer<>(consumerConfig);
embeddedKafka.consumeFromAnEmbeddedTopic(consumer, TOPIC);
ConsumerRecords<String, byte[]> records = consumer.poll(100L);
// Tests
final Iterator<ConsumerRecord<String, byte[]>> recordIterator = records.iterator();
while (recordIterator.hasNext()) {
System.out.println("received:" + new String(recordIterator.next().value()));
}
}
This code prints for example (but the order can change):
received:TEST2
received:TEST4
received:TEST1
received:TEST3
received:TEST5
In Kafka, you can be sure that order of messages is the same on the same partition, but not on the topic.
Note that as a topic typically has multiple partitions, there is
no guarantee of message time-ordering across the entire topic, just within a single
partition
Quote from the book Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale .
What you can do about this and how to receive messages in order?
Option 1:
kafkaTemplate.send(TOPIC,"1", "TEST1".getBytes()).get();
kafkaTemplate.send(TOPIC,"1", "TEST2".getBytes()).get();
kafkaTemplate.send(TOPIC,"1", "TEST3".getBytes()).get();
kafkaTemplate.send(TOPIC,"1", "TEST4".getBytes()).get();
kafkaTemplate.send(TOPIC,"1", "TEST5".getBytes()).get();
This way, for every value, you send the same key "1". Kafka will choose partition based on your key. Since all keys are equal, all messages will go to the same partition and you will receive your records in order.
Option 2:
Initialize KafkaEmbedded this way:
new KafkaEmbedded(1, true,1, TOPIC);
This way you are telling kafka that for this topic you would like to have only one partition so every record will go to that partition.
I'm building up an Instances object, adding Attributes, and then adding data in the form of Instance objects.
When I go to write it out, the toString() method is already throwing an OutOfBoundsException and unable to evaluate the data in the Instances. I receive the error when I try to print out the data and I can see the exception being thrown just in the Debugger as it shows it can't evaluate the toString() for the data object.
The only clue I have is that the error message seems to be using the first data element (StudentId) and using it as an index. I'm confused as to why.
The code:
// Set up the attributes for the Weka data model
ArrayList<Attribute> attributes = new ArrayList<>();
attributes.add(new Attribute("StudentIdentifier", true));
attributes.add(new Attribute("CourseGrade", true));
attributes.add(new Attribute("CourseIdentifier"));
attributes.add(new Attribute("Term", true));
attributes.add(new Attribute("YearCourseTaken", true));
// Create the data model object - I'm not happy that capacity is required and fixed? But that's another issue
Instances dataSet = new Instances("Records", attributes, 500);
// Set the attribute that will be used for prediction purposes - that will be CourseIdentifier
dataSet.setClassIndex(2);
// Pull back all the records in this term range, create Weka Instance objects for each and add to the data set
List<Record> records = recordsInTermRangeFindService.find(0, 10);
int count = 0;
for (Record r : records) {
Instance i = new DenseInstance(attributes.size());
i.setValue(attributes.get(0), r.studentIdentifier);
i.setValue(attributes.get(1), r.courseGrade);
i.setValue(attributes.get(2), r.courseIdentifier);
i.setValue(attributes.get(3), r.term);
i.setValue(attributes.get(4), r.yearCourseTaken);
dataSet.add(i);
}
System.out.println(dataSet.size());
BufferedWriter writer = null;
try {
writer = new BufferedWriter(new FileWriter("./test.arff"));
writer.write(dataSet.toString());
writer.flush();
writer.close();
} catch (IOException e) {
e.printStackTrace();
}
The error message:
Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 1010, Size: 0
I figured it out finally. I was setting the Attributes to strings with the 'true' second parameter in the Constructors, but they were integers coming out of the database table. I needed to change my lines to convert the integers to strings:
i.setValue(attributes.get(0), Integer.toString(r.studentIdentifier));
However, that created a different set of issues for me as things like the Apriori algorithm don't work on strings! I'm continuing to plug along learning Weka.
I have been using org.elasticsearch.test.ElasticsearchIntegrationTesting class for testing elasticsearch functionality.
I Use below code to import data into Elasticsearch and to search that data :**
#Test
public void searchData() throws Exception{
logger.info("searchData : searching data from Elasticsearch");
try {
csvImporter.client = client();
csvImporter.importData(); // In the importData method after data is imported we close the client
//In the below line getting Exception and it is inconsistent. Some times works fine
SearchResponse searchResponse = csvImporter.client.prepareSearch().setQuery(matchAllQuery()).execute().actionGet();
for (SearchHit searchHit : searchResponse.getHits()){
logger.info("Source Data >>>>>>>>>>>>>>>>>>>>> " + searchHit.sourceAsString());
}
} finally {
csvImporter.client.close();
}
}
Exception:
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at __randomizedtesting.SeedInfo.seed([86EB1B4AAE1739A5:99315B8D3D924DA1]:0)
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:305)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:200)
at org.elasticsearch.client.transport.support.InternalTransportIndicesAdminClient.execute(InternalTransportIndicesAdminClient.java:86)
at org.elasticsearch.client.support.AbstractIndicesAdminClient.deleteTemplate(AbstractIndicesAdminClient.java:687)
at org.elasticsearch.action.admin.indices.template.delete.DeleteIndexTemplateRequestBuilder.doExecute(DeleteIndexTemplateRequestBuilder.java:40)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
at org.elasticsearch.test.TestCluster.wipeTemplates(TestCluster.java:177)
at org.elasticsearch.test.TestCluster.wipe(TestCluster.java:76)
at org.elasticsearch.test.ElasticsearchIntegrationTest.beforeInternal(ElasticsearchIntegrationTest.java:292)
at org.elasticsearch.test.ElasticsearchIntegrationTest.before(ElasticsearchIntegrationTest.java:1946)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1665)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:898)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:914)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:59)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:873)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:775)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:809)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:820)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:59)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Since it is a Unit Testing I have not used configuration Settings to create client.
is there a way to recreate the client Object with same configuration settings??
I'm trying to implement custom pig storer that based on org.apache.pig.builtin.PigStorage . In my storer I want to calculate the real hdfs location from the string that sent at 'INTO' word and other job properties.
-- pig
STORE data INTO 'MyObject' using ...
I extend PigStorage and override setStoreLocation function, like this:
public void setStoreLocation(String location, Job job) throws IOException {
// location is 'MyObject' here
// my manipulations on location
String newLocation = basePath + '/' + someVar + '/' + location;
super.setStoreLocation(newLocation, job);
}
It works and write the file into the newLocation (/basePath/someVar/MyObject) , but I'm getting following message in the log, how I can to avoid it?
java.io.FileNotFoundException: File hdfs://myMachine:8020/user/hdfs/MyObject does not exist.
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:654)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.FileBasedOutputSizeReader.getOutputSize(FileBasedOutputSizeReader.java:65)
at org.apache.pig.tools.pigstats.JobStats.getOutputSize(JobStats.java:543)
at org.apache.pig.tools.pigstats.JobStats.addOneOutputStats(JobStats.java:567)
at org.apache.pig.tools.pigstats.JobStats.addOutputStatistics(JobStats.java:516)
at org.apache.pig.tools.pigstats.PigStatsUtil.addSuccessJobStats(PigStatsUtil.java:360)
at org.apache.pig.tools.pigstats.PigStatsUtil.accumulateStats(PigStatsUtil.java:257)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:341)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
at org.apache.pig.PigServer.execute(PigServer.java:1297)
at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
at org.apache.pig.PigServer.executeBatch(PigServer.java:353)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:478)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:286)
at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:226)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Looks like overriding of org.apache.pig.StoreFuncInterface#relToAbsPathForStoreLocation solves the problem:
#Override
public String relToAbsPathForStoreLocation(String location, Path curDir)
throws IOException {
return location;
}