Project Reactor: wait while broadcaster finish - unit-testing

There is a Broadcaster, that accepts strings and append them to a StringBuilder.
I want to test it.
I have to use Thread#sleep to wait, while the broadcaster finish processing of strings. I want to remove sleep.
I tried to use Control#debug() unsuccessfully.
public class BroadcasterUnitTest {
#Test
public void test() {
//prepare
Environment.initialize();
Broadcaster<String> sink = Broadcaster.create(Environment.newDispatcher()); //run broadcaster in separate thread (dispatcher)
StringBuilder sb = new StringBuilder();
sink
.observe(s -> sleep(100)) //long-time operation
.consume(sb::append);
//do
sink.onNext("a");
sink.onNext("b");
//assert
sleep(500);//wait while broadcaster finished (if comment this line then the test will fail)
assertEquals("ab", sb.toString());
}
private void sleep(int millis) {
try {
Thread.sleep(millis);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
}

I'm not familiar with Broadcaster (and it's probably deprecated since the question is old), but these 3 ways could be helpful in general:
When testing Project-Reactor's Fluxes and stuff, you're probably better of using their testing library made specially for this. Their reference and the Javadoc on that part are pretty good, and I'll just copy an example that speaks for itself here:
#Test
public void testAppendBoomError() {
Flux<String> source = Flux.just("foo", "bar");
StepVerifier.create(
appendBoomError(source))
.expectNext("foo")
.expectNext("bar")
.expectErrorMessage("boom")
.verify();
}
You could just block() by yourself on the Fluxes and Monos and then run checks. And note that if an error is emitted, this will result in an exception. But have a feeling you'll find yourself needing to write more code for some cases (e.g., checking the Flux has emitted 2 items X & Y then terminated with error) and you'd be then re-implementing StepVerifier.
#Test
public void testFluxOrMono() {
Flux<String> source = Flux.just(2, 3);
List<Integer> result = source
.flatMap(i -> multiplyBy2Async(i))
.collectList()
.block();
// run your asserts on the list. Reminder: the order may not be what you expect because of the `flatMap`
// Or with a Mono:
Integer resultOfMono = Mono.just(5)
.flatMap(i -> multiplyBy2Async(i))
.map(i -> i * 4)
.block();
// run your asserts on the integer
}
You could use the general solutions to async testing like CountDownLatch, but, again, wouldn't recommend and would give you trouble in some cases. For example, if you don't know the number of receivers in advance you'll need to use something else.

Per answer above, I found blockLast() helped.
#Test
public void MyTest()
{
Logs.Info("Start test");
/* 1 */
// Make a request
WebRequest wr1 = new WebRequest("1", "2", "3", "4");
String json1 = wr1.toJson(wr1);
Logs.Info("Flux");
Flux<String> responses = controller.getResponses(json1);
/* 2 */
Logs.Info("Responses in");
responses.subscribe(s -> mySub.myMethod(s)); // Test for strings is in myMethod
Logs.Info("Test thread sleeping");
Thread.sleep(2000);
/* 3 */
Logs.Info("Test thread blocking");
responses.blockLast();
Logs.Info("Finish test");
}

Related

BulkProcessor .add() not finishing when number of bulks > concurrentRequests

Here is a sample of the code flow:
Trigger the process with an API specifying bulkSize and totalRecords.
Use those parameters to acquire data from DB
Create a processor with the bulkSize.
Send both the data and processor into a method which:
-iterates over the resultset, assembles a JSON for each result, calls a method if the final JSON is not empty and adds that final JSON to the process using processor.add() method.
This is where the outcome of the code is split
After this, if the concurrentRequest parameter is 0 or 1 or any value < (totalRecords/bulkSize), the processor.add() line is where the code stalls and never continues to the next debug line.
However, when we increase the concurrentRequest parameter to a value > (totalRecords/bulkSize), the code is able to finish the .add() function and move onto the next line.
My reasoning leads me to believe we might be having issues with our BulkProcessListener which is making the .add() no close or finish like it is supposed to. I would really appreciate some more insight about this topic!
Here is the Listener we are using:
private class BulkProcessorListener implements Listener {
#Override
public void beforeBulk(long executionId, BulkRequest request) {
// Some log statements
}
#Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
// More log statements
}
#Override
public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
// Log statements
}
}
Here is the createProcessor():
public synchronized BulkProcessor createProcessor(int bulkActions) {
Builder builder = BulkProcessor.builder((request, bulkListener) -> {
long timeoutMin = 60L;
try {
request.timeout(TimeValue.timeValueMinutes(timeoutMin));
// Log statements
client.bulkAsync(request, RequestOptions.DEFAULT,new ResponseActionListener<BulkResponse>());
}catch(Exception ex) {
ex.printStackTrace();
}finally {
}
}, new BulkProcessorListener());
builder.setBulkActions(bulkActions);
builder.setBulkSize(new ByteSizeValue(buldSize, ByteSizeUnit.MB));
builder.setFlushInterval(TimeValue.timeValueSeconds(5));
builder.setConcurrentRequests(0);
builder.setBackoffPolicy(BackoffPolicy.noBackoff());
return builder.build();
}
Here is the method where we call processor.add():
#SuppressWarnings("deprecation")
private void addData(BulkProcessor processor, String indexName, JSONObject finalDataJSON, Map<String, String> previousUniqueObject) {
// Debug logs
processor.add(new IndexRequest(indexName, INDEX_TYPE,
previousUniqueObject.get(COMBINED_ID)).source(finalDataJSON.toString(), XContentType.JSON));
// Debug logs
}

Unit testing suspend coroutine

a bit new to Kotlin and testing it... I am trying to test a dao object wrapper with using a suspend method which uses an awaitFirst() for an SQL return object. However, when I wrote the unit test for it, it is just stuck in a loop. And I would think it is due to the awaitFirst() is not in the same scope of the testing
Implementation:
suspend fun queryExecution(querySpec: DatabaseClient.GenericExecuteSpec): OrderDomain {
var result: Map<String, Any>?
try {
result = querySpec.fetch().first().awaitFirst()
} catch (e: Exception) {
if (e is DataAccessResourceFailureException)
throw CommunicationException(
"Cannot connect to " + DatabaseConstants.DB_NAME +
DatabaseConstants.ORDERS_TABLE + " when executing querySelect",
"querySelect",
e
)
throw InternalException("Encountered R2dbcException when executing SQL querySelect", e)
}
if (result == null)
throw ResourceNotFoundException("Resource not found in Aurora DB")
try {
return OrderDomain(result)
} catch (e: Exception) {
throw InternalException("Exception when parsing to OrderDomain entity", e)
} finally {
logger.info("querySelect;stage=end")
}
}
Unit Test:
#Test
fun `get by orderid id, null`() = runBlocking {
// Assign
Mockito.`when`(fetchSpecMock.first()).thenReturn(monoMapMock)
Mockito.`when`(monoMapMock.awaitFirst()).thenReturn(null)
// Act & Assert
val exception = assertThrows<ResourceNotFoundException> {
auroraClientWrapper.queryExecution(
databaseClient.sql("SELECT * FROM orderTable WHERE orderId=:1").bind("1", "123") orderId
)
}
assertEquals("Resource not found in Aurora DB", exception.message)
}
I noticed this issue on https://github.com/Kotlin/kotlinx.coroutines/issues/1204 but none of the work around has worked for me...
Using runBlocking within Unit Test just causes my tests to never complete. Using runBlockingTest explicitly throws an error saying "Job never completed"... Anyone has any idea? Any hack at this point?
Also I fairly understand the point of you should not be using suspend with a block because that kinda defeats the purposes of suspend since it is releasing the thread to continue later versus blocking forces the thread to wait for a result... But then how does this work?
private suspend fun queryExecution(querySpec: DatabaseClient.GenericExecuteSpec): Map {
var result: Map<String, Any>?
try {
result = withContext(Dispatchers.Default) {
querySpec.fetch().first().block()
}
return result
}
Does this mean withContext will utilize a new thread, and re-use the old thread elsewhere? Which then doesnt really optimize anything since I will still have one thread that is being blocked regardless of spawning a new context?
Found the solution.
The monoMapMock is a mock value from Mockito. Seems like the kotlinx-test coroutines can't intercept an async to return a mono. So I forced the method that I can mock, to return a real Mono value instead of a Mocked Mono. To do so, as suggested by Louis. I stop mocking it and return a real value
#Test
fun `get by orderid id, null`() = runBlocking {
// Assign
Mockito.`when`(fetchSpecMock.first()).thenReturn(Mono.empty())
Mockito.`when`(monoMapMock.awaitFirst()).thenReturn(null)
// Act & Assert
val exception = assertThrows<ResourceNotFoundException> {
auroraClientWrapper.queryExecution(
databaseClient.sql("SELECT * FROM orderTable WHERE orderId=:1").bind("1", "123") orderId
)
}
assertEquals("Resource not found in Aurora DB", exception.message)
}

How to define testable timer loop in kotlin (android)?

I want to have a periodic timer loop (e.g. 1 second intervals). There are many ways to do that, but I haven't found a solution that would be suitable for unit testing.
Timer should be precise
Unit test should be able to skip the waiting
The closest that I came to a solution was to use coroutines: A simple loop with delay, runBlockingTest and advanceTimeBy.
coScope.launch {
while (isActive) {
// do stuff
delay(1000L)
}
}
and
#Test
fun timer_test() = coScope.runBlockingTest {
... // start job
advanceTimeBy(9_000L)
... // cancel job
}
It works to some degree, but the timer is not precise as it does not account for the execution time.
I haven't found a way to query internal timer used in a coroutine scope or a remaining timeout value inside withTimeoutOrNull:
coScope.launch {
withTimeoutOrNull(999_000_000L) { // max allowed looping time
while (isActive) {
// do stuff
val timeoutLeft // How to get that value ???
delay(timeoutLeft.mod(1000L))
}
}
}
Next idea was to use ticker:
coScope.launch {
val tickerChannel = ticker(1000L, 0L, coroutineContext)
var referenceTimer = 0L
for (event in tickerChannel) {
// do stuff
println(referenceTimer)
referenceTimer += 1000L
}
}
However, the connection between TestCoroutineDispatcher() and ticker does not produce right results:
private val coDispatcher = TestCoroutineDispatcher()
#Test timerTest() = runBlockingTest(coDispatcher) {
myTimer.lauchPeriodicJob()
advanceTimeBy(20_000L) // or delay(20_000L)
myTimer.cancelPeriodicJob()
println("end_of_test")
}
rather consistently results in:
0
1000
2000
3000
4000
5000
6000
end_of_test
I am also open for any alternative approaches that satisfy the two points above.

Graceful termination

I am trying to implement the following use case as part of my akka learning
I would like to calculate the total streets in all cities of all states. I have a database that contain the details needed. Here is what i have so far
Configuration
akka.actor.deployment {
/CityActor{
router = random-pool
nr-of-instances = 10
}
/StateActor {
router = random-pool
nr-of-instances = 1
}}
Main
public static void main(String[] args) {
try {
Config conf = ConfigFactory
.parseReader(
new FileReader(ClassLoader.getSystemResource("config/forum.conf").getFile()))
.withFallback(ConfigFactory.load());
System.out.println(conf);
final ActorSystem system = ActorSystem.create("AkkaApp", conf);
final ActorRef masterActor = system.actorOf(Props.create(MasterActor.class), "Migrate");
masterActor.tell("", ActorRef.noSender());
} catch (Exception e) {
e.printStackTrace();
}
}
MasterActor
public class MasterActor extends UntypedActor {
private final ActorRef randomRouter = getContext().system()
.actorOf(Props.create(StateActor.class).withRouter(new akka.routing.FromConfig()), "StateActor");
#Override
public void onReceive(Object message) throws Exception {
if (message instanceof String) {
getContext().watch(randomRouter);
for (String aState : getStates()) {
randomRouter.tell(aState, getSelf());
}
randomRouter.tell(new Broadcast(PoisonPill.getInstance()), getSelf());
} else if (message instanceof Terminated) {
Terminated ater = (Terminated) message;
if (ater.getActor().equals(randomRouter)) {
getContext().system().terminate();
}
}
}
public List<String> getStates() {
return new ArrayList<String>(Arrays.asList("CA", "MA", "TA", "NJ", "NY"));
};}
StateActor
public class StateActor extends UntypedActor {
private final ActorRef randomRouter = getContext().system()
.actorOf(Props.create(CityActor.class).withRouter(new akka.routing.FromConfig()), "CityActor");
#Override
public void onReceive(Object message) throws Exception {
if (message instanceof String) {
System.out.println("Processing state " + message);
for (String aCity : getCitiesForState((String) message)) {
randomRouter.tell(aCity, getSelf());
}
Thread.sleep(1000);
}
}
public List<String> getCitiesForState(String stateName) {
return new ArrayList<String>(Arrays.asList("Springfield-" + stateName, "Salem-" + stateName,
"Franklin-" + stateName, "Clinton-" + stateName, "Georgetown-" + stateName));
};}
CityActor
public class CityActor extends UntypedActor {
#Override
public void onReceive(Object message) throws Exception {
if (message instanceof String) {
System.out.println("Processing city " + message);
Thread.sleep(1000);
}
}}
Did i implement this use case properly?
I cannot get the code to terminate properly, i get dead letters messages. I know why i am getting them, but not sure how to properly implement it.
Any help is greatly appreciated.
Thanks
I tested and ran your use case with Akka 2.4.17. It works and terminate properly, without any dead letters logged.
Here are some remarks/suggestions to improve your understanding of the Akka toolkit:
Do not use Thread.sleep() inside an actor. Basically, it is never a good practice since a same thread may do many tasks for many actors (this is the default behavior with a shared thread pool). Instead, you can use an Akka scheduler or assign a single thread to a specific Actor (see this post for more details). See also the Akka documentation about that topic.
Having some dead letters is not always an issue. It generally arises when the system stops an Actor that had some messages within its mailbox. In this case, the remaining unprocessed messages are sent to deadLetters of the ActorSystem. I recommend you to check the configuration you provided for the logging of dead letters. If the file forum.conf you provided is your complete configuration file for Akka, you may want to customize some additional settings. See the page Logging of Dead Letters and Stopping actors on Akka's website. For instance, you could have a section like this:
akka {
# instead of System.out.println(conf);
log-config-on-start = on
# Max number of dead letters to log
log-dead-letters = 10
log-dead-letters-during-shutdown = on
}
Instead of using System.out.println() to log/debug, it is more convenient to set up a dedicated logger for each Actor that provides you additional information such as dispatchers, Actor name, etc. If your are interested, have a look to the Logging page.
Use some custom immutable message objects instead of systematic Strings. At first, it may seem painful to have to declare new additional classes but in the end it helps to better design complex behaviors and it's more readable. For instance, an actor A can answer to a RequestMsg coming from an actor B with an AnswerMsg or a custom ErrorMsg. Then, for your actor B, you will end up with the following onReceive() method:
#Override
public void onReceive(Object message) {
if (message instanceof AnswerMsg) {
// OK
AnswerMsg answerMsg = (AnswerMsg) message;
// ...
}
if (message instanceof ErrorMsg) {
// Not OK
ErrorMsg errorMsg = (ErrorMsg) message;
// ...
}
else {
// Unexpected behaviour, log it
log.error("Error, received " + message.toString() + " object.")
}
}
I hope that these resources will be useful for you.
Have a happy Akka programming! ;)

Clojure java class casting

Is it possible to cast in clojure with java style?
This is java code which I want to implement in clojure:
public class JavaSoundRecorder {
// the line from which audio data is captured
TargetDataLine line;
/**
* Captures the sound and record into a WAV file
*/
void start() {
try {
AudioFormat format = new AudioFormat(16000, 8,
2, true, true);
DataLine.Info info = new DataLine.Info(TargetDataLine.class, format);
System.out.println(AudioSystem.isLineSupported(info));
// checks if system supports the data line
if (!AudioSystem.isLineSupported(info)) {
System.out.println("Line not supported");
System.exit(0);
}
//line = (TargetDataLine) AudioSystem.getLine(info);
line = AudioSystem.getTargetDataLine(format);
line.open(format);
line.start(); // start capturing
System.out.println("Start capturing...");
AudioInputStream ais = new AudioInputStream(line);
System.out.println("Start recording...");
// start recording
AudioSystem.write(ais, AudioFileFormat.Type.WAVE, new File("RecordAudio.wav"));
} catch (LineUnavailableException ex) {
ex.printStackTrace();
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
/**
* Closes the target data line to finish capturing and recording
*/
void finish() {
line.stop();
line.close();
System.out.println("Finished");
}
/**
* Entry to run the program
*/
public static void main(String[] args) {
final JavaSoundRecorder recorder = new JavaSoundRecorder();
// creates a new thread that waits for a specified
// of time before stopping
Thread stopper = new Thread(new Runnable() {
public void run() {
try {
Thread.sleep(6000);
} catch (InterruptedException ex) {
ex.printStackTrace();
}
recorder.finish();
}
});
stopper.start();
// start recording
recorder.start();
}
}
And this is what I made in clojure
(def audioformat (new javax.sound.sampled.AudioFormat 16000 8 2 true true))
(def info (new javax.sound.sampled.DataLine$Info javax.sound.sampled.TargetDataLine audioformat))
(if (not= (javax.sound.sampled.AudioSystem/isLineSupported info))(print "dataline not supported")(print "ok lets start\n"))
(def line (javax.sound.sampled.AudioSystem/getTargetDataLine audioformat))
(.open line audioformat)
are there any solutions?
this issue was explained rather well on the Clojure group here:
https://groups.google.com/forum/#!topic/clojure/SNcT6d-TTaQ
You should not need to do the cast (see the discussion in the comments about the super types of the object we have), however you will need to type hint the invocation of open:
(.open ^javax.sound.sampled.TargetDataLine line audioformat)
Remember that java casts don't really do very much (not like C++ where a cast might completely transform an underlying object).
I am not sure what this code is supposed to do, so I don't know whether it has worked or not. Certainly, I can now run your example without error.