Scala Futures Returning Empty List after Await - list

I have a program that performs an:
Await.result(Processor.validateEntries(queuedEntries)), Duration.Inf)
And the validateEntries method calls some other method that performs:
val validatedEntries: ListBuffer[Entries] = new ListBuffer[Entries]
for (entry <- queuedEntries) {
checkEntry(entry.name).map(.......... validatedEntries += Entries(...) )
}
Future(validatedEntries.toList)
where checkEntry returns a Future[Boolean].
def checkEntry(name: String): Future[Boolean] = {
checkNameAlreadyExists(name).flatMap(exists =>
buildRequest(exists, name).map(response => {
if (!response.contains("error")) {
true
} else {
false
}
})
)
}
At the top level where I perform the Await.result I also get back an empty list: List(). Any suggestions would greatly help!

Mixing mutable collections and concurrency is not a good idea. Consider refactoring checkEntry to return, say, Future[Option[Entry]] instead of Future[Boolean], where Some would represent successful validation, whilst None unssucessful, and then you might do something like
case class Entry(v: Int)
val queuedEntries = List(Entry(1), Entry(2), Entry(3))
def checkEntry(entry: Entry): Future[Option[Entry]] = ???
Future
.traverse(queuedEntries)(checkEntry)
.map(_.flatten)
If keeping checkEntry as it is, then you might try something like
case class Entry(v: Int)
val queuedEntries = List(Entry(1), Entry(2), Entry(3))
def checkEntry(entry: Entry): Future[Boolean] = Future(Random.nextBoolean)
Future
.traverse(queuedEntries)(checkEntry)
.map(checkedEntries => checkedEntries zip queuedEntries)
.map(_.collect { case (validated, entry) if validated => entry} )

You have to use for comprehension. Basically first you have to read the list and in yield, you have to call function one by one and wait for future to complete by yield
package com.vimit.StackOverflow
import scala.concurrent._
import ExecutionContext.Implicits.global
object FutureProblem extends App {
val list = List(1, 2, 3)
val outputList = List()
val result = for {
value <- list
} yield {
for {
result <- getValue(value).map(res => outputList ++ List(value))
} yield result
}
print(result)
def getValue(value: Int) = Future(value)
}

Related

Kotlin Mockk : Unable to mock a list correctly

I am trying to verify that .shuffled() on a list is called, but get an error on running because of a prior .take(6) call on the list, and I cannot see a way around this.
Here is some code that gets the same error:
val mockList =
mockk<List<String>> { every { shuffled() } returns mockk(relaxed = true) }
val choiceList = spyk(listOf("String1", "String2")) { every { take(6) } returns mockList }
val tmp = choiceList.take(6)
val tmp2 = tmp.shuffled()
verify {mockList.shuffled())
On line 4, I get the following error:
class io.mockk.renamed.java.util.List$Subclass0 cannot be cast to class java.lang.Integer (io.mockk.renamed.java.util.List$Subclass0 is in unnamed module of loader 'app'; java.lang.Integer is in module java.base of loader 'bootstrap')
Attempting to go around by directly verifying on choiceList.take(6).shuffled() and combining the two tmp vals into one has had no success, as it gets true whether or not .shuffled() gets called. Also, switching from a spy to a mock for choiceList has also not worked.
Edit: Note, since this is a toy example, the take() is completely necessary, and cannot be removed, as it has real use in the actual code.
Interesting one!
I think in current implementation it is not possible. The easy answer would be "this test misses the declaration of wrapping static class" (as extension methods are just the same as java static methods for JVM). But if we add it...
#Test
fun test() {
mockkStatic("kotlin.reflect.jvm.internal.impl.utils.CollectionsKt")
val iterClass = mockkClass(Iterable::class)
val mockList = mockk<List<String>> { every { shuffled() } returns mockk(relaxed = true) }
with(iterClass) {
every { take(6) } returns mockList
val tmp = take(6)
val tmp2 = tmp.shuffled()
verify {
mockList.shuffled()
}
}
}
we have a Recursion detected in a lazy value under LockBasedStorageManager#1d2ad266 (DeserializationComponentsForJava.ModuleData) which is understandable - we just mocked the whole extensions package. And it is not possible to mock only one extension method leaving others intact. (source: https://github.com/mockk/mockk#extension-functions)
However, I'd do the following. Why not make our own extension functions which call the original and mock those?
It would go like this:
Main.kt:
package root
...
fun <T> Iterable<T>.take(n: Int): Iterable<T> {
val m = Iterable<T>::take
return m.call(this)
}
fun <T> Iterable<T>.shuffled(): Iterable<T> {
val m = Iterable<T>::shuffled
return m.call(this)
}
Test.kt:
package root
...
#Test
fun test() {
// note this changed
mockkStatic("root.MainKt")
val iterClass = mockkClass(Iterable::class)
val mockList = mockk<List<String>> { every { shuffled() } returns mockk(relaxed = true) }
with(iterClass) {
every { take(6) } returns mockList
val tmp = take(6)
val tmp2 = tmp.shuffled()
verify {
mockList.shuffled()
}
}
}
The only downside here I think is that it's reflection (duh!) So, this can possibly affect performance and has the requirement to have implementation(kotlin("reflect")) in the dependencies (to use call()). If it is not feasible I think there's no clean solution.
val mockList: List<String> = mockk(relaxed = true)
mockList.shuffled()
verify { mockList.shuffled() }
This works for me. The problem is that take of choiceList cannot be mocked somehow. Is that really necessary?

What's an idiomatic way to traverse and update data structures functionally in Scala?

I'm coming from a Python-heavy background and trying to learn Scala through a basic "Design a Parking Lot" exercise. I have Scala code that looks something like:
class ParkingLot(spaces: Set[ParkingSpace]) {
var openSpaces: Set[ParkingSpace] = spaces;
var usedSpaces: Set[ParkingSpace] = Set()
def assign(vehicle: Vehicle): Boolean = {
var found = false;
for (s <- openSpaces) {
(s.isCompatibleWithVehicle(vehicle)) match {
case true => {
if (!found) {
s.acceptVehicle(vehicle)
openSpaces -= s
usedSpaces += s
found = true
}
}
case _ => {}
}
}
found
}
}
The logic is pretty simple - I have a ParkingLot with Sets of open and occupied ParkingSpaces. I want to define a function assign that takes in a Vehicle, loops through all the openSpaces and if it finds an available space, will update the open and used spaces. I'm having a hard time coming up with a good idiomatic way to do this. Any thoughts and suggestions about how to reframe questions into a Scala mindset?
The main problem with this code is use of mutable state (var). Rather than changing an existing object, functional code creates new, modified objects. So the functional approach is to create a new ParkingLot each time with the appropriate allocation of spaces.
case class ParkingLot(open: Set[ParkingSpace], used: Set[ParkingSpace])
{
def assignVehicle(vehicle: Vehicle): Option[ParkingLot] =
open.find(_.isCompatibleWithVehicle(vehicle)).map { space =>
ParkingLot(open - space, used + space.acceptVehicle(vehicle))
}
}
assignVehicle can return a new parking lot with the spaces appropriately updated. It returns an Option because there might not be a compatible space, in which case it returns None. The caller can take whatever action is necessary in this case.
Note that ParkingSpace now as an acceptVehicle that returns a new ParkingSpace rather than modifying itself.
As also the answer by #Tim mentioned, you need to avoid mutations, and try to handle this kind of state managements in functions. I'm not gonna dive into the details since Tim mentioned some, I'm just proposing a new approach to the implementation, which uses a map of spaces to weather they're used, and returns a new (not optional) instance every time you assign a new vehicle (if the vehicle fits in, updated instance is returned, and if not, the same instance):
class ParkingLot(spaces: Map[ParkingSpace, Boolean]) {
def withVehicleAssigned(vehicle: Vehicle): ParkingLot =
spaces.collectFirst {
case (space, used) if !used && space.isCompatibleWithVehicle(vehicle) =>
new ParkingLot(spaces.updated(space, true))
}.getOrElse(this)
}
Almost the same process goes for removing vehicles, the usage would be something like this:
parkingLot
.withVehicleAssigned(v1)
.withVehicleAssigned(v2)
.withVehicleRemoved(v1)
Since most answers already explained the importance of immutability and creating new objects, I am just going to propose two alternative models and solutions.
1. Using a queue of empty spaces plus a set of used ones.
final case class ParkingLot(freeSpaces: List[ParkingSpace], occupiedSpaces: Set[ParkingSpace]) {
// Returns the used space and the new parking lot.
// An option is used since the parking lot may be full.
def assign(vehicle: Vehicle): Option[(ParkingSpace, ParkingLot)] =
freeSpaces match {
case freeSpace :: remainingSpaces =>
val usedSpace = freeSpace.withVehicle(vehicle)
Some(copy(freeSpaces = remainingSpaces, usedSpaces = usedSpace + usedSpaces))
case Nil =>
None
}
}
2. Using a List[(ParkingSpace, Boolean)] and a tail-recursive function.
final case class ParkingLot(parkingSpaces: List[(ParkingSpace, Boolean)]) {
// Returns the used space and the new parking lot.
// An option is used since the parking lot may be full.
def assign(vehicle: Vehicle): Option[(ParkingSpace, ParkingLot)] = {
#annotation.tailrec
def loop(remaining: List[(ParkingSpace, Boolean)], acc: List[(ParkingSpace, Boolean)]): Option[(ParkingSpace, List[(ParkingSpace, Boolean)])] =
remaining match {
case (parkingSpace, occupied) :: tail =>
if (occupied) loop(remaining = tail, (parkingSpace, occupied) :: acc)
else {
val usedSpace = parkingSpace.withVehicle(vehicle)
val newSpaces = acc reverse_::: ((usedSpace -> true) :: tail)
Some(usedSpace -> newSpaces)
}
case Nil =>
None
}
loop(remaining = parkingSpaces, acc = List.empty).map {
case (usedSpace, newSpaces) =>
usedSpace -> copy(newSpaces)
}
}
}
Note, the boolean may be redundant since the ParkingSpace should be able to tell us if it is empty or not.

Unit testing retries with project reactor

I have a piece of code that goes something like this:
somePublisher
.subscribeOn(...)
.flatMap { x -> someFunctionThatReturnsMono(x) }
.retry(3)
.subscribe()
So far I have managed to unit test happy paths, like whether the code inside map {...} is getting called, using the tools from reactor-test.
Now I want to test errors and retries. How can I test to make sure someFunctionThatReturnsMono(x) is getting called at most 4 times when consecutive errors occur?
Simple way is to mock function and count its calls:
def 'Test retry'(){
setup:
someBean = Mock(SomeBean)
def testMono = Mono.just(X)
.flatMap { x -> someBean.someFunctionThatReturnsMono(x) }
.retry(3)
.subscribeOn(Schedulers.elastic())
when:
StepVerifier.create(testMono).verifyError(IllegalArgumentException)
then:
4 * someBean.someFunctionThatReturnsMono(_ as X) >> Mono.error(new IllegalArgumentException())
}
Another one is to mock funtion to return PublisherProbe and count probe subscriptions:
def 'Test retry'() {
when:
PublisherProbe probe = PublisherProbe.of(Mono.error(new IllegalArgumentException()))
someBean = Mock(SomeBean) {
someFunctionThatReturnsMono(_) >> probe
}
def testMono = Mono.just(X)
.flatMap { x -> someBean.someFunctionThatReturnsMono(x) }
.retry(3)
.subscribeOn(Schedulers.elastic())
then:
StepVerifier.create(testMono).verifyError(IllegalArgumentException)
probe.subscribeCount() == 4
}
This is how I do it for one of my use-cases:
def "should retry failed action given number of times"() {
given:
def invocationCounter = new AtomicInteger(0)
when:
Mono<Void> mono = myService.doSth()
.doOnError { invocationCounter.incrementAndGet() }
then:
StepVerifier.create(mono)
.verifyErrorMatches { it == expectedError }
and:
invocationCounter.get() == 1 + retryNum
}

Processing Akka stream in Slick transaction

Software versions:
Akka 2.4.4
Slick 3.1.0
I want to process elements from an Akka stream in a Slick transaction.
Here is some simplified code to illustrate one possible approach:
def insert(d: AnimalFields): DBIO[Long] =
animals returning animals.map(_.id) += d
val source: Source[AnimalFields, _]
val sourceAsTraversable = ???
db.run((for {
ids <- DBIO.sequence(sourceAsTraversable.map(insert))
} yield { ids }).transactionally)
One solution I could come up with so far is blocking each future to traverse the elements:
class TraversableQueue[T](sinkQueue: SinkQueue[T]) extends Traversable[T] {
#tailrec private def next[U](f: T => U): Unit = {
val nextElem = Await.result(sinkQueue.pull(), Duration.Inf)
if (nextElem.isDefined) {
f(nextElem.get)
next(f)
}
}
def foreach[U](f: T => U): Unit = next(f)
}
val sinkQueue = source.runWith(Sink.queue())
val queue = new TraversableQueue(sinkQueue)
Now I can pass the traversable queue to DBIO.sequence(). This defeats the purpose of streamed processing, though.
Another approach I found is this:
def toDbioAction[T](queue: SinkQueue[DBIOAction[S, NoStream, Effect.All]]):
DBIOAction[Queue[T], NoStream, Effect.All] =
DBIO.from(queue.pull() map { tOption =>
tOption match {
case Some(action) =>
action.flatMap(t => toDbioAction(queue).map(_ :+ t))
case None => DBIO.successful(Queue())
}
}).flatMap(r => r)
With this method, a sequence of DBIOActions can be generated without blocking:
toDbioAction(source.runWith(Sink.queue()))
Is there any better / more idiomatic way to achieve the desired result?
Here is my implementation of sourceAsTraversable:
import scala.collection.JavaConverters._
def sourceAsTraversable[A](source: Source[A, _])(implicit mat: Materializer): Traversable[A] =
source.runWith(StreamConverters.asJavaStream()).iterator().asScala.toIterable
The issue with TraversableQueue was that the forEach had to finish processing the stream fully - it did not support the "break" concept, so methods like "drop"/"take", etc. would still have to process whole source. This could be important from error handling point of view and failing fast.

Scala way / idiom of dealing with immutable List

I have found successes using ideas of immutable List but I am stumped when come to this piece of code here. I find myself has written something more Java than of Scala style. I would prefer to use List(...) instead of Buffer(...) but I don't see how I can pass the same modified immutable List to the next function. guesses is also modified within eliminate(...).
Any suggestions to help me to make this the Scala way of doing this is appreciated. Thanks
val randomGuesses = List(...) // some long list of random integers
val guesses = randomGuesses.zipWithIndex.toBuffer
for ( s <- loop()) {
val results = alphaSearch(guesses)
if (results.size == 1) {
guesses(resultes.head._2) = results.head._1
eliminate(guesses, resultes.head._2)
}
else {
val results = betaSearch(guesses)
if (results.size == 1) {
guesses(resultes.head._2) = results.head._1
eliminate(guesses, resultes.head._2)
} else {
val results = betaSearch(guesses)
if (results.size == 1) {
guesses(resultes.head._2) = results.head._1
eliminate(guesses, resultes.head._2)
}
}
}
}
Here are some general tips since this might be better suited for codereview and the code posted is incomplete with no samples.
You can use pattern matching instead of if and else for checking the size.
results.size match{
case 1 => ... //Code in the if block
case _ => ... //Code in the else block
}
Instead of mutating guesses create a new List.
val newGuesses = ...
Then pass newGuesses into eliminate.
Lastly, it looks like eliminate modifies guesses. Change this to return a new list. e.g.
def eliminate(list: List[Int]) = {
//Eliminate something from list and return a new `List`
}