Software versions:
Akka 2.4.4
Slick 3.1.0
I want to process elements from an Akka stream in a Slick transaction.
Here is some simplified code to illustrate one possible approach:
def insert(d: AnimalFields): DBIO[Long] =
animals returning animals.map(_.id) += d
val source: Source[AnimalFields, _]
val sourceAsTraversable = ???
db.run((for {
ids <- DBIO.sequence(sourceAsTraversable.map(insert))
} yield { ids }).transactionally)
One solution I could come up with so far is blocking each future to traverse the elements:
class TraversableQueue[T](sinkQueue: SinkQueue[T]) extends Traversable[T] {
#tailrec private def next[U](f: T => U): Unit = {
val nextElem = Await.result(sinkQueue.pull(), Duration.Inf)
if (nextElem.isDefined) {
f(nextElem.get)
next(f)
}
}
def foreach[U](f: T => U): Unit = next(f)
}
val sinkQueue = source.runWith(Sink.queue())
val queue = new TraversableQueue(sinkQueue)
Now I can pass the traversable queue to DBIO.sequence(). This defeats the purpose of streamed processing, though.
Another approach I found is this:
def toDbioAction[T](queue: SinkQueue[DBIOAction[S, NoStream, Effect.All]]):
DBIOAction[Queue[T], NoStream, Effect.All] =
DBIO.from(queue.pull() map { tOption =>
tOption match {
case Some(action) =>
action.flatMap(t => toDbioAction(queue).map(_ :+ t))
case None => DBIO.successful(Queue())
}
}).flatMap(r => r)
With this method, a sequence of DBIOActions can be generated without blocking:
toDbioAction(source.runWith(Sink.queue()))
Is there any better / more idiomatic way to achieve the desired result?
Here is my implementation of sourceAsTraversable:
import scala.collection.JavaConverters._
def sourceAsTraversable[A](source: Source[A, _])(implicit mat: Materializer): Traversable[A] =
source.runWith(StreamConverters.asJavaStream()).iterator().asScala.toIterable
The issue with TraversableQueue was that the forEach had to finish processing the stream fully - it did not support the "break" concept, so methods like "drop"/"take", etc. would still have to process whole source. This could be important from error handling point of view and failing fast.
Related
I'm coming from a Python-heavy background and trying to learn Scala through a basic "Design a Parking Lot" exercise. I have Scala code that looks something like:
class ParkingLot(spaces: Set[ParkingSpace]) {
var openSpaces: Set[ParkingSpace] = spaces;
var usedSpaces: Set[ParkingSpace] = Set()
def assign(vehicle: Vehicle): Boolean = {
var found = false;
for (s <- openSpaces) {
(s.isCompatibleWithVehicle(vehicle)) match {
case true => {
if (!found) {
s.acceptVehicle(vehicle)
openSpaces -= s
usedSpaces += s
found = true
}
}
case _ => {}
}
}
found
}
}
The logic is pretty simple - I have a ParkingLot with Sets of open and occupied ParkingSpaces. I want to define a function assign that takes in a Vehicle, loops through all the openSpaces and if it finds an available space, will update the open and used spaces. I'm having a hard time coming up with a good idiomatic way to do this. Any thoughts and suggestions about how to reframe questions into a Scala mindset?
The main problem with this code is use of mutable state (var). Rather than changing an existing object, functional code creates new, modified objects. So the functional approach is to create a new ParkingLot each time with the appropriate allocation of spaces.
case class ParkingLot(open: Set[ParkingSpace], used: Set[ParkingSpace])
{
def assignVehicle(vehicle: Vehicle): Option[ParkingLot] =
open.find(_.isCompatibleWithVehicle(vehicle)).map { space =>
ParkingLot(open - space, used + space.acceptVehicle(vehicle))
}
}
assignVehicle can return a new parking lot with the spaces appropriately updated. It returns an Option because there might not be a compatible space, in which case it returns None. The caller can take whatever action is necessary in this case.
Note that ParkingSpace now as an acceptVehicle that returns a new ParkingSpace rather than modifying itself.
As also the answer by #Tim mentioned, you need to avoid mutations, and try to handle this kind of state managements in functions. I'm not gonna dive into the details since Tim mentioned some, I'm just proposing a new approach to the implementation, which uses a map of spaces to weather they're used, and returns a new (not optional) instance every time you assign a new vehicle (if the vehicle fits in, updated instance is returned, and if not, the same instance):
class ParkingLot(spaces: Map[ParkingSpace, Boolean]) {
def withVehicleAssigned(vehicle: Vehicle): ParkingLot =
spaces.collectFirst {
case (space, used) if !used && space.isCompatibleWithVehicle(vehicle) =>
new ParkingLot(spaces.updated(space, true))
}.getOrElse(this)
}
Almost the same process goes for removing vehicles, the usage would be something like this:
parkingLot
.withVehicleAssigned(v1)
.withVehicleAssigned(v2)
.withVehicleRemoved(v1)
Since most answers already explained the importance of immutability and creating new objects, I am just going to propose two alternative models and solutions.
1. Using a queue of empty spaces plus a set of used ones.
final case class ParkingLot(freeSpaces: List[ParkingSpace], occupiedSpaces: Set[ParkingSpace]) {
// Returns the used space and the new parking lot.
// An option is used since the parking lot may be full.
def assign(vehicle: Vehicle): Option[(ParkingSpace, ParkingLot)] =
freeSpaces match {
case freeSpace :: remainingSpaces =>
val usedSpace = freeSpace.withVehicle(vehicle)
Some(copy(freeSpaces = remainingSpaces, usedSpaces = usedSpace + usedSpaces))
case Nil =>
None
}
}
2. Using a List[(ParkingSpace, Boolean)] and a tail-recursive function.
final case class ParkingLot(parkingSpaces: List[(ParkingSpace, Boolean)]) {
// Returns the used space and the new parking lot.
// An option is used since the parking lot may be full.
def assign(vehicle: Vehicle): Option[(ParkingSpace, ParkingLot)] = {
#annotation.tailrec
def loop(remaining: List[(ParkingSpace, Boolean)], acc: List[(ParkingSpace, Boolean)]): Option[(ParkingSpace, List[(ParkingSpace, Boolean)])] =
remaining match {
case (parkingSpace, occupied) :: tail =>
if (occupied) loop(remaining = tail, (parkingSpace, occupied) :: acc)
else {
val usedSpace = parkingSpace.withVehicle(vehicle)
val newSpaces = acc reverse_::: ((usedSpace -> true) :: tail)
Some(usedSpace -> newSpaces)
}
case Nil =>
None
}
loop(remaining = parkingSpaces, acc = List.empty).map {
case (usedSpace, newSpaces) =>
usedSpace -> copy(newSpaces)
}
}
}
Note, the boolean may be redundant since the ParkingSpace should be able to tell us if it is empty or not.
I have a program that performs an:
Await.result(Processor.validateEntries(queuedEntries)), Duration.Inf)
And the validateEntries method calls some other method that performs:
val validatedEntries: ListBuffer[Entries] = new ListBuffer[Entries]
for (entry <- queuedEntries) {
checkEntry(entry.name).map(.......... validatedEntries += Entries(...) )
}
Future(validatedEntries.toList)
where checkEntry returns a Future[Boolean].
def checkEntry(name: String): Future[Boolean] = {
checkNameAlreadyExists(name).flatMap(exists =>
buildRequest(exists, name).map(response => {
if (!response.contains("error")) {
true
} else {
false
}
})
)
}
At the top level where I perform the Await.result I also get back an empty list: List(). Any suggestions would greatly help!
Mixing mutable collections and concurrency is not a good idea. Consider refactoring checkEntry to return, say, Future[Option[Entry]] instead of Future[Boolean], where Some would represent successful validation, whilst None unssucessful, and then you might do something like
case class Entry(v: Int)
val queuedEntries = List(Entry(1), Entry(2), Entry(3))
def checkEntry(entry: Entry): Future[Option[Entry]] = ???
Future
.traverse(queuedEntries)(checkEntry)
.map(_.flatten)
If keeping checkEntry as it is, then you might try something like
case class Entry(v: Int)
val queuedEntries = List(Entry(1), Entry(2), Entry(3))
def checkEntry(entry: Entry): Future[Boolean] = Future(Random.nextBoolean)
Future
.traverse(queuedEntries)(checkEntry)
.map(checkedEntries => checkedEntries zip queuedEntries)
.map(_.collect { case (validated, entry) if validated => entry} )
You have to use for comprehension. Basically first you have to read the list and in yield, you have to call function one by one and wait for future to complete by yield
package com.vimit.StackOverflow
import scala.concurrent._
import ExecutionContext.Implicits.global
object FutureProblem extends App {
val list = List(1, 2, 3)
val outputList = List()
val result = for {
value <- list
} yield {
for {
result <- getValue(value).map(res => outputList ++ List(value))
} yield result
}
print(result)
def getValue(value: Int) = Future(value)
}
I'm trying to listen to sqs using akka streams and i get messages from it's q
using this code snippet:
of course this code snippet get messages one-by-one (then ack it):
implicit val system = ActorSystem()
implicit val mat = ActorMaterializer()
implicit val ec = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(ioThreadPoolSize))
val awsSqsClient: AmazonSQSAsync = AmazonSQSAsyncClientBuilder
.standard()
.withCredentials(new ClasspathPropertiesFileCredentialsProvider())
.withEndpointConfiguration(new EndpointConfiguration(sqsEndpoint, configuration.regionName))
.build()
val future = SqsSource(sqsEndpoint)(awsSqsClient)
.takeWhile(_ => true)
.mapAsync(parallelism = 2)(m => {
val msgBody = SqsMessage.deserializeJson(m.getBody)
msgBody match {
case Right(body) => val id = getId(body) //do some stuff with the message may save state according the id
}
Future(m, Ack())
})
.to(SqsAckSink(sqsEndpoint)(awsSqsClient))
.run()
my question is:
can i get several messages, and save them for example in a stateful map for latter use?
for example that after receiving 5 messages (all of them will saved (per state))
then if specific condition happens i will ack them all, and if not they will return into queue (will happen anyway because visibility timeout)?
thanks.
Could be that you're looking for grouped (or groupedWithin) combinator. These allow you to batch messages and process them in groups. groupedWithin allows you to release a batch after a certain time in case it hasn't yet reached your determined size. Docs reference here.
In a subsequent check flow you can perform any logic you need, and emit the sequence in case you want the messages to be acked, or not emit them otherwise.
Example:
val yourCheck: Flow[Seq[MessageActionPair], Seq[MessageActionPair], NotUsed] = ???
val future = SqsSource(sqsEndpoint)(awsSqsClient)
.takeWhile(_ => true)
.mapAsync(parallelism = 2){ ... }
.grouped(5)
.via(yourCheck)
.mapConcat(identity)
.to(SqsAckSink(sqsEndpoint)(awsSqsClient))
.run()
I'm using Akka and I want to define my own supervision strategy for User Guardian actor. I've defined two types of actors, called TaskActor and MessageActor. They're instantiated as top-level actors.
I wish the user guardian to apply the following supervision strategy: Stop TaskActor and Resume MessageActor when they throw an Exception (I don't mind what particular type of Exception is thrown). How can I do it?
I've found the following solution:
object ActorClassUtil {
val ACTORS_PACKAGE = "utils.actors."
val TASK_ACTOR_CLASS_NAME = ACTORS_PACKAGE + "TaskActor"
val MSG_ACTOR_CLASS_NAME = ACTORS_PACKAGE + "MessageActor"
def getFailedActorClassFrom(e: Exception): String = {
val stackTrace = e.getStackTrace
// FQCN (Fully Qualified Class Name)
stackTrace(0) getClassName
}
}
class UserGuardianSupervisorStrategy extends SupervisorStrategyConfigurator {
import ActorClassUtil._
def create(): OneForOneStrategy = {
OneForOneStrategy(maxNrOfRetries = 20, withinTimeRange = 1 minute) {
case _: ActorInitializationException => Stop
case _: ActorKilledException => Stop
case _: DeathPactException => Stop
case exception: Exception =>
val actorClassName = getFailedActorClassFrom(exception)
if (actorClassName startsWith TASK_ACTOR_CLASS_NAME)
Stop
else
if (actorClassName startsWith MSG_ACTOR_CLASS_NAME)
Resume
else
Stop
}
}
}
I have two webservice calls. Webservice1 returns Promise[Option[Model]] and Webservice2 should take Model as a parameter and then return a Promise[Option[String]]. This is how I have structured my code:
def call1: Promise[Option[Model]] = for {
response1 <- callService1(requestHolderForService1)
} yield {
for {
response <- response1
} yield ParserForResponse(response)
}
After, this I want to chain my call to service 2 that takes the result from service 1 as a parameter:
def call2:Promise[Option[String]] = call1.flatMap{
optionModel => optionModel.flatMap{
model => callService2(requestHolderForService2(model)).map{
service2Option => service2Option.map{
service2Result => ResultParse(service2Result)
}
}
}
}
The problem is that my call1 returns a Promise[Option[Model]] whereas the return from call2 needs to be Promise[Option[String]]. The issue stems from the intermediate service call
callService2
which returns Promise[Option[JsValue]] and I am unable to figure out the transition from Promise[Option[Model]] -> Promise[Option[JsValue]] -> Promise[Option[String]]
Can somebody point out how I might be able to chain these two calls together using map or flatMap?
Thanks
The "normal" way would not to be working with the promises directly, but futures for those promises, which you can access from scala.concurrent.Promise.future
First you map the future, think of it as when the option arrives, do this transformation with it, then you need to handle the fact that the option may not exist, which also is a map, think of it as if there is a value to this tranformation on it.
so:
val future1: Future[Option[Model]] = (...).future
val future2: Future[Option[String]] = future1.map { maybeModel =>
maybeModel.map(model => model.toString) // or hovewer you make it a string
}
Which you then can use in your second web service call