unmarshall nested json in spray-json - akka

Using spray-json (as I'm using spray-client) in order to get a latitude,longitude object from the google maps API I need to have the whole response structure set up:
case class AddrComponent(long_name: String, short_name: String, types: List[String])
case class Location(lat: Double, lng: Double)
case class ViewPort(northeast: Location, southwest: Location)
case class Geometry(location: Location, location_type: String, viewport: ViewPort)
case class EachResult(address_components: List[AddrComponent],
formatted_address: String,
geometry: Geometry,
types: List[String])
case class GoogleApiResult[T](status: String, results: List[T])
object AddressProtocol extends DefaultJsonProtocol {
implicit val addrFormat = jsonFormat3(AddrComponent)
implicit val locFormat = jsonFormat2(Location)
implicit val viewPortFormat = jsonFormat2(ViewPort)
implicit val geomFormat = jsonFormat3(Geometry)
implicit val eachResFormat = jsonFormat4(EachResult)
implicit def GoogleApiFormat[T: JsonFormat] = jsonFormat2(GoogleApiResult.apply[T])
}
import AddressProtocol._
Is there any way I can just get Location from the json in the response and avoid all this gumph?
The spray-client code:
implicit val system = ActorSystem("test-system")
import system.dispatcher
private val pipeline = sendReceive ~> unmarshal[GoogleApiResult[EachResult]]
def getPostcode(postcode: String): Point = {
val url = s"http://maps.googleapis.com/maps/api/geocode/json?address=$postcode,+UK&sensor=true"
val future = pipeline(Get(url))
val result = Await.result(future, 10 seconds)
result.results.size match {
case 0 => throw new PostcodeNotFoundException(postcode)
case x if x > 1 => throw new MultipleResultsException(postcode)
case _ => {
val location = result.results(0).geometry.location
new Point(location.lng, location.lat)
}
}
}
Or alternatively how can I use jackson with spray-client?

Following jrudolph's advice to json-lenses I also got in quite a bit of fiddling but finally got things to work. I found it quite difficult (as a newbie) and also I am sure this solution is far from the most elegant - nevertheless I think this might help people or inspire others for improvements.
Given JSON:
{
"status": 200,
"code": 0,
"message": "",
"payload": {
"statuses": {
"emailConfirmation": "PENDING",
"phoneConfirmation": "DONE",
}
}
}
And case class for unmarshalling statuses only:
case class UserStatus(emailConfirmation: String, phoneConfirmation: String)
One can do this to unmarshal response:
import scala.concurrent.Future
import spray.http.HttpResponse
import spray.httpx.unmarshalling.{FromResponseUnmarshaller, MalformedContent}
import spray.json.DefaultJsonProtocol
import spray.json.lenses.JsonLenses._
import spray.client.pipelining._
object UserStatusJsonProtocol extends DefaultJsonProtocol {
implicit val userStatusUnmarshaller = new FromResponseUnmarshaller[UserStatus] {
implicit val userStatusJsonFormat = jsonFormat2(UserStatus)
def apply(response: HttpResponse) = try {
Right(response.entity.asString.extract[UserStatus]('payload / 'statuses))
} catch { case x: Throwable =>
Left(MalformedContent("Could not unmarshal user status.", x))
}
}
}
import UserStatusJsonProtocol._
def userStatus(userId: String): Future[UserStatus] = {
val pipeline = sendReceive ~> unmarshal[UserStatus]
pipeline(Get(s"/api/user/${userId}/status"))
}

Related

Akka Stream Materialized Value

I want to reference the materialized value from the flow. Below is the code snippet, but its not compiling, error:
type mismatch;
found : (akka.NotUsed, scala.concurrent.Future[akka.Done])
required: (Playground.DomainObj, scala.concurrent.Future[akka.Done])
Code:
import akka.actor.ActorSystem
import akka.stream.scaladsl._
import scala.concurrent.Future
import akka.NotUsed
import akka.Done
implicit val actorSystem = ActorSystem("example")
case class DomainObj(name: String, age: Int)
val customFlow1:Flow[String,DomainObj,NotUsed] = Flow[String].map(s => {
DomainObj(s, 50)
})
val customFlow2 = Flow[DomainObj].map(s => {
s.age + 10
})
val printAnySink: Sink[Any, Future[Done]] = Sink.foreach(println)
val c1 = Source.single("John").viaMat(customFlow1)(Keep.right).viaMat(customFlow2)(Keep.left).toMat(printAnySink)(Keep.both)
val res: (DomainObj, Future[Done]) = c1.run()
Find the code in playground: https://scastie.scala-lang.org/P9iSx49cQcaOZfKtVCzTPA
I want to reference the DomainObj after the stream completes/
The materialized value of a Flow[String, DomainObj, NotUsed] is NotUsed, not a DomainObj, therefore c1's materialized value is (NotUsed, Future[Done]).
It looks like the intent here is to capture the DomainObj which is created in customFlow1. That can be accomplished with
val customFlow1: Flow[String, DomainObj, Future[DomainObj]] =
Flow[String]
.map { s => DomainObj(s, 50) }
.alsoTo(Sink.head)
val res: (Future[DomainObj], Future[Done]) = c1.run()
Note that Sink.head effectively requires that customFlow1 can only be used downstream of something that only emits once.

How to access metrics of Alpakka CommittableSource with back off?

Accessing the metrics of an Alpakka PlainSource seems fairly straight forward, but how can I do the same thing with a CommittableSource?
I currently have a simple consumer, something like this:
class Consumer(implicit val ma: ActorMaterializer, implicit val ec: ExecutionContext) extends Actor {
private val settings = ConsumerSettings(
context.system,
new ByteArrayDeserializer,
new StringDeserializer)
.withProperties(...)
override def receive: Receive = Actor.emptyBehavior
RestartSource
.withBackoff(minBackoff = 2.seconds, maxBackoff = 20.seconds, randomFactor = 0.2)(consumer)
.runForeach { handleMessage }
private def consumer() = {
AkkaConsumer
.committableSource(settings, Subscriptions.topics(Set(topic)))
.log(getClass.getSimpleName)
.withAttributes(ActorAttributes.supervisionStrategy(_ => Supervision.Resume))
}
private def handleMessage(message: CommittableMessage[Array[Byte], String]): Unit = {
...
}
}
How can I get access to the consumer metrics in this case?
We are using the Java prometheus client and I solved my issue with a custom collector that fetches its metrics directly from JMX:
import java.lang.management.ManagementFactory
import java.util
import io.prometheus.client.Collector
import io.prometheus.client.Collector.MetricFamilySamples
import io.prometheus.client.CounterMetricFamily
import io.prometheus.client.GaugeMetricFamily
import javax.management.ObjectName
import scala.collection.JavaConverters._
import scala.collection.mutable
class ConsumerMetricsCollector(val labels: Map[String, String] = Map.empty) extends Collector {
val metrics: mutable.Map[String, MetricFamilySamples] = mutable.Map.empty
def collect: util.List[MetricFamilySamples] = {
val server = ManagementFactory.getPlatformMBeanServer
for {
attrType <- List("consumer-metrics", "consumer-coordinator-metrics", "consumer-fetch-manager-metrics")
name <- server.queryNames(new ObjectName(s"kafka.consumer:type=$attrType,client-id=*"), null).asScala
attrInfo <- server.getMBeanInfo(name).getAttributes.filter { _.getType == "double" }
} yield {
val attrName = attrInfo.getName
val metricLabels = attrName.split(",").map(_.split("=").toList).collect {
case "client-id" :: (id: String) :: Nil => ("client-id", id)
}.toList ++ labels
val metricName = "kafka_consumer_" + attrName.replaceAll(raw"""[^\p{Alnum}]+""", "_")
val labelKeys = metricLabels.map(_._1).asJava
val metric = metrics.getOrElseUpdate(metricName,
if(metricName.endsWith("_total") || metricName.endsWith("_sum")) {
new CounterMetricFamily(metricName, attrInfo.getDescription, labelKeys)
} else {
new GaugeMetricFamily(metricName, attrInfo.getDescription, labelKeys)
}: MetricFamilySamples
)
val metricValue = server.getAttribute(name, attrName).asInstanceOf[Double]
val labelValues = metricLabels.map(_._2).asJava
metric match {
case f: CounterMetricFamily => f.addMetric(labelValues, metricValue)
case f: GaugeMetricFamily => f.addMetric(labelValues, metricValue)
case _ =>
}
}
metrics.values.toList.asJava
}
}

Is it possible to create an "infinite" stream from a database table using Akka Stream

I'm playing with Akka Streams 2.4.2 and am wondering if it's possible to setup a stream which uses a database table for a source and whenever there is a record added to the table that record is materialized and pushed downstream?
UPDATE: 2/23/16
I've implemented the solution from #PH88. Here's my table definition:
case class Record(id: Int, value: String)
class Records(tag: Tag) extends Table[Record](tag, "my_stream") {
def id = column[Int]("id")
def value = column[String]("value")
def * = (id, value) <> (Record.tupled, Record.unapply)
}
Here's the implementation:
implicit val system = ActorSystem("Publisher")
implicit val materializer = ActorMaterializer()
val db = Database.forConfig("pg-postgres")
try{
val newRecStream = Source.unfold((0, List[Record]())) { n =>
try {
val q = for (r <- TableQuery[Records].filter(row => row.id > n._1)) yield (r)
val r = Source.fromPublisher(db.stream(q.result)).collect {
case rec => println(s"${rec.id}, ${rec.value}"); rec
}.runFold((n._1, List[Record]())) {
case ((id, xs), current) => (current.id, current :: xs)
}
val answer: (Int, List[Record]) = Await.result(r, 5.seconds)
Option(answer, None)
}
catch { case e:Exception => println(e); Option(n, e) }
}
Await.ready(newRecStream.throttle(1, 1.second, 1, ThrottleMode.shaping).runForeach(_ => ()), Duration.Inf)
}
finally {
system.shutdown
db.close
}
But my problem is that when I attempt to call flatMapConcat the type I get is Serializable.
UPDATE: 2/24/16
Updated to try db.run suggestion from #PH88:
implicit val system = ActorSystem("Publisher")
implicit val materializer = ActorMaterializer()
val db = Database.forConfig("pg-postgres")
val disableAutoCommit = SimpleDBIO(_.connection.setAutoCommit(false))
val queryLimit = 1
try {
val newRecStream = Source.unfoldAsync(0) { n =>
val q = TableQuery[Records].filter(row => row.id > n).take(queryLimit)
db.run(q.result).map { recs =>
Some(recs.last.id, recs)
}
}
.throttle(1, 1.second, 1, ThrottleMode.shaping)
.flatMapConcat { recs =>
Source.fromIterator(() => recs.iterator)
}
.runForeach { rec =>
println(s"${rec.id}, ${rec.value}")
}
Await.ready(newRecStream, Duration.Inf)
}
catch
{
case ex: Throwable => println(ex)
}
finally {
system.shutdown
db.close
}
Which works (I changed query limit to 1 since I only have a couple items in my database table currently) - except once it prints the last row in the table the program exists. Here's my log output:
17:09:27,982 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
17:09:27,982 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
17:09:27,982 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/Users/xxxxxxx/dev/src/scratch/scala/fpp-in-scala/target/scala-2.11/classes/logback.xml]
17:09:28,062 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
17:09:28,064 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
17:09:28,079 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
17:09:28,102 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [application] to DEBUG
17:09:28,103 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
17:09:28,103 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
17:09:28,103 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
17:09:28,104 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator#4278284b - Registering current configuration as safe fallback point
17:09:28.117 [main] INFO com.zaxxer.hikari.HikariDataSource - pg-postgres - is starting.
1, WASSSAAAAAAAP!
2, WHAAAAT?!?
3, booyah!
4, what!
5, This rocks!
6, Again!
7, Again!2
8, I love this!
9, Akka Streams rock
10, Tuning jdbc
17:09:39.000 [main] INFO com.zaxxer.hikari.pool.HikariPool - pg-postgres - is closing down.
Process finished with exit code 0
Found the missing piece - need to replace this:
Some(recs.last.id, recs)
with this:
val lastId = if(recs.isEmpty) n else recs.last.id
Some(lastId, recs)
The call to recs.last.id was throwing java.lang.UnsupportedOperationException: empty.last when the result set was empty.
In general SQL database is a 'passive' construct and does not actively push changes like what you described. You can only 'simulate' the 'push' with periodic polling like:
val newRecStream = Source
// Query for table changes
.unfold(initState) { lastState =>
// query for new data since lastState and save the current state into newState...
Some((newState, newRecords))
}
// Throttle to limit the poll frequency
.throttle(...)
// breaks down into individual records...
.flatMapConcat { newRecords =>
Source.unfold(newRecords) { pendingRecords =>
if (records is empty) {
None
} else {
// take one record from pendingRecords and save to newRec. Save the rest into remainingRecords.
Some(remainingRecords, newRec)
}
}
}
Updated: 2/24/2016
Pseudo code example based on the 2/23/2016 updates of the question:
implicit val system = ActorSystem("Publisher")
implicit val materializer = ActorMaterializer()
val db = Database.forConfig("pg-postgres")
val queryLimit = 10
try {
val completion = Source
.unfoldAsync(0) { lastRowId =>
val q = TableQuery[Records].filter(row => row.id > lastRowId).take(queryLimit)
db.run(q.result).map { recs =>
Some(recs.last.id, recs)
}
}
.throttle(1, 1.second, 1, ThrottleMode.shaping)
.flatMapConcat { recs =>
Source.fromIterator(() => recs.iterator)
}
.runForeach { rec =>
println(s"${rec.id}, ${rec.value}")
}
// Block forever
Await.ready(completion, Duration.Inf)
} catch {
case ex: Throwable => println(ex)
} finally {
system.shutdown
db.close
}
It will repeatedly execute the query in unfoldAsync against the DB, retrieving at most 10 (queryLimit) records a time and send the records downstream (-> throttle -> flatMapConcat -> runForeach). The Await at the end will actually block forever.
Updated: 2/25/2016
Executable 'proof-of-concept' code:
import akka.actor.ActorSystem
import akka.stream.{ThrottleMode, ActorMaterializer}
import akka.stream.scaladsl.Source
import scala.concurrent.duration.Duration
import scala.concurrent.{Await, Future}
import scala.concurrent.duration._
object Infinite extends App{
implicit val system = ActorSystem("Publisher")
implicit val ec = system.dispatcher
implicit val materializer = ActorMaterializer()
case class Record(id: Int, value: String)
try {
val completion = Source
.unfoldAsync(0) { lastRowId =>
Future {
val recs = (lastRowId to lastRowId + 10).map(i => Record(i, s"rec#$i"))
Some(recs.last.id, recs)
}
}
.throttle(1, 1.second, 1, ThrottleMode.Shaping)
.flatMapConcat { recs =>
Source.fromIterator(() => recs.iterator)
}
.runForeach { rec =>
println(rec)
}
Await.ready(completion, Duration.Inf)
} catch {
case ex: Throwable => println(ex)
} finally {
system.shutdown
}
}
Here is database infinite streaming working code. This has been tested with millions of records being inserted into postgresql database while streaming app is running -
package infinite.streams.db
import akka.NotUsed
import akka.actor.ActorSystem
import akka.stream.alpakka.slick.scaladsl.SlickSession
import akka.stream.scaladsl.{Flow, Sink, Source}
import akka.stream.{ActorMaterializer, ThrottleMode}
import org.slf4j.LoggerFactory
import slick.basic.DatabaseConfig
import slick.jdbc.JdbcProfile
import scala.concurrent.duration._
import scala.concurrent.{Await, ExecutionContextExecutor}
case class Record(id: Int, value: String) {
val content = s"<ROW><ID>$id</ID><VALUE>$value</VALUE></ROW>"
}
object InfiniteStreamingApp extends App {
println("Starting app...")
implicit val system: ActorSystem = ActorSystem("Publisher")
implicit val ec: ExecutionContextExecutor = system.dispatcher
implicit val materializer: ActorMaterializer = ActorMaterializer()
println("Initializing database configuration...")
val databaseConfig: DatabaseConfig[JdbcProfile] = DatabaseConfig.forConfig[JdbcProfile]("postgres3")
implicit val session: SlickSession = SlickSession.forConfig(databaseConfig)
import databaseConfig.profile.api._
class Records(tag: Tag) extends Table[Record](tag, "test2") {
def id = column[Int]("c1")
def value = column[String]("c2")
def * = (id, value) <> (Record.tupled, Record.unapply)
}
val db = databaseConfig.db
println("Prime for streaming...")
val logic: Flow[(Int, String), (Int, String), NotUsed] = Flow[(Int, String)].map {
case (id, value) => (id, value.toUpperCase)
}
val fetchSize = 5
try {
val done = Source
.unfoldAsync(0) {
lastId =>
println(s"Fetching next: $fetchSize records with id > $lastId")
val query = TableQuery[Records].filter(_.id > lastId).take(fetchSize)
db.run(query.result.withPinnedSession)
.map {
recs => Some(recs.last.id, recs)
}
}
.throttle(5, 1.second, 1, ThrottleMode.shaping)
.flatMapConcat {
recs => Source.fromIterator(() => recs.iterator)
}
.map(x => (x.id, x.content))
.via(logic)
.log("*******Post Transformation******")
// .runWith(Sink.foreach(r => println("SINK: " + r._2)))
// Use runForeach or runWith(Sink)
.runForeach(rec => println("REC: " + rec))
println("Waiting for result....")
Await.ready(done, Duration.Inf)
} catch {
case ex: Throwable => println(ex.getMessage)
} finally {
println("Streaming end successfully")
db.close()
system.terminate()
}
}
application.conf
akka {
loggers = ["akka.event.slf4j.Slf4jLogger"]
loglevel = "INFO"
}
# Load using SlickSession.forConfig("slick-postgres")
postgres3 {
profile = "slick.jdbc.PostgresProfile$"
db {
dataSourceClass = "slick.jdbc.DriverDataSource"
properties = {
driver = "org.postgresql.Driver"
url = "jdbc:postgresql://localhost/testdb"
user = "postgres"
password = "postgres"
}
numThreads = 2
}
}

How to mock spray-client response

I have a simple spray client :
val pipeline = sendReceive ~> unmarshal[GoogleApiResult[Elevation]]
val responseFuture = pipeline {Get("http://maps.googleapis.com/maps/api/elevation/jsonlocations=27.988056,86.925278&sensor=false") }
responseFuture onComplete {
case Success(GoogleApiResult(_, Elevation(_, elevation) :: _)) =>
log.info("The elevation of Mt. Everest is: {} m", elevation)
shutdown()
case Failure(error) =>
log.error(error, "Couldn't get elevation")
shutdown()
}
Full code can be found here.
I want to mock the response of the server to test the logic in the Success and Failure cases. The only relevant information i found was here but I haven't been able to use the cake pattern to mock the sendReceive method.
Any suggestion or example would be greatly appreciated.
Here's an example of one way to mock it using specs2 for the test spec and mockito for the mocking. First, the Main object refactored into a class setup for mocking:
class ElevationClient{
// we need an ActorSystem to host our application in
implicit val system = ActorSystem("simple-spray-client")
import system.dispatcher // execution context for futures below
val log = Logging(system, getClass)
log.info("Requesting the elevation of Mt. Everest from Googles Elevation API...")
import ElevationJsonProtocol._
import SprayJsonSupport._
def sendAndReceive = sendReceive
def elavation = {
val pipeline = sendAndReceive ~> unmarshal[GoogleApiResult[Elevation]]
pipeline {
Get("http://maps.googleapis.com/maps/api/elevation/json?locations=27.988056,86.925278&sensor=false")
}
}
def shutdown(): Unit = {
IO(Http).ask(Http.CloseAll)(1.second).await
system.shutdown()
}
}
Then, the test spec:
class ElevationClientSpec extends Specification with Mockito{
val mockResponse = mock[HttpResponse]
val mockStatus = mock[StatusCode]
mockResponse.status returns mockStatus
mockStatus.isSuccess returns true
val json = """
{
"results" : [
{
"elevation" : 8815.71582031250,
"location" : {
"lat" : 27.9880560,
"lng" : 86.92527800000001
},
"resolution" : 152.7032318115234
}
],
"status" : "OK"
}
"""
val body = HttpEntity(ContentType.`application/json`, json.getBytes())
mockResponse.entity returns body
val client = new ElevationClient{
override def sendAndReceive = {
(req:HttpRequest) => Promise.successful(mockResponse).future
}
}
"A request to get an elevation" should{
"return an elevation result" in {
val fut = client.elavation
val el = Await.result(fut, Duration(2, TimeUnit.SECONDS))
val expected = GoogleApiResult("OK",List(Elevation(Location(27.988056,86.925278),8815.7158203125)))
el mustEqual expected
}
}
}
So my approach here was to first define an overridable function in the ElevationClient called sendAndReceive that just delegates to the spray sendReceive function. Then, in the test spec, I override that sendAndReceive function to return a function that returns a completed Future wrapping a mock HttpResponse. This is one approach for doing what you want to do. I hope this helps.
There's no need to introduce mocking in this case, as you can simply build a HttpResponse much more easily using the existing API:
val mockResponse = HttpResponse(StatusCodes.OK, HttpEntity(ContentTypes.`application/json`, json.getBytes))
(Sorry for posting this as another answer, but don't have enough karma to comment)

Play2 testing controller methods using Action composition

I want to test a controller action using Action composition.
Here's an example of the composed action and its test code.
The Secured trait:
trait Secured {
def username(request: RequestHeader) = request.session.get(Security.username)
def onUnauthorized(request: RequestHeader) = Results.Redirect(routes.Auth.login)
def withAuth(f: => String => Request[AnyContent] => Result) = {
Security.Authenticated(username, onUnauthorized) { user =>
Action(request => f(user)(request))
}
}
The controller:
MyController extends Contrller with Secured {
def simple = Action { Ok("ok") }
def simpleWithauth = withAuth { implicit username => implicit request=> Ok("ok") }
}
The test code:
// This work fine
val result1 = controller.simple()(FakeRequest())
// This wont compile
val result2 = controller.simpleWithAuth()(FakeRequest())
The latter would require a Request[Action[AnyContent], AnyContent]
but FakeRequest returns a Request[AnyContent]
Any pointers on how to create a fake request of the appropriate type?
This is what I had to do to test secured action
def wrappedActionResult[A](wrapped: Action[(Action[A], A)], request: Request[A]): Result = wrapped.parser(request).run.await.get match {
case Left(errorResult) => errorResult
case Right((innerAction, _)) => innerAction(request)
}
and the test
running(app) {
val result = wrappedActionResult(FakeController().securedAction, invalidRequest)
status(result) must_== UNAUTHORIZED
contentAsString(result) must_== "must be authenticated"
}
That doesn't work with Play 2.1 though, types have been changed...
UPDATE:
in Play 2.1 it's even easier
I've added some sugar
implicit class ActionExecutor(action: EssentialAction) {
def process[A](request: Request[A]): Result = concurrent.Await.result(action(request).run, Duration(1, "sec"))
}
and the test now looks like this
running(app) {
val result = FakeController().securedAction process invalidRequest
status(result) must_== UNAUTHORIZED
contentAsString(result) must_== "must be authenticated"
}
UPDATE: here is a related blog post i wrote sometime ago
http://www.daodecode.com/blog/2013/03/08/testing-security-dot-authenticated-in-play-2-dot-0-and-2-dot-1/
Have you tried using the routeAndCall approach?
val Some(result) = routeAndCall(FakeRequest(POST, "/someRoute"))
You can add whatever parameters you need in your request with the asFormUrlEncodedBody method and/or adding things to the session with the withSession method.