Play Scala : Recursive web services call without using Await.result - web-services

Question

I work with Play! Scala 2.2 and I need to make recursive web services call (until I get all the results).
For the moment I managed to make it with Await.result as following :
def getTitleSetFromJson( jsValue: JsValue ): Set[ String ] = {
val titleReads: Reads[Option[String]] = (__ \\ "title").readNullable[String]
(jsValue \ "response" \ "songs")
.asOpt[Set[Option[String]]](Reads.set(titleReads))
.getOrElse(Set.empty)
.flatten
}
def findEchonestSongs(start: Long, echonestId: String): Future[Set[String]] = {
val titleReads: Reads[Option[String]] = (__ \\ "title").readNullable[String]
val endpoint = s"$baseUrl/artist/songs"
val assembledUrl = s"$endpoint?api_key=$echonestApiKey&id=$artistId&format=json&start=$start&results=20"
WS.url(assembledUrl).get().map { songs =>
if ((songs.json \ "response" \ "total").asOpt[Int].getOrElse(0) > start + 20) {
getTitleSetFromJson(songs.json) ++
Await.result( findEchonestSongs(start + 20, echonestId), 3.seconds )
} else {
getTitleSetFromJson(songs.json)
}
}
}
Now I would like to make the same thing but with the non blocking way i.e. without using Await.result but with everything I tried I got errors : types mismatch because of the nested futures.
Could you please give me the way to do this?

Answer 1

There are many different ways to achieve your requirement. But maybe the simplest way is using flatMap.
Here is an improved version of your function (for making it simpler I took the liberty to make it return a Future[Set[JsValue]]).
import play.api.libs.json.JsValue
import play.api.libs.ws.WS
import scala.concurrent.Future
import play.api.Play.current
import scala.concurrent.ExecutionContext.Implicits.global
object Test {
val apiKey = "//YOUR_ECHONEST_API_KEY//"
val baseUrl = "http://developer.echonest.com/api/v4"
def findSongs(start: Long, artistId: String): Future[Set[JsValue]] = {
def endpoint = s"$baseUrl/artist/songs"
def assembledUrl = s"$endpoint?api_key=$apiKey&id=$artistId&format=json&start=$start&results=20"
def response = WS.url(assembledUrl).get()
def futureJson = response map (_.json)
futureJson flatMap { result =>
def total = (result \ "response" \ "total").asOpt[Int]
def songs = (result \ "response" \ "songs").as[Set[JsValue]]
total exists (_ > start + 20) match {
case false => Future.successful(songs)
case true => findSongs(start + 20, artistId) map (songs ++ _)
}
}
}
}
Working on the result
If you really want a Set[String] you can easily use map on the result.
val sample = findSongs(0, "someone")
sample.map(_.map(_ \ "title").map(_.as[String]))
Improvements
There is a lot of room for improvement here. For instance there is no need to pass the artistId every time you cal the function recursively. So we can define an inner function which is recursive.
It is not that difficult to make it tail recursive by using an accumulator.
Finally it might make sense to use a Seq[Future] and use fold instead of recursion, which is again a common pattern in functional programming. But all these stuff are already provided nicely by Stream api. You can refer to the Stream API for more info.

Answer 2

You can map the results from other ( from nested call ) future to be added to current Set.
Also... One very important thing. Readability Counts... Specially on StackOverFLow.
def findEchonestSongs(start: Long, echonestId: String): Future[Set[String]] = {
val titleReads: Reads[Option[String]] = (__ \\ "title").readNullable[String]
val requestUrl = "http://developer.echonest.com/api/v4/artist/songs?api_key=" + echonestApiKey + "&id=" + echonestId +
"&format=json&start=" + start + "&results=20"
WS.url( requestUrl ).get().flatMap { songs =>
var nTotal = (songs.json \ "response" \ "total").asOpt[Int].getOrElse(0)
if( nTotal > start + 20 ) {
val titleSet = getTitleSetFromJson( songs.json )
val moreTitleSetFuture = findEchonestSongs( start + 20, echonestId )
moreTitleSetFuture.map( moreTitleSet => titleSet ++ moreTitleSet )
}
else {
Future.successful( getTitleSetFromJson( songs.json ) )
}
}
}
def getTitleSetFromJson( jsValue: JsValue ): Set[ String ] = {
(songs.json \ "response" \ "songs")
.asOpt[Set[Option[String]]](Reads.set(titleReads))
.getOrElse(Set.empty)
.flatten
}