Calling arbitrary number of WS.url().get() in sequence - list

Question

I have a List[String] of URLs that I want to load and process (parse, store to database) in sequence.
I found only fixed-length examples, like:
def readUrls = Action {
implicit request => {
implicit val context = scala.concurrent.ExecutionContext.Implicits.global
val url1 = "http://some-website.com"
val url2 = "http://other-website.com"
Async {
for {
result1 <- WS.url(url1).get()
result2 <- WS.url(url2).get()
} yield {
Ok(result1.body + result2.body)
}
}
}
But instead of url1 and url2, I need to process this puppy:
val urls = List("http://some-website.com", "http://other-website.com")
Thanks a bunch for any tips and advice!

Answer 1

If you want to chain Futures together arbitrarily in sequence, foldLeft ought to do the job:
urls.foldLeft(Future.successful[String]("")){ case (left, nextUrl) =>
left.flatMap{ aggregatedResult =>
WS.url(nextUrl).get().map( newResult =>
aggregatedResult + newResult.body
)
}
}
Since you're just combining the request bodies together, I gave the foldLeft an initial value of a Future empty String, which each step in the fold will then add on the next response body.

Answer 2

def executeUrls(urls: List[String]): Future[String] = {
urls.foldLeft(Future(""))((accumulator, url) => {
accumulator.flatMap(acc => {
WS.url(url).get().map(response => {
acc + response.body
})
}
})
}
This should be what you're looking for, note that it returns a new Future.
Edit: apparently LimbSoup was faster.