Spray http client and thousands requests - akka

Question

I want to configure spray http client in a way that controls max number of request that were sent to the server. I need this because server that i'm sending requests to blocks me if there are more then 2 request were sent. I get
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://smallTasks/user/IO-HTTP#151444590]] after [15000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://smallTasks/user/IO-HTTP#151444590]] after [15000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://smallTasks/user/IO-HTTP#151444590]] after [15000 ms]
akka.pattern.AskTimeoutException: Ask timed out on
I need to send thousands of requests but i get blocked after i got responses from ~ 100 requests.
I have this method:
implicit val system = ActorSystem("smallTasks")
implicit val timeout = new Timeout(15.seconds)
import system.dispatcher
def doHttpRequest(url: String): Future[HttpResponse] = {
(IO(Http) ? HttpRequest(GET, Uri(url))).mapTo[HttpResponse]
}
And here i catch responses and retry if it fails(recursively):
def getOnlineOffers(modelId: Int, count: Int = 0): Future[Any] = {
val result = Promise[Any]()
AkkaSys.doHttpRequest(Market.modelOffersUrl(modelId)).map(response => {
val responseCode = response.status.intValue
if (List(400, 404).contains(responseCode)) {
result.success("Bad request")
} else if (responseCode == 200) {
Try {
Json.parse(response.entity.asString).asOpt[JsObject]
} match {
case Success(Some(obj)) =>
Try {
(obj \\ "onlineOffers").head.as[Int]
} match {
case Success(offers) => result.success(offers)
case _ => result.success("Can't find property")
}
case _ => result.success("Wrong body")
}
} else {
result.success("Unexpected error")
}
}).recover { case err =>
if (count > 5) {
result.success("Too many tries")
} else {
println(err.toString)
Thread.sleep(200)
getOnlineOffers(modelId, count + 1).map(r => result.success(r))
}
}
result.future
}
How to do this properly? May be i need to configure akka dispatcher somehow?

Answer 1

you can use http://spray.io/documentation/1.2.2/spray-client/ and write you personal pipeline
val pipeline: Future[SendReceive] =
for (
Http.HostConnectorInfo(connector, _) <-
IO(Http) ? Http.HostConnectorSetup("www.spray.io", port = 80)
) yield sendReceive(connector)
val request = Get("/segment1/segment2/...")
val responseFuture: Future[HttpResponse] = pipeline.flatMap(_(request))
to get HttpResponse
import scala.concurrent.Await
import scala.concurrent.duration._
val response: HttpResponse = Aweit(responseFuture, ...)
to convert
import spray.json._
response.entity.asString.parseJson.convertTo[T]
to check
Try(response.entity.asString.parseJson).isSuccess
too many brackets. In scala you can write it shorter