Continuous REST consumption using Akka stream and http - akka

Question

I am trying to build an akka based system which will periodically(every 15 second) send REST request, do some filter, some data cleansing and validation on the received data and save into HDFS.
Below is the code that I wrote.
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.{Flow, Sink, Source}
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.{HttpRequest, HttpResponse, StatusCodes}
import akka.actor.Props
import akka.event.Logging
import akka.actor.Actor
import scala.concurrent.{ExecutionContext, Future}
import scala.util.Try
import akka.http.scaladsl.client.RequestBuilding._
/**
* Created by rabbanerjee on 4/6/2017.
*/
class MyActor extends Actor {
val log = Logging(context.system, this)
import scala.concurrent.ExecutionContext.Implicits.global
def receive = {
case j:HttpResponse => log.info("received" +j)
case k:AnyRef => log.info("received unknown message"+k)
}
}
object STest extends App{
implicit val system = ActorSystem("Sys")
import system.dispatcher
implicit val materializer = ActorMaterializer()
val ss = system.actorOf(Props[MyActor])
val httpClient = Http().outgoingConnection(host = "rest_server.com", port = 8080)
val filterSuccess = Flow[HttpResponse].filter(_.status.isSuccess())
val runnnn = Source.tick(
FiniteDuration(1,TimeUnit.SECONDS),
FiniteDuration(15,TimeUnit.SECONDS),
Get("/"))
.via(httpClient)
.via(filterSuccess)
.to(Sink.actorRef(ss,onCompleteMessage = "done"))
runnnn.run()
}
The problem I am currently facing is,
Even though I used a repeat/tick with source, I can see the result once. It's not repetitively firing the request.
I am also trying to find grouping the result of say 50 such request, coz as I will be writing it to hadoop, I cant write every request, as it will flood HDFS with multiple file.

Answer 1

You are not consuming the responses you are getting back from the HTTP call. It is compulsory to consume the entity bytes returned by Akka HTTP, even if you are not interested in them.
More about this can be found in the docs.
In your example, as you are not using the response entity, you can just discard its bytes. See example below:
val runnnn = Source.tick(FiniteDuration(1,TimeUnit.SECONDS),FiniteDuration(15,TimeUnit.SECONDS),Get("/"))
.via(httpClient)
.map{resp => resp.discardEntityBytes(); resp}
.via(filterSuccess)
.to(Sink.actorRef(ss,onCompleteMessage = "done"))