I have the following server
object FileUploadServer {
implicit val system = ActorSystem("fileUploadServer")
implicit val materializer = ActorMaterializer()
implicit val ec = system.dispatcher
val route =
path("upload") {
post {
extractRequest { request =>
println(request)
val file = File.createTempFile("debug",".zip")
val futureDone: Future[Long] = request.entity.dataBytes.runWith(Sink.file(file))
onComplete(futureDone) { result =>
complete(s"path:${file.getAbsolutePath}, size:${result.get}")
}
}
}
}
def main(args: Array[String]) {
val serverFuture: Future[ServerBinding] = Http().bindAndHandle(route, "localhost", 8080)
println("FileUpload server is running at http://localhost:8080\nPress RETURN to stop ...")
readLine()
serverFuture
.flatMap(_.unbind())
.onComplete(_ => system.terminate())
}
}
and I have a following property in application.conf
akka.http.parsing.max-content-length = 1000m
But when I try to upload a 300MB zip file, it returns only when 159MB
$ time curl -X POST -H 'Content-Type: application/octet-stream' -d #bluecoat_proxy_big.zip http://localhost:8080/upload
path:/var/folders/_2/q9xz_8ks73d0h6hq80c7439s8_x7qx/T/debug3960545903381659311.zip, size:155858257
real 0m1.426s
user 0m0.375s
sys 0m0.324s
What am I missing?
Related
Accessing the metrics of an Alpakka PlainSource seems fairly straight forward, but how can I do the same thing with a CommittableSource?
I currently have a simple consumer, something like this:
class Consumer(implicit val ma: ActorMaterializer, implicit val ec: ExecutionContext) extends Actor {
private val settings = ConsumerSettings(
context.system,
new ByteArrayDeserializer,
new StringDeserializer)
.withProperties(...)
override def receive: Receive = Actor.emptyBehavior
RestartSource
.withBackoff(minBackoff = 2.seconds, maxBackoff = 20.seconds, randomFactor = 0.2)(consumer)
.runForeach { handleMessage }
private def consumer() = {
AkkaConsumer
.committableSource(settings, Subscriptions.topics(Set(topic)))
.log(getClass.getSimpleName)
.withAttributes(ActorAttributes.supervisionStrategy(_ => Supervision.Resume))
}
private def handleMessage(message: CommittableMessage[Array[Byte], String]): Unit = {
...
}
}
How can I get access to the consumer metrics in this case?
We are using the Java prometheus client and I solved my issue with a custom collector that fetches its metrics directly from JMX:
import java.lang.management.ManagementFactory
import java.util
import io.prometheus.client.Collector
import io.prometheus.client.Collector.MetricFamilySamples
import io.prometheus.client.CounterMetricFamily
import io.prometheus.client.GaugeMetricFamily
import javax.management.ObjectName
import scala.collection.JavaConverters._
import scala.collection.mutable
class ConsumerMetricsCollector(val labels: Map[String, String] = Map.empty) extends Collector {
val metrics: mutable.Map[String, MetricFamilySamples] = mutable.Map.empty
def collect: util.List[MetricFamilySamples] = {
val server = ManagementFactory.getPlatformMBeanServer
for {
attrType <- List("consumer-metrics", "consumer-coordinator-metrics", "consumer-fetch-manager-metrics")
name <- server.queryNames(new ObjectName(s"kafka.consumer:type=$attrType,client-id=*"), null).asScala
attrInfo <- server.getMBeanInfo(name).getAttributes.filter { _.getType == "double" }
} yield {
val attrName = attrInfo.getName
val metricLabels = attrName.split(",").map(_.split("=").toList).collect {
case "client-id" :: (id: String) :: Nil => ("client-id", id)
}.toList ++ labels
val metricName = "kafka_consumer_" + attrName.replaceAll(raw"""[^\p{Alnum}]+""", "_")
val labelKeys = metricLabels.map(_._1).asJava
val metric = metrics.getOrElseUpdate(metricName,
if(metricName.endsWith("_total") || metricName.endsWith("_sum")) {
new CounterMetricFamily(metricName, attrInfo.getDescription, labelKeys)
} else {
new GaugeMetricFamily(metricName, attrInfo.getDescription, labelKeys)
}: MetricFamilySamples
)
val metricValue = server.getAttribute(name, attrName).asInstanceOf[Double]
val labelValues = metricLabels.map(_._2).asJava
metric match {
case f: CounterMetricFamily => f.addMetric(labelValues, metricValue)
case f: GaugeMetricFamily => f.addMetric(labelValues, metricValue)
case _ =>
}
}
metrics.values.toList.asJava
}
}
Here i have created a singleton actor. The master and the seed node is the same. From a different project i am trying to add into the cluster and want to send message. I am able to join in the cluster but cannot be able to send message.
My master and seed node:
package Demo
import akka.actor._
import akka.cluster.singleton.{ClusterSingletonManager, ClusterSingletonManagerSettings, ClusterSingletonProxy, ClusterSingletonProxySettings}
import com.typesafe.config.ConfigFactory
import scala.concurrent.duration._
object MainObject1 extends App{
DemoMain1.start(8888)
}
object DemoMain1 {
val role = "test"
val singletonname = "Ruuner"
val mastername = "Master"
def start(port: Int): ActorSystem = {
val conf = ConfigFactory.parseString(
s"""
|akka.actor.provider = "akka.cluster.ClusterActorRefProvider"
|
|akka.remote.netty.tcp.port = $port
|akka.remote.netty.tcp.hostname = 127.0.0.1
|akka.cluster.roles = ["$role"]
|akka.cluster.seed-nodes = ["akka.tcp://DemoMain1#127.0.0.1:8888"]
""".stripMargin
)
val system = ActorSystem("DemoMain1", conf)
val settings = ClusterSingletonManagerSettings(system).withRole(role)
val manager = ClusterSingletonManager.props(Props[DemoMain1], PoisonPill, settings)
val actor=system.actorOf(manager, mastername)
system
}
class DemoMain1 extends Actor with Identification {
import context._
override def preStart(): Unit = {
println(s"Master is created with id $id in $system")
println(self.path.address.host)
system.scheduler.scheduleOnce(100.seconds, self, 'exit)
}
override def receive : Receive={
case 'exit => println("$id is exiting")
context stop self
//SupervisorStrategy.Restart
case msg => println(s"messasge from $system is $msg ")
sender() ! 'how
}
}
}
The other node which is trying to join the cluster and send message.
import akka.actor._
import akka.cluster.singleton.{ClusterSingletonProxy, ClusterSingletonProxySettings}
import com.typesafe.config.ConfigFactory
import scala.concurrent.duration._
object Ping extends App{
def ping: ActorSystem = {
val conf = ConfigFactory.parseString(
s"""
|akka.actor.provider = "akka.cluster.ClusterActorRefProvider"
|
|akka.remote.netty.tcp.port = 0
|akka.remote.netty.tcp.hostname = 127.0.0.1
|akka.cluster.roles = ["slave"]
|akka.cluster.seed-nodes = ["akka.tcp://DemoMain1#127.0.0.1:8888"]
|akka.cluster.auto-down-unreachable-after = 10s
""".stripMargin
)
val system = ActorSystem("DemoMain1", conf)
system.actorOf(Props[Ping])
system
}
class Ping extends Actor {
import context._
val path = "akka.tcp://DemoMain1#127.0.0.1:8888/DemoMain1/user/Master/actor"
val settings = ClusterSingletonProxySettings(system).withRole("slave")
val actor = context.actorOf(ClusterSingletonProxy.props(path, settings))
val pingInterval = 1.seconds
override def preStart(): Unit = {
system.scheduler.schedule(pingInterval, pingInterval) {
println(s"Locate Ping $system")
actor ! 'hi
}
}
override def receive: Receive = {
case msg => println(s"The message is $msg")
}
}
Ping.ping
}
If i give ip addresses of the system then also the message s not sent.
It appears the role in your ClusterSingletonProxySettings(system).withRole("slave") settings for your Ping actor doesn't match that of your ClusterSingletonManagerSettings(system).withRole(role) where role = "test".
ClusterSingletonProxy is supposed to be present on all nodes with the specified role on which the cluster is set up, hence its role settings should match ClusterSingletonManager's. Here's a sample configuration.
I have the following case, where I'm trying to delegate processing to an actor. What I want to happen is that whenever my flow processes a message, it sends it to the actor, and the actor will uppercase it and write it to the stream as a response.
So I should be able to connect to port 8000, type in "hello", have the flow send it to the actor, and have the actor publish it back to the stream so it's echoed back to me uppercased. The actor itself is pretty basic, from the ActorPublisher example in the docs.
I know this code doesn't work, I cleaned up my experiments to get it to compile. Right now, it's just two separate streams. I tried to experiment with merging the sources or the sinks, to no avail.
object Sample {
def main(args: Array[String]): Unit = {
implicit val system = ActorSystem("sample")
implicit val materializer = ActorMaterializer()
val connections: Source[IncomingConnection,
Future[ServerBinding]] = Tcp().bind("localhost", 8000)
val filter = Source.actorPublisher[ByteString](Props[Filter])
val filterRef = Flow[ByteString]
.to(Sink.ignore)
.runWith(filter)
connections runForeach { conn =>
val echo = Flow[ByteString] .map {
// would like to send 'p' to the actor,
// and have it publish to the stream
case p:ByteString => filterRef ! p
}
}
}
}
// this actor is supposed to simply uppercase all
// input and write it to the stream
class Filter extends ActorPublisher[ByteString] with Actor
{
var buf = Vector.empty[ByteString]
val delay = 0
def receive = {
case p: ByteString =>
if (buf.isEmpty && totalDemand > 0)
onNext(p)
else {
buf :+= ByteString(p.utf8String.toUpperCase)
deliverBuf()
}
case Request(_) =>
deliverBuf()
case Cancel =>
context.stop(self)
}
#tailrec final def deliverBuf(): Unit =
if (totalDemand > 0) {
if (totalDemand <= Int.MaxValue) {
val (use, keep) = buf.splitAt(totalDemand.toInt)
buf = keep
use foreach onNext
} else {
val (use, keep) = buf.splitAt(Int.MaxValue)
buf = keep
use foreach onNext
deliverBuf()
}
}
}
I've had this problem too before, solved it in a bit of roundabout way, hopefully you're ok with this way. Essentially, it involves creating a sink that immediately forwards the messages it gets to the src actor.
Of course, you can use a direct flow (commented it out), but I guess that's not the point of this exercise :)
object Sample {
def main(args: Array[String]): Unit = {
implicit val system = ActorSystem("sample")
implicit val materializer = ActorMaterializer()
val connections: Source[IncomingConnection,
Future[ServerBinding]] = Tcp().bind("localhost", 8000)
def filterProps = Props[Filter]
connections runForeach { conn =>
val actorRef = system.actorOf(filterProps)
val snk = Sink.foreach[ByteString]{s => actorRef ! s}
val src = Source.fromPublisher(ActorPublisher[ByteString](actorRef))
conn.handleWith(Flow.fromSinkAndSource(snk, src))
// conn.handleWith(Flow[ByteString].map(s => ByteString(s.utf8String.toUpperCase())))
}
}
}
// this actor is supposed to simply uppercase all
// input and write it to the stream
class Filter extends ActorPublisher[ByteString]
{
import akka.stream.actor.ActorPublisherMessage._
var buf = mutable.Queue.empty[String]
val delay = 0
def receive = {
case p: ByteString =>
buf += p.utf8String.toUpperCase
deliverBuf()
case Request(n) =>
deliverBuf()
case Cancel =>
context.stop(self)
}
def deliverBuf(): Unit = {
while (totalDemand > 0 && buf.nonEmpty) {
val s = ByteString(buf.dequeue() + "\n")
onNext(s)
}
}
I want to configure spray http client in a way that controls max number of request that were sent to the server. I need this because server that i'm sending requests to blocks me if there are more then 2 request were sent. I get
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://smallTasks/user/IO-HTTP#151444590]] after [15000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://smallTasks/user/IO-HTTP#151444590]] after [15000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://smallTasks/user/IO-HTTP#151444590]] after [15000 ms]
akka.pattern.AskTimeoutException: Ask timed out on
I need to send thousands of requests but i get blocked after i got responses from ~ 100 requests.
I have this method:
implicit val system = ActorSystem("smallTasks")
implicit val timeout = new Timeout(15.seconds)
import system.dispatcher
def doHttpRequest(url: String): Future[HttpResponse] = {
(IO(Http) ? HttpRequest(GET, Uri(url))).mapTo[HttpResponse]
}
And here i catch responses and retry if it fails(recursively):
def getOnlineOffers(modelId: Int, count: Int = 0): Future[Any] = {
val result = Promise[Any]()
AkkaSys.doHttpRequest(Market.modelOffersUrl(modelId)).map(response => {
val responseCode = response.status.intValue
if (List(400, 404).contains(responseCode)) {
result.success("Bad request")
} else if (responseCode == 200) {
Try {
Json.parse(response.entity.asString).asOpt[JsObject]
} match {
case Success(Some(obj)) =>
Try {
(obj \\ "onlineOffers").head.as[Int]
} match {
case Success(offers) => result.success(offers)
case _ => result.success("Can't find property")
}
case _ => result.success("Wrong body")
}
} else {
result.success("Unexpected error")
}
}).recover { case err =>
if (count > 5) {
result.success("Too many tries")
} else {
println(err.toString)
Thread.sleep(200)
getOnlineOffers(modelId, count + 1).map(r => result.success(r))
}
}
result.future
}
How to do this properly? May be i need to configure akka dispatcher somehow?
you can use http://spray.io/documentation/1.2.2/spray-client/ and write you personal pipeline
val pipeline: Future[SendReceive] =
for (
Http.HostConnectorInfo(connector, _) <-
IO(Http) ? Http.HostConnectorSetup("www.spray.io", port = 80)
) yield sendReceive(connector)
val request = Get("/segment1/segment2/...")
val responseFuture: Future[HttpResponse] = pipeline.flatMap(_(request))
to get HttpResponse
import scala.concurrent.Await
import scala.concurrent.duration._
val response: HttpResponse = Aweit(responseFuture, ...)
to convert
import spray.json._
response.entity.asString.parseJson.convertTo[T]
to check
Try(response.entity.asString.parseJson).isSuccess
too many brackets. In scala you can write it shorter
I'm using Play 2.3.7 with Scala 2.11.4, Java 7. I want to use Play WS to connect to an HTTPS endpoint, that requires client to present its certificate. To do it I create my own SSLContext:
val sslContext = {
val keyStore = KeyStore.getInstance("pkcs12")
keyStore.load(new FileInputStream(clientKey), clientKeyPass.to[Array])
val kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm)
kmf.init(keyStore, clientKeyPass.to[Array])
val trustStore = KeyStore.getInstance("jks")
trustStore.load(new FileInputStream(trustStoreFile), trustStorePass.to[Array])
val tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm)
tmf.init(trustStore)
val ctx = SSLContext.getInstance("TLSv1.2")
ctx.init(kmf.getKeyManagers, tmf.getTrustManagers, new SecureRandom())
ctx
}
I know, that the SSLContext is valid, because I can use it with URLConnection successfully:
def urlConnection = Action {
val conn = new URL(url).openConnection()
conn.asInstanceOf[HttpsURLConnection].setSSLSocketFactory(sslContext.getSocketFactory)
conn.connect()
Ok(scala.io.Source.fromInputStream(conn.getInputStream).getLines().mkString("\n"))
}
But when I try one of two ways below I get java.nio.channels.ClosedChannelException.
def ning = Action.async {
val builder = new AsyncHttpClientConfig.Builder()
builder.setSSLContext(sslContext)
val client = new NingWSClient(builder.build())
client.url(url).get() map { _ => Ok("ok") }
}
def asyncHttpClient = Action {
val builder = new AsyncHttpClientConfig.Builder()
builder.setSSLContext(sslContext)
val httpClient = new AsyncHttpClient(builder.build())
httpClient.prepareGet(url).execute().get(10, TimeUnit.SECONDS)
Ok("ok")
}
I also get the same exception when I go after suggestion of Will Sargent and use NingAsyncHttpClientConfigBuilder with parsed config (note, that config references exactly the same values, the hand-crafted sslContext does).
def ningFromConfig = Action.async {
val config = play.api.Configuration(ConfigFactory.parseString(
s"""
|ws.ssl {
| keyManager = {
| stores = [
| { type: "PKCS12", path: "$clientKey", password: "$clientKeyPass" }
| ]
| }
| trustManager = {
| stores = [
| { type: "JKS", path: "$trustStoreFile", password: "$trustStorePass" },
| ]
| }
|}
|# Without this one I get InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
|ws.ssl.disabledKeyAlgorithms="RSA keySize < 1024"
""".stripMargin))
val parser = new DefaultWSConfigParser(config, play.api.Play.application.classloader)
val builder = new NingAsyncHttpClientConfigBuilder(parser.parse())
val client = new NingWSClient(builder.build())
client.url(url).get() map { _ => Ok("ok") }
}
How to make it work with Play WS?
You should use a client certificate from application.conf as defined in https://www.playframework.com/documentation/2.3.x/KeyStores and https://www.playframework.com/documentation/2.3.x/ExampleSSLConfig -- if you need to use a custom configuration, you should use the parser directly:
https://github.com/playframework/playframework/blob/2.3.x/framework/src/play-ws/src/test/scala/play/api/libs/ws/DefaultWSConfigParserSpec.scala#L14
to create a config, then the Ning builder:
https://github.com/playframework/playframework/blob/2.3.x/framework/src/play-ws/src/test/scala/play/api/libs/ws/ning/NingAsyncHttpClientConfigBuilderSpec.scala#L33
which will give you the AHCConfig object that you can pass in. There's examples in the "Using WSClient" section of https://www.playframework.com/documentation/2.3.x/ScalaWS.