I am trying to implement an HTTP client in my Akka app in order to consume a 3rd party API.
What I am trying to configure are the timeout and the number of retries in case of failure.
Is the below code the right approach to do it?
val timeoutSettings =
ConnectionPoolSettings(config).withIdleTimeout(10 minutes)
.withMaxConnections(3)
val responseFuture: Future[HttpResponse] =
Http().singleRequest(
HttpRequest(
uri = "https://api.com"
),
timeoutSettings
)
That is not the right approach (for below I refer to the settings via .conf file rather than the programmatic approach, but that should correspond easily).
idle-timeout corresponds to the
time after which an idle connection pool (without pending requests)
will automatically terminate itself
on the pool level, and on akka.http.client level to
The time after which an idle connection will be automatically closed.
So you'd rather want the connection-timeout setting.
And for the retries its the max-retries setting.
The max-connections setting is only:
The maximum number of parallel connections that a connection pool to a
single host endpoint is allowed to establish
See the official documentation
Related
Let say I have a webservice (WCF and ASMX .net framework 4.8) which is hosted on IIS 10. Webservice has a method with this content:
public CustomerListResponse Get(CustomerListRequest request)
{
//sleep for 1 hour
System.Threading.Thread.Sleep(TimeSpan.FromHours(1));
return new CustomerListResponse();
}
The line that is performing sleep on thread is just to show that there is code that in some cases can take long time.
What I'm looking is setting or way to limit allowed processing time for example to one minute and error returned to client. I want the processing be killed by IIS/WCF/ASMX if the execution time will exceed one minute.
Unfortunately I didn't found a way in IIS for that. Also I don't have access to client code to set this limit on other side - change is possible only on server side.
What I tried:
on binding for WCF there is couple of properties openTimeout="00:01:00" closeTimeout="00:01:00" sendTimeout="00:01:00" receiveTimeout="00:01:00" - I set them all but it didn't work. Code can still process for long time.
<httpRuntime targetFramework="4.8" executionTimeout="60" /> - also didn't work
I don't have other ideas how to achieve that, but I believe there should be some solution to be able control how long we want to spend on processing.
You need to set the timeout on both client-side and server-side.
Client-side:
SendTimeout is used to initialize OperationTimeout, which manages the entire interaction of sending a message (including receiving a reply message in a request-reply case). This timeout also applies when a reply message is sent from the CallbackContract method.
OpenTimeout and CloseTimeout are used to open and close channels (when no explicit timeout value is passed).
ReceiveTimeout not used.
Server-side:
Send, open, and close timeouts are the same as on the client side (for callbacks).
ReceiveTimeout is used by the ServiceFramework layer to initialize idle session timeouts.
We have a Reactive REST API using Spring Data Neo4j (SpringBoot v2.7.5) deployed to Kubernetes. When running a stress test to determine the breaking point, once the volume of requests that the service can handle has been breached, the service does not auto-recover, even after the load has dropped to a level at which the service can handle.
After the service has fallen over the Neo4J health indicator shows:
“org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms”
With respect to connection/configuration settings we are using defaults configured by SDN.
Observations:
Up until the point at which the service breaks only a small number of connections are utilised, at the point at which it breaks the connections in use jumps up to the max pool size and the above mentioned error is observed. No matter how much time passes (even well beyond the max connection lifetime) the service is unable to acquire a connection from the pool. Upon manually shutting down and restarting the service/pod the service returns to a healthy state.
As an interim solution we now check the Neo4J health indicator, if the mentioned error is present the liveness state is set to down which triggers Kubernetes to restart the service automatically. However, I’m wondering if there is an underlying issue with the connections in the pool not getting ‘cleaned up’?
You can take a look at this discussion https://github.com/spring-projects/spring-data-neo4j/issues/2632
I had the same issue. The problem is that either Spring Framework or Neo4j reactive transaction manager doesn't close connections in a complex reactive flow e.g. when there are a lot of inner calls/mappings and somewhere inside an exception is thrown.
So as a workaround you can add #Transactional in such places to avoid multiple transactions to be created.
I am using the below AWS S3 AsyncClient from my Spring webflux application.As part of performance testing I had to introduce eventLoopGroup in S3 Client Config below to enable the S3 operations when the request rate was really high with 50 concurrent users triggering 20 request each having 5 documents to upload to S3.Before this I was getting the below error
reactor.core.publisher.Operators : Operator called default onErrorDropped\njava.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.\nConsider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate.\nIncreasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process
Introducing eventLoopGroup with numberOfThreads as 15 drastically reduced the error count to 6 from around a 1000. However once it fails, it didn't allow the next requests and failed with the same error.I was assuming that after sometime the used threads will be returned to the thread pool and would be available for use.But I have to restart the spring webflux application to get rid of the errors.Kindly let me know that the changes I have made below are correct.
#Bean
public S3AsyncClient s3client() {
SdkAsyncHttpClient httpClient =
NettyNioAsyncHttpClient.builder()
.readTimeout(Duration.ofSeconds(30))
.writeTimeout(Duration.ZERO)
.maxConcurrency(300)
.eventLoopGroup(SdkEventLoopGroup.builder().numberOfThreads(15).build())
.connectionAcquisitionTimeout(Duration.ofSeconds(30))
.build();
S3Configuration serviceConfiguration =
S3Configuration.builder()
.checksumValidationEnabled(false)
.chunkedEncodingEnabled(true)
.build();
S3AsyncClientBuilder s3AsyncClientBuilder =
S3AsyncClient.builder()
.httpClient(httpClient)
.region(Region.AP_SOUTHEAST_2)
.serviceConfiguration(serviceConfiguration);
return s3AsyncClientBuilder.build();
}
2019-08-01 06:04:43,263 | ERROR | Could not accept connection :
org.apache.activemq.transport.tcp.ExceededMaximumConnectionsException:
Exceeded the maximum number of allowed client connections. See the
'maximumConnections' property on the TCP transport configuration URI
in the ActiveMQ configuration file (e.g., activemq.xml) |
org.apache.activemq.broker.TransportConnector | ActiveMQ Transport
Server Thread Handler:
nio+ssl://b-e13f27f2-1fa3-419f-819c-a24277e973a8-2.mq.us-west-2.amazonaws.com:61617?maximumConnections=100&wireFormat.maxFrameSize=104857600
Getting above exception on amazonMQ, earlier we were using activeMQ where we were setting something like
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600"/>
</transportConnectors>
In amazonMQ we are unable to find such options and broker is throwing exception. We did checked transportConnector on amazonMQ supports :
name
updateClusterClients
rebalanceClusterClients
updateClusterClientsOnRemove
Any idea how can we increase size of maximum connections?
As listed here that limit can be changed per AWS account.
You will need to open up an AWS support ticket requesting a limit increase
I guess I have to ask, Why so many connections?
Large has 1000 connections with Micro allowing 100 connections. Seeing in your error message you have 100 connections, are you on Micro? Maybe a Micro instance can't handle the load?
Are the producers/consumers something you control or is this a third party app? I would review code before increasing these levels if that is something you can do. Connections are to be shared as much as they can be. Are they being closed correctly when done? Are all your producers opening and maintaining their own connections?
Producer Connections should be grouped and shared with the PooledConnectionFactory.
Consider routes containing all the HTTP services
val routes:Route = ...
I wish to throttle number of requests so I used Route.handleFlow(routes) to create flow and called throttle method with finite duration.
Finally, I created HTTP binding using
Http().bindAndHandle(flowObjectAfterThrottling, hostname, port)
When HTTP requests are fired from a loop throttling is not obeyed by akka.
One possibility is that the http requests being "fired from a loop" may be using separate connections. Each incoming connection is throttling at the appropriate rate but the aggregate throughput is higher than expected.
Use Configurations Instead
You don't need to write software to set limiting rates for your Route.
If you are only concerned with consumption of a resource, such as disk or RAM, then you can remove the rate logic and use akka configuration settings instead:
# The maximum number of concurrently accepted connections when using the
# `Http().bindAndHandle` methods.
max-connections = 1024
# The maximum number of requests that are accepted (and dispatched to
# the application) on one single connection before the first request
# has to be completed.
pipelining-limit = 16
This doesn't provide the ability to set a maximum frequency, but it does at least allow for the specification of a maximum concurrent usage which is usually sufficient for resource protection.