HttpHandler low concurrent request processing - concurrency

I have a simple .NET 4.5 HttpHandler that does not seem to scale beyond 10 concurrent requests. I'm expecting a much higher figure. The handler does nothing more than sleep for a second and return a simple string.
concurrent requests requests/minute
1 60
8 480
9 540
10 600
11 600
12 600
15 600
32 600
512 600
I've got a 64-bit Win7 machine with a 4-core i7 with 32GB of ram and an SSD, and the machine is idle during all the tests. IIS is as configured out of the box. I've also tried running this with an app pool created by IISTuner from Codeplex, but with no change in result.
Changing IsReusable between true/false does nothing.
I've added an entry to web.config to disable session state. I'm using SoapUI for the testing, and have the close connections after each request flag set (and can see this reflected in the http headers). Again, nothing seems to change.
Amending the number of processes per app pool does raise the number, but still gives nowhere near the throughput I'm expecting (hundreds/thousands of concurrent requests).
Here is the handler:
class TestHandler : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
Thread.Sleep(1000);
context.Response.Write("I slept for 1s");
}
public bool IsReusable { get { return true; } }
}
Here is the associated web.config file:
<?xml version="1.0"?>
<configuration>
<system.web>
<compilation debug="true" targetFramework="4.5" />
<httpRuntime targetFramework="4.5" />
<sessionState mode="Off" timeout ="20" cookieless="false"></sessionState>
</system.web>
<system.webServer>
<handlers>
<add name="Proposal01_Test" verb="*"
path="*.test"
type="Proposal01.TestHandler, Proposal01"
resourceType="Unspecified" />
</handlers>
</system.webServer>
</configuration>
What am I missing?

The problem is as #Damien states - thank you again - this is a non-server OS limitation in Win7/IIS7.5, which I never picked up on as we've had Java services on the same machine scale into the thousands of concurrent request territory. Yey MS.

Related

Request time out 500 in asp.net4.5

When CPU Utilisation riches 70% to 80 % at that time when user call web service method for e.g like time 12.01.00 but it riches to database server after one minute like 12.02.00 and database server gives response in fraction of milliseconds for that request and we receive this error System.Web.HttpException (0x80004005): Request timed out in elmah .There is a time difference between receiving request at web service method and sending it to database. When CPU normal then everything works smoothly within fraction of seconds.I don't understand why this one min time difference occurring between received request at .net side and sending it to database. After reset iis every thing work normal .This happens only production server but not in development and UAT Server.
IIS version:-8.5 and sql 2012
In web.config update httpRuntime executionTimeout value:
<configuration>
<system.web>
<httpRuntime executionTimeout="300" />
</system.web>
</configuration>
executionTimeout: Specifies the maximum number of seconds that a request is allowed to execute before being automatically shut down by ASP.NET.The default is 110 seconds.

AppFabric session state provider - LRU eviction has been disabled for the cache used to store Session

I'm using AppFabric to share session between 2 or more different Web Applications.
But I got a problem which said: "Expected the Session item 'FULL_NAME' to be present in the cache, but it could not be found, indicating that the current Session is corrupted. Please verify that LRU eviction has been disabled for the cache used to store Session."
My configuration:
<dataCacheClient>
<hosts>
<host name="CACHE1" cachePort="22233" />
<host name="CACHE2" cachePort="22233" />
<host name="CACHE3" cachePort="22233" />
</hosts>
<machineKey validationKey="C7415df6847D0C0B5146F5605B5973EBD59kjh67EE6414ECD534A95F528F153B6B5F42CFFA9EBF65B2169F7DA5D801C0F9053454A159505253DC687A" decryptionKey="3AE9EE73F1A2781B73DEC6C3fgdgdfD28E0C730284DD314118DA8B" validation="SHA1" decryption="AES" />
<sessionState timeout="40" mode="Custom" customProvider="AFCacheSessionStateProvider">
<providers>
<add name="AFCacheSessionStateProvider" type="Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider, Microsoft.Web.DistributedCache" cacheName="XXXXX" shareID="YYYYY" retryCount="10" useBlobMode="false" />
</providers>
</sessionState>
is there anyone know what is problem?
It's not the asp.net config you need to check, it's the cache itself.
On a cache host, see the output of the command Get-CacheConfig XXXXX - as the docs say you will see output like this: check the EvictionType:
CacheName : Cache1
TimeToLive : 10 mins
CacheType : Partitioned
Secondaries : 0
IsExpirable : True
EvictionType : LRU
NotificationsEnabled : False
For more information on eviction settings, see the Expiration and Eviction documentation. If your eviction type is not LRU, check cache memory usage, as that page states:
When the memory consumption of the cache service on a cache server
exceeds the low watermark threshold, AppFabric starts evicting objects
that have already expired. When memory consumption exceeds the high
watermark threshold, objects are evicted from memory, regardless of
whether they have expired or not, until memory consumption goes back
down to the low watermark.
There's an expiration troubleshooting page with more info.

CXF java.net.ConnectException: Connection timed out

I am getting a connection timed out when I try to invoke from a WS client a method from a CXF Web service I have deployed. Both are using custom interceptors, and the service is overloaded due to multiple invocations.
Caused by: java.net.ConnectException: ConnectException invoking http://xxx.xx.xx.xx:12005/myservice/repository?wsdl: Connection timed out
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1338)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1322)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:622)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
... 36 more
I tried multiple solutions to disabled the timeout or to increase it but all failed.
First, I tried to create a CXF configuration file like the following:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:http-conf="http://cxf.apache.org/transports/http/configuration"
xsi:schemaLocation="http://cxf.apache.org/transports/http/configuration
http://cxf.apache.org/schemas/configuration/http-conf.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<http-conf:conduit name="*.http-conduit">
<http-conf:client CacheControl="no-cache"
ConnectionTimeout="0" ReceiveTimeout="0" AllowChunking="false" />
</http-conf:conduit>
</beans>
Then, I forced my application to load it by using the Java system property -Dcxf.config.file=/home/test/resources/cxf.xml
In the logs I can see that the configuration is read and thus probably applied
INFO: Loaded configuration file /home/test/resources/cxf.xml.
Unfortunately the connection timed out still occurs.
The second solution I tried consists of setting the policy programmatically on all the clients by using the following piece of code:
public static void setHTTPPolicy(Client client) {
HTTPConduit http = (HTTPConduit) client.getConduit();
HTTPClientPolicy httpClientPolicy = new HTTPClientPolicy();
httpClientPolicy.setConnectionTimeout(0);
httpClientPolicy.setReceiveTimeout(0);
httpClientPolicy.setAsyncExecuteTimeout(0);
http.setClient(httpClientPolicy);
}
but again the connection timeout occurs.
Do I miss something? Is there some other timeouts to configure? any help is welcome.
CXF allows you to configure threadpooling for your webservice endpoint. This way, you can cater for timeouts occurring as a result of scarce request processing resources. Below is a sample config using the <jaxws:endpoint/> option in cxf:
<jaxws:endpoint id="serviceBean" implementor="#referenceToServiceBeanDefinition" address="/MyEndpointAddress">
<jaxws:executor>
<bean id="threadPool" class="java.util.concurrent.ThreadPoolExecutor">
<!-- Minimum number of waiting threads in the pool -->
<constructor-arg index="0" value="2"/>
<!-- Maximum number of working threads in the pool -->
<constructor-arg index="1" value="5"/>
<!-- Maximum wait time for a thread to complete execution -->
<constructor-arg index="2" value="400000"/>
<!-- Unit of wait time -->
<constructor-arg index="3" value="#{T(java.util.concurrent.TimeUnit).MILLISECONDS}"/>
<!-- Storage data structure for waiting thread tasks -->
<constructor-arg index="4" ref="taskQueue"/>
</bean>
</jaxws:executor>
</jaxws:endpoint>
<!-- Basic data structure to temporarily hold waiting tasks-->
<bean id="taskQueue" class="java.util.concurrent.LinkedBlockingQueue"/>

AppFabric Cluster

I am having an issue with AppFabric in a load balanced environment. For some reason when I start the AppFabric cluster, I don’t see anything listening on the ports (22233). Below is our config file:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="dataCache" type="Microsoft.ApplicationServer.Caching.DataCacheSection, Microsoft.ApplicationServer.Caching.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
</configSections>
<dataCache size="Small">
<hosts>
<host replicationPort="22236" arbitrationPort="22235" clusterPort="22234"
hostId="2035002654" size="4095" leadHost="true" account="ServiceAccount"
cacheHostName="AppFabricCachingService" name="Server2"
cachePort="22233" />
<host replicationPort="22236" arbitrationPort="22235" clusterPort="22234"
hostId="434992406" size="4095" leadHost="true" account="ServiceAccount"
cacheHostName="AppFabricCachingService" name="Server1"
cachePort="22233" />
</hosts>
</dataCache>
</configuration>
If I change the config to make ONE of the servers a leadHost, that host now listens on port 22233. But in order to keep high availability, I need both up and running.
As far as i know about AppFabric, you must have a server configured as a leadhost. Also there has to be only 1 leadhost in the cache cluster.
Please let me know if i am wrong.
Thanks and Regards
Gagan Janjua

Jetty is getting stucked during high-load tests

I have a web service composed by 2 Jettys (running the same content) load-balanced by a HA Proxy. During a test that consists in a medium requests per second rate (less than 100) and each request having a big body (21 KB), Jetty gets stucked -- It doesnt respond to any request.
The only way to bring Jetty up is restarting it.
I didn't find any information in log files (2011_05_20.stderrout.log, 2011_05_20.log) -- It seems to stop logging.
There are any other useful log files that I should enable in Jetty configs ?
Have anyone ever experienced this weird behaviour ?
Could I retrieve some info about thread status from Jetty (I'm not sure if all threads are busy, the request is rejected) ?
Thanks in advance!
How many threads have you specified in jetty.xml ? I think per default (at least for embedded Jetty), the maximum number of threads is set at around 50. You can change this either programatically, or via jetty.xml. Rather than just setting max to a high number, you should figure out a correct value, depending on server resources and load requirements.
<Configure id="Server" class="org.eclipse.jetty.server.Server">
<!-- =========================================================== -->
<!-- Server Thread Pool -->
<!-- =========================================================== -->
<Set name="ThreadPool">
<New class="org.eclipse.jetty.util.thread.QueuedThreadPool">
<!-- initial threads set to 50 -->
<Set name="minThreads">50</Set>
<!-- the thread pool will grow only up to 768 -->
<Set name="maxThreads">768</Set>
</New>
</Set>
</Configure>
Use something like jVisualVM or BTrace to find out how many threads that are in your threadpool. Heres a link to a BTrace script to print out thread counts: https://github.com/joachimhs/EurekaJ/blob/master/EurekaJ.Scripts/btrace/1.2/btraceScripts/ThreadCounter.java