Jetty rejects connection even when there are lot of free threads available in thread pool - jetty

Question

In production we are using jetty 9.2.9.v20150224 with the following configuration:
new Server(new QueuedThreadPool(200, 5, 30000, new ArrayBlockingQueue<Runnable>(128)));
maxThreads = 200, minThreads = 5, idleTimout = 30000 ms
I tested our application and it was able to handle 200 requests/sec, but few of our clients complain that sometime even with very less load jetty does not accept any new connection and from the log that's what I found STARTED,5<=18<=200,i=4,q=128. Which as far as I understand shows there are 18 live threads in the thread pool which is way less then 200, but somehow the new connection is still rejected. Following is the snipped from the log:
2021-01-08 00:20:47,813 WARN [qtp1720926658-14444] QueuedThreadPool : QueuedThreadPool[qtp1720926658]#669341c2{STARTED,5<=18<=200,i=4,q=128}[ReservedThreadExecutor#58a5954d{s=3/4,p=1}] rejected org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint#263c181
2021-01-08 00:20:47,813 WARN [qtp1720926658-14444] QueuedThreadPool : QueuedThreadPool[qtp1720926658]#669341c2{STARTED,5<=18<=200,i=3,q=128}[ReservedThreadExecutor#58a5954d{s=3/4,p=1}] rejected org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint#431081c5
2021-01-08 00:20:47,813 WARN [qtp1720926658-14445] QueuedThreadPool : QueuedThreadPool[qtp1720926658]#669341c2{STARTED,5<=18<=200,i=2,q=128}[ReservedThreadExecutor#58a5954d{s=3/4,p=1}] rejected org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint#3866d8cd
2021-01-08 00:20:47,813 WARN [qtp1720926658-14442] QueuedThreadPool : QueuedThreadPool[qtp1720926658]#669341c2{STARTED,5<=18<=200,i=6,q=126}[ReservedThreadExecutor#58a5954d{s=1/4,p=2}] rejected CEP:SocketChannelEndPoint#4aa81d65{/10.0.1.29:53143<->/10.0.1.28:7777,OPEN,fill=FI,flush=-,to=16442/30000}{io=1/0,kio=1,kro=1}->HttpConnection#32cdb0b1[p=HttpParser{s=START,0 of -1},g=HttpGenerator#22c521c0{s=START}]=>HttpChannelOverHttp#e02e50f{r=1,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
2021-01-08 00:20:47,829 WARN [qtp1720926658-14442] EatWhatYouKill :
java.util.concurrent.RejectedExecutionException: CEP:SocketChannelEndPoint#4aa81d65{/10.0.1.29:53143<->/10.0.1.28:7777,OPEN,fill=FI,flush=-,to=16442/30000}{io=1/0,kio=1,kro=1}->HttpConnection#32cdb0b1[p=HttpParser{s=START,0 of -1},g=HttpGenerator#22c521c0{s=START}]=>HttpChannelOverHttp#e02e50f{r=1,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:440)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.execute(EatWhatYouKill.java:370)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:305)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

Answer 1

STARTED,5<=18<=200,i=4,q=128 says ...
5 is your configured minimum threads
<=18<= is your remaining threads available in the pool
200 is your configured max threads
i=4 you have 4 idle threads in the pool (eligible for cleanup/removal)
q=128 you have 128 tasks/jobs in the queue (this value is too high, it hints at your QTP configuration being insufficient for your load)
Keep in mind that when you experience the rejected tasks/jobs it can be the queue or the pool that rejects them, we don't know which one in Jetty 9.2.x (this information is now available in the logs on Jetty 9.4.x)
This limited view of the QTP in the Jetty 9.2.x series does not show you even half of what's going on internally.
Example: in Jetty 9.4.x+ you see ...
Configured Minimum
Configured Maximum
idle thread count
active thread count
busy thread count
inactive thread count
reserved thread count
leased thread count
utilized count = threads - idle - reserved - leased
max threads possible = inactive + idle + reserved + leased + utilized
threads total = idle + reserved + leased + utilized
busy threads = leased + utilized
ready threads = idle + reserved
available = max - busy
maxAvailable = max - leased
utilization = utilized / maxAvailable (0.0 means idle, 1.0 means at capacity)
All of which are exposed in JMX as well.