How to find the reason for a jetty 400 error? - jetty

I have a jetty server (version 9.4.11. It is used in combination with org.restlet-2.4.3). A client is sending a POST message to an api on my server (route "/pos/v1/purchasable_items") and jetty responds with 400 code. I cannot figure out the reason. I would really appreciate some help.
Here is the full log of jetty:
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Queued change org.eclipse.jetty.io.ManagedSelector$Accept#616b6a22 on org.eclipse.jetty.io.ManagedSelector#79c26be5 id=1 keys=0 selected=0 updates=0
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - wakeup on submit org.eclipse.jetty.io.ManagedSelector#79c26be5 id=1 keys=0 selected=0 updates=1
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e woken with none selected
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e woken up from select, 0/0/0 selected
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e processing 0 keys, 1 updates
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - updateable 1
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - update org.eclipse.jetty.io.ManagedSelector$Accept#616b6a22
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - queue org.eclipse.jetty.io.ManagedSelector$Accept#616b6a22
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - updates 0
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e waiting with 1 keys
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - run org.eclipse.jetty.io.ManagedSelector$Accept#616b6a22
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.IdleTimeout - SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=8/100000}{io=0/0,kio=0,kro=0}-><null> idle timeout check, elapsed: 7 ms, remaining: 99993 ms
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.PreEncodedHttpField - HttpField encoders loaded: [org.eclipse.jetty.http.Http1FieldPreEncoder#1525e6cb]
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - new HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0} -> SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=72/100000}{io=0/0,kio=0,kro=0}-><null>,null,HttpChannelState#7ae84ea1{s=IDLE a=NOT_ASYNC i=true r=IDLE w=false}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - New HTTP Connection HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=80/100000}{io=0/0,kio=0,kro=0}-><null>
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.AbstractEndPoint - onOpen SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=81/100000}{io=0/0,kio=0,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.AbstractConnection - onOpen HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=81/100000}{io=0/0,kio=0,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.AbstractConnection - fillInterested HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=81/100000}{io=0/0,kio=0,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.FillInterest - interested FillInterest#72454313{AC.ReadCB#30023a7c{HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=0/100000}{io=0/0,kio=0,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}}}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - changeInterests p=false 0->1 for SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=0/100000}{io=0/1,kio=0,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Queued change org.eclipse.jetty.io.ChannelEndPoint$1#528ccae5 on org.eclipse.jetty.io.ManagedSelector#79c26be5 id=1 keys=1 selected=0 updates=0
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - wakeup on submit org.eclipse.jetty.io.ManagedSelector#79c26be5 id=1 keys=1 selected=0 updates=1
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e woken with none selected
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e woken up from select, 0/0/1 selected
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e processing 0 keys, 1 updates
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - updateable 1
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - update org.eclipse.jetty.io.ChannelEndPoint$1#528ccae5
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Created SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=0/100000}{io=0/1,kio=0,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - ran org.eclipse.jetty.io.ManagedSelector$Accept#616b6a22
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - Key interests updated 0 -> 1 on SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=1/100000}{io=1/1,kio=1,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - updates 0
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e waiting with 1 keys
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e woken up from select, 1/1/1 selected
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e processing 1 keys, 0 updates
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - selected 1 channel=java.nio.channels.SocketChannel[connected local=/127.0.0.1:8182 remote=/127.0.0.1:50589], selector=sun.nio.ch.WindowsSelectorImpl#5c1f2b6e, interestOps=1, readyOps=1 SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=1/100000}{io=1/1,kio=1,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - onSelected 1->0 r=true w=false for SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=1/100000}{io=1/0,kio=1,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - task CEP:SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=1/100000}{io=1/0,kio=1,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
12:56:05 [36mDEBUG[0;39m o.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor#77a9ea73{s=0/8,p=0} tryExecute EatWhatYouKill#218cf600/SelectorProducer#4aecb548/PRODUCING/p=false/QueuedThreadPool[qtp103099676]#6252d1c{STARTED,8<=8<=300,i=3,q=0}[ReservedThreadExecutor#77a9ea73{s=0/8,p=0}][pc=0,pic=0,pec=0,epc=0]#2021-06-17T12:56:05.8701778+02:00
12:56:05 [36mDEBUG[0;39m o.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor#77a9ea73{s=0/8,p=1} startReservedThread p=1
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - queue ReservedThreadExecutor#77a9ea73{s=0/8,p=1}#73ca3be9
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - run ReservedThreadExecutor#77a9ea73{s=0/8,p=1}#73ca3be9
12:56:05 [36mDEBUG[0;39m o.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor#77a9ea73{s=1/8,p=1}#73ca3be9 started
12:56:05 [36mDEBUG[0;39m o.e.jetty.util.thread.strategy.EatWhatYouKill - EatWhatYouKill#218cf600/SelectorProducer#4aecb548/PRODUCING/p=false/QueuedThreadPool[qtp103099676]#6252d1c{STARTED,8<=8<=300,i=2,q=0}[ReservedThreadExecutor#77a9ea73{s=0/8,p=1}][pc=0,pic=0,pec=0,epc=0]#2021-06-17T12:56:05.8721778+02:00 m=PRODUCE_EXECUTE_CONSUME t=CEP:SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=3/100000}{io=1/0,kio=1,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING/BLOCKING
12:56:05 [36mDEBUG[0;39m o.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor#77a9ea73{s=1/8,p=0}#73ca3be9 waiting
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - queue CEP:SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=3/100000}{io=1/0,kio=1,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - updateable 0
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - updates 0
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - Key interests updated 1 -> 0 on SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=3/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e waiting with 1 keys
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - run CEP:SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=3/100000}{io=0/0,kio=1,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.FillInterest - fillable FillInterest#72454313{AC.ReadCB#30023a7c{HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=4/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}}}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=4/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0} onFillable enter HttpChannelState#7ae84ea1{s=IDLE a=NOT_ASYNC i=true r=IDLE w=false} null
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - filled 181 SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=6/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=3/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0} filled 181 HeapByteBuffer#25fdf14c[p=0,l=181,c=8192,r=181]={<<<POST /pos/v1/purc...00-continue\r\n\r\n>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=3/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0} parse HeapByteBuffer#25fdf14c[p=0,l=181,c=8192,r=181]={<<<POST /pos/v1/purc...00-continue\r\n\r\n>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} {}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - parseNext s=START HeapByteBuffer#25fdf14c[p=0,l=181,c=8192,r=181]={<<<POST /pos/v1/purc...00-continue\r\n\r\n>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - START --> SPACE1
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - SPACE1 --> URI
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - URI --> SPACE2
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - SPACE2 --> REQUEST_VERSION
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - REQUEST_VERSION --> HEADER
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Content-Type: application/x-www-form-urlencoded --> IN_VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Content-Type: application/x-www-form-urlencoded --> FIELD
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Host --> VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Host --> IN_VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Host --> FIELD
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Content-Length --> VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Content-Length --> IN_VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Content-Length --> FIELD
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Expect --> VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Expect --> IN_VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Expect --> FIELD
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER --> CONTENT
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - REQUEST for //127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05 on HttpChannelOverHttp#3b503229{r=1,c=false,a=IDLE,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=1}
POST //127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05 HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: 127.0.0.1:8182
Content-Length: 228
Expect: 100-continue
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=IDLE,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=2}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=13/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=IDLE,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=2} parsed true HttpParser{s=CONTENT,0 of 228}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - releaseRequestBuffer HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=IDLE,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=2}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=13/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=IDLE,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=2}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp#3b503229{r=1,c=false,a=IDLE,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=3} handle //127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannelState - handling HttpChannelState#7ae84ea1{s=IDLE a=NOT_ASYNC i=true r=IDLE w=false}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=4} action DISPATCH
12:56:05 [36mTRACE[0;39m org.restlet.Component.ServerRouter - Call score for the "org.restlet.routing.VirtualHost#24386839" host: 1.0
12:56:05 [36mDEBUG[0;39m org.restlet.Component.ServerRouter - Default virtual host selected
12:56:05 [36mDEBUG[0;39m org.restlet.Component.ServerRouter - Base URI: "http://127.0.0.1:8182". Remaining part: "/pos/v1/purchasable_items?d=17.06.2021%2012:56:05"
12:56:05 [36mTRACE[0;39m org.restlet.VirtualHost - Call score for the "" URI pattern: 0.5
12:56:05 [36mDEBUG[0;39m org.restlet.VirtualHost - Selected route: "" -> de.sage.scheduler.access.server.AccessServerImpl#440e3ce6
12:56:05 [36mTRACE[0;39m org.restlet.VirtualHost - No characters were matched
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - filled 0 SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=47/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=36}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - filled 0 SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=47/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=36}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=36}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=47/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=36} filled 0 HeapByteBuffer#25fdf14c[p=0,l=0,c=8192,r=0]={<<<>>>POST /pos/v1/purc...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=36}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=48/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=37} parse HeapByteBuffer#25fdf14c[p=0,l=0,c=8192,r=0]={<<<>>>POST /pos/v1/purc...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} {}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - parseNext s=CONTENT HeapByteBuffer#25fdf14c[p=0,l=0,c=8192,r=0]={<<<>>>POST /pos/v1/purc...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=37}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=48/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=37} parsed false HttpParser{s=CONTENT,0 of 228}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - releaseRequestBuffer HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=37}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=48/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=37}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - sendResponse info=HTTP/1.1{s=100,h=-1,cl=-1} content=null complete=false committing=true callback=Blocker#5a2856fc{null}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - COMMIT for /pos/v1/purchasable_items on HttpChannelOverHttp#3b503229{r=1,c=true,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=37}
100 null HTTP/1.1
null
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback#364501c7[PROCESSING][i=HTTP/1.1{s=100,h=-1,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$Commit100Callback#d25f545] generate: NEED_HEADER (null,null,false)#START
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback#364501c7[PROCESSING][i=HTTP/1.1{s=100,h=-1,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$Commit100Callback#d25f545] generate: FLUSH ([p=0,l=25,c=8192,r=25],null,false)#COMPLETING_1XX
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.WriteFlusher - write: WriteFlusher#786527a2{IDLE}->null [HeapByteBuffer#25fdf14c[p=0,l=25,c=8192,r=25]={<<<HTTP/1.1 100 Continue\r\n\r\n>>>items?d=17.06.202...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}]
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.WriteFlusher - update WriteFlusher#786527a2{WRITING}->null:IDLE-->WRITING
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - flushed 25 SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=W,to=55/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=COMPLETING_1XX}]=>HttpChannelOverHttp#3b503229{r=1,c=true,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=44}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.WriteFlusher - Flushed=true written=25 remaining=0 WriteFlusher#786527a2{WRITING}->null
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.WriteFlusher - update WriteFlusher#786527a2{IDLE}->null:WRITING-->IDLE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback#364501c7[PROCESSING][i=HTTP/1.1{s=100,h=-1,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$Commit100Callback#d25f545] generate: DONE ([p=25,l=25,c=8192,r=0],null,false)#START
12:56:09 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - sendResponse info=null content=DirectByteBuffer#3b173fda[p=0,l=84,c=32768,r=84]={<<<{"DESC":"The requ...formed syntax"}>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} complete=false committing=true callback=Blocker#5a2856fc{null}
12:56:09 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - COMMIT for /pos/v1/purchasable_items on HttpChannelOverHttp#3b503229{r=1,c=true,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=3888}
400 null HTTP/1.1
Content-Type: application/json;charset=utf-8
Date: Thu, 17 Jun 2021 10:56:09 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.4.3
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Authorization, Content-Type
Access-Control-Allow-Methods: DELETE, POST, GET, OPTIONS, PUT
I have also tried to debug it with Wireshark without any success. This is the overview of the communication (Port 8182 im by server):
This is the POST request in detail from Wireshark:
I would be happy about anyone giving me a hint.

Your client is sending a POST with an Expect: 100-Continue header (a normal enough concept).
It's a bit hard to tell from your wireshark, but it looks like your client isn't handling the HTTP/1.1 100 Continue response properly.
Your client should wait on sending the body of the request until it receives the HTTP/1.1 100 Continue response.
And when it sends the request body, it should only contain the request body, it should not send the request headers again. (which your logs on jetty, and your wireshark both seem to show).
These headers do not conform to the Content-Type: application/x-www-form-urlencoded and result in a 400 Bad Request response.
Advice, Upgrade both your Restlet version, and your Jetty version.
Judging by the headers sent on the request, you have a very old version of restlet (there are some headers that i'm expecting to see, but do not).
Your Jetty version 9.4.11 is quite old and is missing many important fixes (one of which is more clear error messages around 400 Bad Request, which would have helped you figure this out yourself). You are 3,081 updates (yes, three thousand eighty one updates) behind version 9.4.42.v20210604.

The problem turned out to be a user-agent black list which blocked the request because it had no user-agent. It had nothing to do with jetty.

Related

Http 307 redirect fails for Put Post calls

I am noticing 307 redirect working only for Get calls in jetty httpclient.
For Post/Put redirection are not happening when body present with request.
We are using jetty-client:11.0.11
Update 1
I did poc code to try out the put post calls with httpclient over http2 and was getting response properly.
As in my actual use case we are using Spring webclient + httpclient over http2 , I need to check if that is somehow causing the problem. As of this point no issue with httpclient library.
Update 2
I was able to recreate the issue with Spring Webclient + jetty httpclient
and getting below exception
Multiple subscriptions not supported on AsyncRequestContent
Also this exception is not coming if we are not sending body with the POST/PUT request.
Code snippets below
build.gradle dependencies added
implementation 'org.springframework.boot:spring-boot-starter-webflux'
implementation 'com.darylteo.gradle:javassist-plugin:0.4.1'
implementation 'commons-codec:commons-codec'
implementation 'org.eclipse.jetty:jetty-http:11.0.11'
implementation 'org.eclipse.jetty.http2:http2-http-client-transport:11.0.11'
implementation 'org.eclipse.jetty:jetty-client:11.0.11'
implementation 'org.eclipse.jetty.http2:http2-client:11.0.11'
implementation 'org.eclipse.jetty:jetty-alpn-client:11.0.11'
implementation 'org.eclipse.jetty:jetty-io:11.0.11'
implementation 'org.eclipse.jetty:jetty-util:11.0.11'
implementation 'org.eclipse.jetty:jetty-alpn-java-client:11.0.11'
implementation 'org.eclipse.jetty.http2:http2-common:11.0.11'
implementation 'org.eclipse.jetty.http2:http2-hpack:11.0.11'
implementation 'org.eclipse.jetty:jetty-reactive-httpclient:3.0.6'
HttpClient over http2transport
private HttpClient httpClient;
#PostConstruct
public HttpClient jettyClient(){
SslContextFactory.Client sslClient = new SslContextFactory.Client(true);
ClientConnector connector = new ClientConnector();
connector.setSslContextFactory(sslClient);
HTTP2Client http2Client = new HTTP2Client(connector);
HttpClientTransportOverHTTP2 transportOverHTTP2 = new HttpClientTransportOverHTTP2(http2Client);
httpClient = new HttpClient(transportOverHTTP2);
httpClient.setFollowRedirects(true);
try {
httpClient.start();
} catch (Exception e) {
e.printStackTrace();
}
return httpClient;
}
public HttpClient getHttpClient() {return httpClient;}
RestController
#Value("${sleepTime}")
long sleep;
#Value("${errorCode}")
int errorCode;
#Autowired
Http2Client http2Client;
#RequestMapping("**")
public ResponseEntity<byte[]> post_greet(RequestEntity<byte[]> requestEntity) throws InterruptedException{
System.out.println("Inside Post greet****");
HttpHeaders httpHeaders = requestEntity.getHeaders();
System.out.println("uri:"+ requestEntity.getUrl());
System.out.println("content-type:" + httpHeaders.getContentType());
System.out.println("content-length:" + httpHeaders.getContentLength());
MultiValueMap<String, String> headers = new LinkedMultiValueMap<>();
headers.addAll(requestEntity.getHeaders());
Thread.sleep(sleep);
headers.remove("accept-encoding");
headers.remove("user-agent");
headers.remove("accept");
headers.remove("content-length");
System.out.println("Exit Post greet****");
return new ResponseEntity<byte[]>(requestEntity.getBody(),headers,HttpStatus.OK);
}
#RequestMapping(value = "/redirect")
public Mono<ResponseEntity<byte[]>> redirect(RequestEntity<byte[]> requestEntity){
System.out.println("Got request now redirecting****");
MultiValueMap<String, String> headers = new LinkedMultiValueMap<>();
headers.add("Location", "http://localhost:8787/redirect-url");
return Mono.just(new ResponseEntity<>(headers,HttpStatus.TEMPORARY_REDIRECT));
}
#GetMapping(value = "/start")
public Mono<ResponseEntity<byte[]>> start(RequestEntity<byte[]> requestEntity)
throws InterruptedException, TimeoutException, ExecutionException{
System.out.println("JettyClient start call****");
ClientHttpConnector clientConnector =
new JettyClientHttpConnector(http2Client.getHttpClient());
WebClient webClient = WebClient.builder()
.clientConnector(clientConnector)
.build();
Mono<ResponseEntity<byte[]>> monoResponse = webClient
.method(HttpMethod.POST)
.uri("http://localhost:8787/redirect")
.headers(hdrs->hdrs.add("Content-Type", "application/json"))
.body(BodyInserters.fromValue("\"greetings\":\"Hello Mr Andersen\""))
.exchange()
.flatMap(clientResponse->{
System.out.println("WebClient response code :: " +clientResponse.statusCode());
return clientResponse.toEntity(byte[].class);
})
.onErrorResume(WebClientException.class,clientResponse->{
System.out.println("Exception is :: " + clientResponse.getLocalizedMessage());
ResponseEntity<byte[]> response = new ResponseEntity<byte[]>("Exception occured".getBytes(StandardCharsets.UTF_8),HttpStatus.INTERNAL_SERVER_ERROR);
return Mono.just(response);
})
.doOnCancel(()->System.out.println("Request was cancelled"));
MultiValueMap<String, String> headers = new LinkedMultiValueMap<>();
headers.addAll(requestEntity.getHeaders());
headers.remove("accept-encoding");
headers.remove("user-agent");
headers.remove("accept");
headers.remove("content-length");
System.out.println("JettyClient start end****");
return monoResponse;
}
Logs attached below
2022-10-20 09:28:02.149 INFO 7992 --- [ main] com.example.demo.StubserverApplication : Started StubserverApplication in 1.829 seconds (JVM running for 2.198)
JettyClient start call****
JettyClient start end****
Got request now redirecting****
Exception is :: Multiple subscriptions not supported on
AsyncRequestContent#49b1a991[demand=0,stalled=true,chunks=0]; nested
exception is java.lang.IllegalStateException: Multiple subscriptions not
supported on
AsyncRequestContent#49b1a991[demand=0,stalled=true,chunks=0]
Flows working
Httpclient only (reactive)
Request request = http2Client.getClient().newRequest("http://localhost:8787/redirect").method(HttpMethod.POST);
request.body(new StringRequestContent("\"greetings\":\"Hello Mr Andersen\"")).headers(s->s.add(HttpHeader.CONTENT_TYPE, "application/json"));
ReactiveRequest reactiveRequest = ReactiveRequest.newBuilder(request).build();
Publisher<ReactiveResponse> publisher = reactiveRequest.response();
Httpclient only (non reactive)
ContentResponse response =
http2Client.getClient().POST("http://localhost:8787/redirect"). content(new
StringContentProvider("\"greetings\":\"Hello Mr Andersen\""),
"application/json").send();
Flow not working
Httpclient with Spring webclient reactive
ClientHttpConnector clientConnector =
new JettyClientHttpConnector(http2Client.getHttpClient());
WebClient webClient = WebClient.builder()
.clientConnector(clientConnector)
.build();
Mono<ResponseEntity<byte[]>> monoResponse = webClient
.method(HttpMethod.POST)...
LOGS
2022-10-26 22:33:52.715 DEBUG 10808 --- [ctor-http-nio-3] o.s.w.s.adapter.HttpWebHandlerAdapter : [5d46ae80/1-2] HTTP POST "/redirect"
2022-10-26 22:33:52.722 DEBUG 10808 --- [ctor-http-nio-3] s.w.r.r.m.a.RequestMappingHandlerMapping : [5d46ae80/1-2] Mapped to com.example.demo.ServerController#redirect(RequestEntity)
2022-10-26 22:33:52.722 DEBUG 10808 --- [ctor-http-nio-3] r.r.m.a.HttpEntityMethodArgumentResolver : [5d46ae80/1-2] Content-Type:application/json
2022-10-26 22:33:52.722 DEBUG 10808 --- [ctor-http-nio-3] r.r.m.a.HttpEntityMethodArgumentResolver : [5d46ae80/1-2] 0..1 [byte[]]
2022-10-26 22:33:52.726 DEBUG 10808 --- [ctor-http-nio-3] o.s.core.codec.ByteArrayDecoder : [5d46ae80/1-2] Read 31 bytes
2022-10-26 22:33:52.730 DEBUG 10808 --- [ctor-http-nio-3] o.s.w.s.adapter.HttpWebHandlerAdapter : [5d46ae80/1-2] Completed 307 TEMPORARY_REDIRECT
2022-10-26 22:33:52.739 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.io.ManagedSelector : Selector sun.nio.ch.WEPollSelectorImpl#2962d398 woken up from select, 1/1/1 selected
2022-10-26 22:33:52.739 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.io.ManagedSelector : Selector sun.nio.ch.WEPollSelectorImpl#2962d398 processing 1 keys, 0 updates
2022-10-26 22:33:52.740 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.io.ManagedSelector : selected 1 channel=java.nio.channels.SocketChannel[connected local=/127.0.0.1:62033 remote=localhost/127.0.0.1:8787], selector=sun.nio.ch.WEPollSelectorImpl#2962d398, interestOps=1, readyOps=1 SocketChannelEndPoint#321dd779[{l=/127.0.0.1:62033,r=localhost/127.0.0.1:8787,OPEN,fill=FI,flush=-,to=44/30000}{io=1/1,kio=1,kro=1}]->[HTTP2ClientConnection#2a930f0e]
2022-10-26 22:33:52.741 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.io.SelectableChannelEndPoint : onSelected 1->0 r=true w=false for SocketChannelEndPoint#321dd779[{l=/127.0.0.1:62033,r=localhost/127.0.0.1:8787,OPEN,fill=FI,flush=-,to=45/30000}{io=1/0,kio=1,kro=1}]->[HTTP2ClientConnection#2a930f0e]
2022-10-26 22:33:52.741 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.io.SelectableChannelEndPoint : task SocketChannelEndPoint#321dd779[{l=/127.0.0.1:62033,r=localhost/127.0.0.1:8787,OPEN,fill=FI,flush=-,to=45/30000}{io=1/0,kio=1,kro=1}]->[HTTP2ClientConnection#2a930f0e]:runFillable:EITHER
2022-10-26 22:33:52.741 DEBUG 10808 --- [ient#eadd4fb-20] o.e.j.u.thread.ReservedThreadExecutor : ReservedThreadExecutor#3688eb5b{reserved=0/4,pending=0} tryExecute org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy$$Lambda$482/0x0000000800f16ca0#1424c3b3
2022-10-26 22:33:52.742 DEBUG 10808 --- [ient#eadd4fb-20] o.e.j.u.thread.ReservedThreadExecutor : ReservedThreadExecutor#3688eb5b{reserved=0/4,pending=1} startReservedThread p=1
2022-10-26 22:33:52.742 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.util.thread.QueuedThreadPool : queue ReservedThread#7abbbb35{PENDING,thread=null} startThread=0
2022-10-26 22:33:52.742 DEBUG 10808 --- [ient#eadd4fb-20] o.e.j.u.t.s.AdaptiveExecutionStrategy : ss=PRODUCE_INVOKE_CONSUME t=SocketChannelEndPoint#321dd779[{l=/127.0.0.1:62033,r=localhost/127.0.0.1:8787,OPEN,fill=FI,flush=-,to=47/30000}{io=1/0,kio=1,kro=1}]->[HTTP2ClientConnection#2a930f0e]:runFillable:EITHER/EITHER AdaptiveExecutionStrategy#1b75c2e3/SelectorProducer#1984b1f/PRODUCING/p=false/QueuedThreadPool[HttpClient#eadd4fb]#f1da57d{STARTED,8<=8<=200,i=6,r=-1,q=0}[ReservedThreadExecutor#3688eb5b{reserved=0/4,pending=1}][pc=0,pic=0,pec=0,epc=0]#2022-10-26T22:33:52.7434704+05:30
2022-10-26 22:33:52.742 DEBUG 10808 --- [ient#eadd4fb-22] o.e.jetty.util.thread.QueuedThreadPool : run ReservedThread#7abbbb35{PENDING,thread=null} in QueuedThreadPool[HttpClient#eadd4fb]#f1da57d{STARTED,8<=8<=200,i=6,r=-1,q=0}[ReservedThreadExecutor#3688eb5b{reserved=0/4,pending=1}]
2022-10-26 22:33:52.743 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.io.FillInterest : fillable FillInterest#36fdedce{org.eclipse.jetty.http2.HTTP2Connection$FillableCallback#60296b33}
2022-10-26 22:33:52.745 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.http2.HTTP2Connection : HTTP2 onFillable HTTP2ClientConnection#2a930f0e::SocketChannelEndPoint#321dd779[{l=/127.0.0.1:62033,r=localhost/127.0.0.1:8787,OPEN,fill=-,flush=-,to=49/30000}{io=1/0,kio=1,kro=1}]->[HTTP2ClientConnection#2a930f0e]
2022-10-26 22:33:52.745 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.http2.HTTP2Connection : HTTP2 produce HTTP2ClientConnection#2a930f0e::SocketChannelEndPoint#321dd779[{l=/127.0.0.1:62033,r=localhost/127.0.0.1:8787,OPEN,fill=-,flush=-,to=49/30000}{io=1/0,kio=1,kro=1}]->[HTTP2ClientConnection#2a930f0e]
2022-10-26 22:33:52.746 DEBUG 10808 --- [ient#eadd4fb-20] o.e.j.u.t.s.AdaptiveExecutionStrategy : AdaptiveExecutionStrategy#17f16706/HTTP2Producer#480486a2/IDLE/p=false/QueuedThreadPool[HttpClient#eadd4fb]#f1da57d{STARTED,8<=8<=200,i=6,r=-1,q=0}[ReservedThreadExecutor#3688eb5b{reserved=1/4,pending=0}][pc=0,pic=0,pec=0,epc=0]#2022-10-26T22:33:52.7460585+05:30 tryProduce false
2022-10-26 22:33:52.746 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.http2.HTTP2Connection : Dequeued task null
2022-10-26 22:33:52.746 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.http2.HTTP2Connection : Acquired org.eclipse.jetty.http2.HTTP2Connection$NetworkBuffer#4830c68a
2022-10-26 22:33:52.747 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.io.SocketChannelEndPoint : filled 53 DirectByteBuffer#771ad687[p=0,l=53,c=16384,r=53]={<<<\x00\x00,\x01\x05\x00\x00\x00\x01H\x03307n"http://lo...st:8787/redirect-url\\\x010>>>\r\x82d?S\x83\xF9c\xE7...\x00\x00\x00\x00\x00\x00\x00}
2022-10-26 22:33:52.745 DEBUG 10808 --- [ient#eadd4fb-22] o.e.j.u.thread.ReservedThreadExecutor : ReservedThread#7abbbb35{PENDING,thread=Thread[HttpClient#eadd4fb-22,5,main]} was=PENDING next=RESERVED size=0+1 capacity=4
2022-10-26 22:33:52.751 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.http2.HTTP2Connection : Filled 53 bytes in org.eclipse.jetty.http2.HTTP2Connection$NetworkBuffer#4830c68a
2022-10-26 22:33:52.752 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.http2.parser.Parser : Parsed [HEADERS|44|5|1] frame header from java.nio.DirectByteBuffer[pos=9 lim=53 cap=16384]#d81b37ff
2022-10-26 22:33:52.752 DEBUG 10808 --- [ient#eadd4fb-22] o.e.j.u.thread.ReservedThreadExecutor : ReservedThread#7abbbb35{RESERVED,thread=Thread[HttpClient#eadd4fb-22,5,main]} waiting ReservedThreadExecutor#3688eb5b{reserved=1/4,pending=0}
2022-10-26 22:33:52.754 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.http2.hpack.HpackDecoder : CtxTbl[ee9c664] decoding 44 octets
2022-10-26 22:33:52.754 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.http2.hpack.HpackDecoder : decode 48033330376e22687474703a2f2f6c6f63616c686f73743a383738372f72656469726563742d75726c5c0130
2022-10-26 22:33:52.755 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.http2.hpack.HpackDecoder : decoded ':status: 307' by IdxName/LitVal/Idx
2022-10-26 22:33:52.756 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.http2.hpack.HpackContext : HdrTbl[ee9c664] added {D,0,:status: 307,44a38f23}
2022-10-26 22:33:52.756 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.http2.hpack.HpackContext : HdrTbl[ee9c664] entries=1, size=42, max=4096
2022-10-26 22:33:52.757 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.http2.hpack.HpackDecoder : decode 6e22687474703a2f2f6c6f63616c686f73743a383738372f72656469726563742d75726c5c0130
2022-10-26 22:33:52.757 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.http2.hpack.HpackDecoder : decoded 'location: http://localhost:8787/redirect-url' by IdxName/LitVal/Idx
2022-10-26 22:33:52.757 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.http2.hpack.HpackContext : HdrTbl[ee9c664] added {D,1,location: http://localhost:8787/redirect-url,247b2db0}
2022-10-26 22:33:52.758 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.http2.hpack.HpackContext : HdrTbl[ee9c664] entries=2, size=116, max=4096
2022-10-26 22:33:52.758 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.http2.hpack.HpackDecoder : decode 5c0130
2022-10-26 22:33:52.758 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.http2.hpack.HpackDecoder : decoded 'Content-Length: 0' by IdxName/LitVal/Idx
2022-10-26 22:33:52.758 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.http2.hpack.HpackContext : HdrTbl[ee9c664] added {D,2,Content-Length: 0,5ee2c293}
2022-10-26 22:33:52.759 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.http2.hpack.HpackContext : HdrTbl[ee9c664] entries=3, size=163, max=4096
2022-10-26 22:33:52.759 DEBUG 10808 --- [ient#eadd4fb-20] o.eclipse.jetty.http2.parser.BodyParser : Parsed HEADERS frame hpack from java.nio.DirectByteBuffer[pos=53 lim=53 cap=16384]
2022-10-26 22:33:52.759 DEBUG 10808 --- [ient#eadd4fb-20] o.e.j.http2.client.HTTP2ClientSession : Received HeadersFrame#43014c4d#1{end=true}
2022-10-26 22:33:52.760 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.http2.HTTP2Stream : Update close for HTTP2Stream#54bc89e6#1#6384107{sendWindow=65504,recvWindow=8388608,demand=0,reset=false/false,LOCALLY_CLOSED,age=106,attachment=HttpReceiverOverHTTP2#88410f7(rsp=IDLE,failure=null)} update=true event=RECEIVED
2022-10-26 22:33:52.760 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.http2.HTTP2Session : Removed local HTTP2Stream#54bc89e6#1#6384107{sendWindow=65504,recvWindow=8388608,demand=0,reset=false/false,CLOSED,age=107,attachment=HttpReceiverOverHTTP2#88410f7(rsp=IDLE,failure=null)} from HTTP2ClientSession#6384107{local:/127.0.0.1:62033,remote:localhost/127.0.0.1:8787,sendWindow=65504,recvWindow=16777216,state=[streams=1,NOT_CLOSED,goAwayRecv=null,goAwaySent=null,failure=null]}
2022-10-26 22:33:52.760 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.http2.HTTP2Session : Closed stream HTTP2Stream#54bc89e6#1#6384107{sendWindow=65504,recvWindow=8388608,demand=0,reset=false/false,CLOSED,age=107,attachment=HttpReceiverOverHTTP2#88410f7(rsp=IDLE,failure=null)} for HTTP2ClientSession#6384107{local:/127.0.0.1:62033,remote:localhost/127.0.0.1:8787,sendWindow=65504,recvWindow=16777216,state=[streams=1,NOT_CLOSED,goAwayRecv=null,goAwaySent=null,failure=null]}
2022-10-26 22:33:52.761 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.http2.HTTP2Session : Destroyed stream #1 for HTTP2ClientSession#6384107{local:/127.0.0.1:62033,remote:localhost/127.0.0.1:8787,sendWindow=65504,recvWindow=16777216,state=[streams=1,NOT_CLOSED,goAwayRecv=null,goAwaySent=null,failure=null]}
2022-10-26 22:33:52.769 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpReceiver : Response HttpResponse[HTTP/2.0 307 null]#79d6a430 found protocol handler org.eclipse.jetty.client.RedirectProtocolHandler#3e6aef30
2022-10-26 22:33:52.769 DEBUG 10808 --- [ient#eadd4fb-20] o.eclipse.jetty.client.HttpConversation : Exchanges in conversation 1, override=org.eclipse.jetty.client.RedirectProtocolHandler#3e6aef30, listeners=[org.eclipse.jetty.client.RedirectProtocolHandler#3e6aef30]
2022-10-26 22:33:52.770 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpReceiver : Response begin HttpResponse[HTTP/2.0 307 null]#79d6a430
2022-10-26 22:33:52.772 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpReceiver : Response headers HttpResponse[HTTP/2.0 307 null]#79d6a430
2022-10-26 22:33:52.773 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpReceiver : Response demand=1/1, resume=false
2022-10-26 22:33:52.773 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpReceiver : Response headers hasDemand=true HttpResponse[HTTP/2.0 307 null]#79d6a430
2022-10-26 22:33:52.774 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpReceiver : Response success HttpResponse[HTTP/2.0 307 null]#79d6a430
2022-10-26 22:33:52.775 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpExchange : Terminated response for HttpExchange#40d70396{req=HttpRequest[POST /redirect HTTP/2.0]#7e536042[TERMINATED/null] res=HttpResponse[HTTP/2.0 307 null]#79d6a430[TERMINATED/null]}, result: Result[HttpRequest[POST /redirect HTTP/2.0]#7e536042 > HttpResponse[HTTP/2.0 307 null]#79d6a430] null
2022-10-26 22:33:52.775 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpReceiver : Response complete HttpResponse[HTTP/2.0 307 null]#79d6a430, result: Result[HttpRequest[POST /redirect HTTP/2.0]#7e536042 > HttpResponse[HTTP/2.0 307 null]#79d6a430] null
2022-10-26 22:33:52.776 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpChannel : HttpExchange#40d70396{req=HttpRequest[POST /redirect HTTP/2.0]#7e536042[TERMINATED/null] res=HttpResponse[HTTP/2.0 307 null]#79d6a430[TERMINATED/null]} disassociated true from HttpChannelOverHTTP2#7a176a5e(exchange=null)[send=HttpSenderOverHTTP2#7fca75e4(req=QUEUED,failure=null),recv=HttpReceiverOverHTTP2#88410f7(rsp=IDLE,failure=null)]
2022-10-26 22:33:52.776 DEBUG 10808 --- [ient#eadd4fb-20] o.e.j.h.c.http.HttpChannelOverHTTP2 : exchange terminated Result[HttpRequest[POST /redirect HTTP/2.0]#7e536042 > HttpResponse[HTTP/2.0 307 null]#79d6a430] null HTTP2Stream#54bc89e6#1#6384107{sendWindow=65504,recvWindow=8388608,demand=0,reset=false/false,CLOSED,age=123,attachment=HttpReceiverOverHTTP2#88410f7(rsp=IDLE,failure=null)}
2022-10-26 22:33:52.776 DEBUG 10808 --- [ient#eadd4fb-20] o.e.j.h.c.http.HttpConnectionOverHTTP2 : Released HttpChannelOverHTTP2#7a176a5e(exchange=null)[send=HttpSenderOverHTTP2#7fca75e4(req=QUEUED,failure=null),recv=HttpReceiverOverHTTP2#88410f7(rsp=IDLE,failure=null)]
2022-10-26 22:33:52.777 DEBUG 10808 --- [ient#eadd4fb-20] o.e.j.h.c.http.HttpChannelOverHTTP2 : released channel? true HttpChannelOverHTTP2#7a176a5e(exchange=null)[send=HttpSenderOverHTTP2#7fca75e4(req=QUEUED,failure=null),recv=HttpReceiverOverHTTP2#88410f7(rsp=IDLE,failure=null)]
2022-10-26 22:33:52.777 DEBUG 10808 --- [ient#eadd4fb-20] o.eclipse.jetty.client.HttpDestination : Released HttpConnectionOverHTTP2#3ee779f7(closed=false)[HTTP2ClientSession#6384107{local:/127.0.0.1:62033,remote:localhost/127.0.0.1:8787,sendWindow=65504,recvWindow=16777216,state=[streams=0,NOT_CLOSED,goAwayRecv=null,goAwaySent=null,failure=null]}]
2022-10-26 22:33:52.777 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.client.AbstractConnectionPool : Released (true) MultiEntry#66ffcba0{IDLE,usage=1,multiplex=0,pooled=HttpConnectionOverHTTP2#3ee779f7(closed=false)[HTTP2ClientSession#6384107{local:/127.0.0.1:62033,remote:localhost/127.0.0.1:8787,sendWindow=65504,recvWindow=16777216,state=[streams=0,NOT_CLOSED,goAwayRecv=null,goAwaySent=null,failure=null]}]} #66ab6845[inUse=0,size=1,max=64,closed=false]
2022-10-26 22:33:52.778 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpReceiver : Request/Response succeeded: Result[HttpRequest[POST /redirect HTTP/2.0]#7e536042 > HttpResponse[HTTP/2.0 307 null]#79d6a430] null, notifying [org.eclipse.jetty.client.RedirectProtocolHandler#3e6aef30]
2022-10-26 22:33:52.778 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpRedirector : Redirecting to http://localhost:8787/redirect-url (Location: http://localhost:8787/redirect-url)
2022-10-26 22:33:52.779 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpClient : Resolved HttpDestination[Origin#d3c73938[http://localhost:8787,tag=null,protocol=Protocol#301a9e[proto=[h2c],nego=false]]]#c02e444,queue=0,pool=MultiplexConnectionPool#2ac1c8a2[c=0/1/64,a=0,i=1,q=0] for HttpRequest[POST /redirect-url HTTP/2.0]#5b096284
2022-10-26 22:33:52.779 DEBUG 10808 --- [ient#eadd4fb-20] o.eclipse.jetty.client.HttpConversation : Exchanges in conversation 2, override=null, listeners=[org.eclipse.jetty.client.HttpRequest$8#7e7a4c19, org.eclipse.jetty.client.HttpRequest$8#1c010dcb, org.eclipse.jetty.client.HttpRequest$10#7b322008, org.eclipse.jetty.client.HttpRequest$13#4ebbf3c, org.eclipse.jetty.client.HttpRequest$14#285d313c, org.eclipse.jetty.client.HttpRequest$15#3c260ea9, org.eclipse.jetty.client.HttpRequest$16#567e5b45, ResponseListenerProcessor#5ab8943f[Reactive[HttpRequest[POST /redirect HTTP/2.0]#7e536042]]]
2022-10-26 22:33:52.780 DEBUG 10808 --- [ient#eadd4fb-20] o.eclipse.jetty.client.HttpDestination : Queued HttpRequest[POST /redirect-url HTTP/2.0]#5b096284 for HttpDestination[Origin#d3c73938[http://localhost:8787,tag=null,protocol=Protocol#301a9e[proto=[h2c],nego=false]]]#c02e444,queue=1,pool=MultiplexConnectionPool#2ac1c8a2[c=0/1/64,a=0,i=1,q=1]
2022-10-26 22:33:52.780 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.client.AbstractConnectionPool : Acquiring create=true on MultiplexConnectionPool#2ac1c8a2[c=0/1/64,a=0,i=1,q=1]
2022-10-26 22:33:52.780 DEBUG 10808 --- [ient#eadd4fb-20] o.e.jetty.client.AbstractConnectionPool : Activated MultiEntry#66ffcba0{ACTIVE,usage=2,multiplex=1,pooled=HttpConnectionOverHTTP2#3ee779f7(closed=false)[HTTP2ClientSession#6384107{local:/127.0.0.1:62033,remote:localhost/127.0.0.1:8787,sendWindow=65504,recvWindow=16777216,state=[streams=0,NOT_CLOSED,goAwayRecv=null,goAwaySent=null,failure=null]}]} #66ab6845[inUse=1,size=1,max=64,closed=false]
2022-10-26 22:33:52.781 DEBUG 10808 --- [ient#eadd4fb-20] o.eclipse.jetty.client.HttpDestination : Processing exchange HttpExchange#3586e5e5{req=HttpRequest[POST /redirect-url HTTP/2.0]#5b096284[PENDING/null] res=HttpResponse[null 0 null]#64221408[PENDING/null]} on HttpConnectionOverHTTP2#3ee779f7(closed=false)[HTTP2ClientSession#6384107{local:/127.0.0.1:62033,remote:localhost/127.0.0.1:8787,sendWindow=65504,recvWindow=16777216,state=[streams=0,NOT_CLOSED,goAwayRecv=null,goAwaySent=null,failure=null]}] of HttpDestination[Origin#d3c73938[http://localhost:8787,tag=null,protocol=Protocol#301a9e[proto=[h2c],nego=false]]]#c02e444,queue=0,pool=MultiplexConnectionPool#2ac1c8a2[c=0/1/64,a=1,i=0,q=0]
2022-10-26 22:33:52.781 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpConnection : Normalizing true HttpRequest[POST /redirect-url HTTP/2.0]#5b096284
2022-10-26 22:33:52.782 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpChannel : HttpExchange#3586e5e5{req=HttpRequest[POST /redirect-url HTTP/2.0]#5b096284[PENDING/null] res=HttpResponse[null 0 null]#64221408[PENDING/null]} associated true to HttpChannelOverHTTP2#7a176a5e(exchange=HttpExchange#3586e5e5{req=HttpRequest[POST /redirect-url HTTP/2.0]#5b096284[PENDING/null] res=HttpResponse[null 0 null]#64221408[PENDING/null]})[send=HttpSenderOverHTTP2#7fca75e4(req=QUEUED,failure=null),recv=HttpReceiverOverHTTP2#88410f7(rsp=IDLE,failure=null)]
2022-10-26 22:33:52.782 DEBUG 10808 --- [ient#eadd4fb-20] org.eclipse.jetty.client.HttpSender : Request begin HttpRequest[POST /redirect-url HTTP/2.0]#5b096284
2022-10-26 22:33:52.783 DEBUG 10808 --- [ient#eadd4fb-20] o.eclipse.jetty.client.HttpConversation : Exchanges in conversation 2, override=null, listeners=[org.eclipse.jetty.client.HttpRequest$8#7e7a4c19, org.eclipse.jetty.client.HttpRequest$8#1c010dcb, org.eclipse.jetty.client.HttpRequest$10#7b322008, org.eclipse.jetty.client.HttpRequest$13#4ebbf3c, org.eclipse.jetty.client.HttpRequest$14#285d313c, org.eclipse.jetty.client.HttpRequest$15#3c260ea9, org.eclipse.jetty.client.HttpRequest$16#567e5b45, ResponseListenerProcessor#5ab8943f[Reactive[HttpRequest[POST /redirect HTTP/2.0]#7e536042]]]
2022-10-26 22:33:52.787 DEBUG 10808 --- [ient#eadd4fb-20] o.e.j.r.c.i.ResponseListenerProcessor : response failure HttpResponse[HTTP/2.0 307 null]#79d6a430 on ResponseListenerProcessor#5ab8943f[Reactive[HttpRequest[POST /redirect HTTP/2.0]#7e536042]]
java.lang.IllegalStateException: Multiple subscriptions not supported on AsyncRequestContent#289f2284[demand=0,stalled=true,chunks=0]
at org.eclipse.jetty.client.util.AsyncRequestContent.subscribe(AsyncRequestContent.java:83) ~[jetty-client-11.0.11.jar!/:11.0.11]
at org.eclipse.jetty.reactive.client.internal.PublisherRequestContent.subscribe(PublisherRequestContent.java:47) ~[jetty-reactive-httpclient-3.0.6.jar!/:na]
at org.eclipse.jetty.client.HttpSender.queuedToBegin(HttpSender.java:106) ~[jetty-client-11.0.11.jar!/:11.0.11]
at org.eclipse.jetty.client.HttpSender.send(HttpSender.java:77) ~[jetty-client-11.0.11.jar!/:11.0.11]
at org.eclipse.jetty.http2.client.http.HttpChannelOverHTTP2.send(HttpChannelOverHTTP2.java:102) ~[http2-http-client-transport-11.0.11.jar!/:11.0.11]
at org.eclipse.jetty.client.HttpChannel.send(HttpChannel.java:122) ~[jetty-client-11.0.11.jar!/:11.0.11]
at org.eclipse.jetty.client.HttpConnection.send(HttpConnection.java:110) ~[jetty-client-11.0.11.jar!/:11.0.11]
at org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.send(HttpConnectionOverHTTP2.java:104) ~[http2-http-client-transport-11.0.11.jar!/:11.0.11]
at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:382) ~[jetty-client-11.0.11.jar!/:11.0.11]
at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:358) ~[jetty-client-11.0.11.jar!/:11.0.11]
at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:313) ~[jetty-client-11.0.11.jar!/:11.0.11]
at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:296) ~[jetty-client-11.0.11.jar!/:11.0.11]
at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:290) ~[jetty-client-11.0.11.jar!/:11.0.11]
at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:267) ~[jetty-client-11.0.11.jar!/:11.0.11]
at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:247) ~[jetty-client-11.0.11.jar!/:11.0.11]

Is anyone else have issue deleting AWS Glacier archives?

I've been following the instructions on the AWS Docs https://docs.aws.amazon.com/amazonglacier/latest/dev/getting-started-delete-archive-cli.html and it works up until the last command where I delete the archive by its ID.
The command executes fine but the archive still in the vault, is there a time delay between aws glacier delete-archive and the archive actually getting deleted?
aws --debug glacier delete-archive --vault-name test-vault --account-id 1234567890 --archive-id "XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw"
2021-02-20 16:37:15,290 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=DeleteArchive) with params: {'url_path': '/1234567890/vaults/test-vault/archives/XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw', 'query_string': {}, 'method': 'DELETE', 'headers': {'User-Agent': 'aws-cli/2.0.38 Python/3.7.4 Darwin/20.3.0 exe/x86_64 command/glacier.delete-archive', 'x-amz-glacier-version': '2012-06-01'}, 'body': b'', 'url': ' 'context': {'client_region': 'eu-west-1', 'client_config': <botocore.config.Config object at 0x7fd5e7a9d8d0>, 'has_streaming_input': False, 'auth_type': None}}
2021-02-20 16:37:15,290 - MainThread - botocore.hooks - DEBUG - Event request-created.glacier.DeleteArchive: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fd5e7a9d790>>
2021-02-20 16:37:15,291 - MainThread - botocore.hooks - DEBUG - Event choose-signer.glacier.DeleteArchive: calling handler <function set_operation_specific_signer at 0x7fd5e67118c0>
2021-02-20 16:37:15,291 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-02-20 16:37:15,291 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
DELETE
/1234567890/vaults/test-vault/archives/XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw
host:glacier.eu-west-1.amazonaws.com
x-amz-date:20210220T163715Z
x-amz-glacier-version:2012-06-01
host;x-amz-date;x-amz-glacier-version
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2021-02-20 16:37:15,291 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20210220T163715Z
20210220/eu-west-1/glacier/aws4_request
3ac54f35e66fba13673a078b721c5db02a9f00c538bd96fc5b2d5b5709926b75
2021-02-20 16:37:15,292 - MainThread - botocore.auth - DEBUG - Signature:
1a09aef04ae7830ae5a7623cedb6ca69bf8cdef7349e3f3493a116967ba03110
2021-02-20 16:37:15,292 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=DELETE, url=https://glacier.eu-west-1.amazonaws.com/1234567890/vaults/test-vault/archives/XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw, headers={'User-Agent': b'aws-cli/2.0.38 Python/3.7.4 Darwin/20.3.0 exe/x86_64 command/glacier.delete-archive', 'x-amz-glacier-version': b'2012-06-01', 'X-Amz-Date': b'20210220T163715Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=<Access Key Removed>/20210220/eu-west-1/glacier/aws4_request, SignedHeaders=host;x-amz-date;x-amz-glacier-version, Signature=1a09aef04ae7830ae5a7623cedb6ca69bf8cdef7349e3f3493a116967ba03110', 'Content-Length': '0'}>
2021-02-20 16:37:15,294 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): glacier.eu-west-1.amazonaws.com:443
2021-02-20 16:37:15,510 - MainThread - urllib3.connectionpool - DEBUG - https://glacier.eu-west-1.amazonaws.com:443 "DELETE /1234567890/vaults/test-vault/archives/XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw HTTP/1.1" 204 0
2021-02-20 16:37:15,511 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': 'mR_DR8mtULOgCU0FDDv_aec7eYp34-QtXBONWBtWXjH1yts', 'Date': 'Sat, 20 Feb 2021 16:37:15 GMT'}
2021-02-20 16:37:15,511 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2021-02-20 16:37:15,511 - MainThread - botocore.hooks - DEBUG - Event needs-retry.glacier.DeleteArchive: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fd5e7aef190>>
2021-02-20 16:37:15,512 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2021-02-20 16:37:15,512 - MainThread - botocore.hooks - DEBUG - Event after-call.glacier.DeleteArchive: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fd5e7a9dc50>>
2021-02-20 16:37:15,521 - MainThread - awscli.formatter - DEBUG - RequestId: mR_DR8mtULOgCU0FDDv_aec7eYp34-QtXBONWBtWXjH1yts
It's been over an hour since I executed the delete command and the vault still have the archive on it.
Has anyone had this issue? or does anyone know how I can delete a vault fully?
Update
The console allows me to delete but fails with the vault not being empty.
ps: I've posted this on AWS Forum as well but that's always slow at getting answers, so I'm trying my luck here. https://forums.aws.amazon.com/thread.jspa?threadID=335827
Per Deleting an Archive in Amazon S3 Glacier:
After you delete an archive, if you immediately download the vault
inventory, it might include the deleted archive in the list because S3
Glacier prepares vault inventory only about once a day.

Jetty is throwing exceptions on ktor i think issue with http2

I'm getting these crashes on server using ktor with jetty. On start of jetty server it show exception of idle timeout then after sometime it throw too many other exceptions which i mentioned and server stop working please tell help me to findout this problem thanks.
2020-05-20 11:56:10.566 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.AbstractEndPoint - close DecryptedEndPoint#1066e08c{/152.32.102.11:14898<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=152/30000}
2020-05-20 11:56:10.566 [qtp897848096-2713] DEBUG org.eclipse.jetty.server.HttpChannel - handle https://{HOST}:7171/{ENDPOINT} ServerHttpChannelOverHTTP2#35396382{s=HttpChannelState#715f84bd{s=WOKEN rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WOKEN,uri=https://{HOST}:7171/{ENDPOINT},age=52973}#3
2020-05-20 11:56:10.566 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.AbstractEndPoint - close(null) DecryptedEndPoint#1066e08c{/152.32.102.11:14898<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=152/30000}
2020-05-20 11:56:10.566 [qtp897848096-2713] DEBUG o.e.jetty.server.HttpChannelState - handling HttpChannelState#715f84bd{s=WOKEN rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0}
2020-05-20 11:56:10.566 [qtp897848096-2713] DEBUG org.eclipse.jetty.server.HttpChannel - action ASYNC_ERROR ServerHttpChannelOverHTTP2#35396382{s=HttpChannelState#715f84bd{s=HANDLING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=HANDLING,uri=https://{HOST}:7171/{ENDPOINT},age=52973}#3
2020-05-20 11:56:10.566 [qtp897848096-2875] DEBUG o.e.j.u.t.ReservedThreadExecutor - ReservedThreadExecutor#6a969fb8{s=1/2,p=0}#2b7501a0 task=EatWhatYouKill#47733c6f/HTTP2Producer#53a8567/IDLE/p=true/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=6,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=1/2,p=0}][pc=0,pic=0,pec=0,epc=2]#2020-05-20T11:56:10.566514Z
2020-05-20 11:56:10.566 [qtp897848096-2713] WARN org.eclipse.jetty.server.HttpChannel - handleException /{ENDPOINT}
java.nio.channels.ClosedChannelException: null
at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:926)
at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:283)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Suppressed: java.lang.Throwable: HttpInput failure
at org.eclipse.jetty.server.HttpInput.failed(HttpInput.java:823)
at org.eclipse.jetty.http2.server.HttpChannelOverHTTP2.onFailure(HttpChannelOverHTTP2.java:323)
at org.eclipse.jetty.http2.server.HTTP2ServerConnection.onStreamFailure(HTTP2ServerConnection.java:221)
at org.eclipse.jetty.http2.server.HTTP2ServerConnection.onSessionFailure(HTTP2ServerConnection.java:261)
at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onFailure(HTTP2ServerConnectionFactory.java:128)
at org.eclipse.jetty.http2.HTTP2Session.notifyFailure(HTTP2Session.java:1156)
at org.eclipse.jetty.http2.HTTP2Session.abort(HTTP2Session.java:1060)
... 23 common frames omitted
and also getting these crashes
2020-05-20 11:56:04.979 [Connector-Scheduler-74f6c5d8-1] DEBUG org.eclipse.jetty.io.IdleTimeout - SocketChannelEndPoint#1aea799a{/110.54.160.90:41470<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=30000/30000}{io=1/1,kio=1,kro=1}->SslConnection#662a1f2b{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#4724da13{/110.54.160.90:41470<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=540007/30000}=>HTTP2ServerConnection#4fc8491c idle timeout check, elapsed: 30000 ms, remaining: 0 ms
2020-05-20 11:56:04.979 [Connector-Scheduler-74f6c5d8-1] DEBUG org.eclipse.jetty.io.IdleTimeout - SocketChannelEndPoint#1aea799a{/110.54.160.90:41470<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=30000/30000}{io=1/1,kio=1,kro=1}->SslConnection#662a1f2b{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#4724da13{/110.54.160.90:41470<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=540007/30000}=>HTTP2ServerConnection#4fc8491c idle timeout expired
2020-05-20 11:56:04.979 [Connector-Scheduler-74f6c5d8-1] DEBUG o.e.jetty.http2.HTTP2Connection - Ignored idle timeout on HTTP2ServerSession#720ff58e{l:/10.16.0.6:7171 <-> r:/110.54.160.90:41470,sendWindow=16777216,recvWindow=1047530,streams=3,NOT_CLOSED,null}: {}
2020-05-20 11:56:05.049 [Connector-Scheduler-74f6c5d8-1] DEBUG org.eclipse.jetty.io.IdleTimeout - HTTP2Stream#ad94434#11{sendWindow=16777216,recvWindow=524288,reset=false/false,REMOTELY_CLOSED,age=450006,attachment=ServerHttpChannelOverHTTP2#7dbb8795{s=HttpChannelState#6d3b17f6{s=WAITING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WAITING,uri=https://{HOST}:7171/{ENDPOINT}=11335&type=sub,age=450007}#11} idle timeout check, elapsed: 30000 ms, remaining: 0 ms
2020-05-20 11:56:05.049 [Connector-Scheduler-74f6c5d8-1] DEBUG org.eclipse.jetty.io.IdleTimeout - HTTP2Stream#ad94434#11{sendWindow=16777216,recvWindow=524288,reset=false/false,REMOTELY_CLOSED,age=450006,attachment=ServerHttpChannelOverHTTP2#7dbb8795{s=HttpChannelState#6d3b17f6{s=WAITING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WAITING,uri=https://{HOST}:7171/{ENDPOINT}=11335&type=sub,age=450007}#11} idle timeout expired
2020-05-20 11:56:05.049 [Connector-Scheduler-74f6c5d8-1] DEBUG org.eclipse.jetty.http2.HTTP2Stream - Idle timeout 30000ms expired on HTTP2Stream#ad94434#11{sendWindow=16777216,recvWindow=524288,reset=false/false,REMOTELY_CLOSED,age=450006,attachment=ServerHttpChannelOverHTTP2#7dbb8795{s=HttpChannelState#6d3b17f6{s=WAITING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WAITING,uri=https://{HOST}:7171/{ENDPOINT}=11335&type=sub,age=450007}#11}
2020-05-20 11:56:05.049 [Connector-Scheduler-74f6c5d8-1] DEBUG o.e.j.h.s.HttpTransportOverHTTP2 - HTTP2 Response #11/11a5a716 idle timeout ignored
java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-05-20 11:56:05.049 [Connector-Scheduler-74f6c5d8-1] DEBUG o.e.jetty.http2.HTTP2Connection - Ignored idle timeout on HTTP2Stream#ad94434#11{sendWindow=16777216,recvWindow=524288,reset=false/false,REMOTELY_CLOSED,age=450006,attachment=ServerHttpChannelOverHTTP2#7dbb8795{s=HttpChannelState#6d3b17f6{s=WAITING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WAITING,uri=https://{HOST}:7171/{ENDPOINT}=11335&type=sub,age=450007}#11}: {}
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl#1530ee6d woken up from select, 1/1/156 selected
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl#1530ee6d processing 1 keys, 0 updates
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.ManagedSelector - selected 1 channel=java.nio.channels.SocketChannel[connected local=/10.16.0.6:7171 remote=/194.208.155.11:40676], selector=sun.nio.ch.EPollSelectorImpl#1530ee6d, interestOps=1, readyOps=1 SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}{io=1/1,kio=1,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.ChannelEndPoint - onSelected 1->0 r=true w=false for SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}{io=1/0,kio=1,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.ChannelEndPoint - task CEP:SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}{io=1/0,kio=1,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d:runFillable:EITHER
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG o.e.j.u.t.ReservedThreadExecutor - ReservedThreadExecutor#6a969fb8{s=2/2,p=0} tryExecute EatWhatYouKill#771db12c/SelectorProducer#522ba524/PRODUCING/p=false/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=8,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=2/2,p=0}][pc=9,pic=39,pec=56,epc=12927]#2020-05-20T11:56:05.101877Z
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG o.e.j.u.t.ReservedThreadExecutor - ReservedThreadExecutor#6a969fb8{s=1/2,p=0}#e0a082e offer EatWhatYouKill#771db12c/SelectorProducer#522ba524/PRODUCING/p=false/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=8,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=1/2,p=0}][pc=9,pic=39,pec=56,epc=12927]#2020-05-20T11:56:05.101908Z
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG o.e.j.u.t.strategy.EatWhatYouKill - EatWhatYouKill#771db12c/SelectorProducer#522ba524/IDLE/p=true/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=8,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=1/2,p=0}][pc=9,pic=39,pec=56,epc=12927]#2020-05-20T11:56:05.10194Z m=EXECUTE_PRODUCE_CONSUME t=CEP:SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}{io=1/0,kio=1,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d:runFillable:EITHER/EITHER
2020-05-20 11:56:05.101 [qtp897848096-2875] DEBUG o.e.j.u.t.ReservedThreadExecutor - ReservedThreadExecutor#6a969fb8{s=1/2,p=0}#e0a082e task=EatWhatYouKill#771db12c/SelectorProducer#522ba524/IDLE/p=true/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=8,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=1/2,p=0}][pc=9,pic=39,pec=56,epc=12927]#2020-05-20T11:56:05.101972Z
2020-05-20 11:56:05.102 [qtp897848096-2875] DEBUG o.e.j.u.t.strategy.EatWhatYouKill - EatWhatYouKill#771db12c/SelectorProducer#522ba524/IDLE/p=true/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=8,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=1/2,p=0}][pc=9,pic=39,pec=56,epc=12928]#2020-05-20T11:56:05.102015Z tryProduce true
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.FillInterest - fillable FillInterest#6289cab9{SSLC.NBReadCB#51776ee4{SslConnection#51776ee4::SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}{io=1/0,kio=1,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d}}
2020-05-20 11:56:05.102 [qtp897848096-2875] DEBUG org.eclipse.jetty.io.ManagedSelector - updateable 0
2020-05-20 11:56:05.102 [qtp897848096-2875] DEBUG org.eclipse.jetty.io.ManagedSelector - updates 0
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - >c.onFillable SslConnection#51776ee4::SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}{io=1/0,kio=1,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.102 [qtp897848096-2875] DEBUG org.eclipse.jetty.io.ChannelEndPoint - Key interests updated 1 -> 0 on SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}{io=0/0,kio=0,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.102 [qtp897848096-2875] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl#1530ee6d waiting with 156 keys
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - onFillable SslConnection#51776ee4::SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}{io=0/0,kio=0,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.FillInterest - fillable FillInterest#2e4aca68{org.eclipse.jetty.http2.HTTP2Connection$FillableCallback#3c01c9c5}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.HTTP2Connection - HTTP2 onFillable HTTP2ServerConnection#5269686d::DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.HTTP2Connection - HTTP2 produce HTTP2ServerConnection#5269686d::DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.j.u.t.strategy.EatWhatYouKill - EatWhatYouKill#3c801a15/HTTP2Producer#a602086/IDLE/p=false/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=8,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=1/2,p=0}][pc=0,pic=0,pec=0,epc=3]#2020-05-20T11:56:05.102181Z tryProduce false
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.HTTP2Connection - Dequeued task null
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.HTTP2Connection - Acquired NetworkBuffer#2e9c5b61{HeapByteBuffer#7e4d63f9[p=0,l=0,c=17408,r=0]={<<<>>>\x00\x00"\x01\x05\x00\x00\x00\x0b...\x00\x00\x00\x00\x00\x00\x00},r=1}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - >fill SslConnection#51776ee4::SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}{io=0/0,kio=0,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - fill NOT_HANDSHAKING
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.ChannelEndPoint - filled 42 HeapByteBuffer#7f14d7ed[p=0,l=42,c=17408,r=42]={<<<\x17\x03\x03\x00%\x00\x00\x00\x00\x00\x00\x00\x08\x17i0\x0f\x02\xB2\x9ac\xD0U99\xDa-q\xFaXT\x07\x90R2\x9c\x13\xE720\x10E>>>\xCe\xC4~ez\x0b\x93\xDa\xAb...\x00\x00\x00\x00\x00\x00\x00}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - net filled=42
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - unwrap net_filled=42 Status = OK HandshakeStatus = NOT_HANDSHAKING bytesConsumed = 42 bytesProduced = 13 encryptedBuffer=[p=42,l=42,c=17408,r=0] unwrapBuffer=HeapByteBuffer#7e4d63f9[p=0,l=13,c=17408,r=13]={<<<\x00\x00\x04\x03\x00\x00\x00\x00\x03\x00\x00\x00\x08>>>\x93\xC6j\n\xC0\xEb\xAd\x89\x8a...\x00\x00\x00\x00\x00\x00\x00} appBuffer=HeapByteBuffer#7e4d63f9[p=0,l=13,c=17408,r=13]={<<<\x00\x00\x04\x03\x00\x00\x00\x00\x03\x00\x00\x00\x08>>>\x93\xC6j\n\xC0\xEb\xAd\x89\x8a...\x00\x00\x00\x00\x00\x00\x00}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - <fill f=13 uf=false SslConnection#51776ee4::SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.HTTP2Connection - Filled 13 bytes in NetworkBuffer#2e9c5b61{HeapByteBuffer#7e4d63f9[p=0,l=13,c=17408,r=13]={<<<\x00\x00\x04\x03\x00\x00\x00\x00\x03\x00\x00\x00\x08>>>\x93\xC6j\n\xC0\xEb\xAd\x89\x8a...\x00\x00\x00\x00\x00\x00\x00},r=1}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.parser.ServerParser - Parsing java.nio.HeapByteBuffer[pos=0 lim=13 cap=17408]
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.http2.parser.Parser - Parsed [RST_STREAM|4|0|3] frame header from java.nio.HeapByteBuffer[pos=9 lim=13 cap=17408]
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG org.eclipse.jetty.http2.HTTP2Session - Received ResetFrame#6859fe12#3{cancel_stream_error}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG org.eclipse.jetty.http2.HTTP2Session - Removed remote HTTP2Stream#728ac07f#3{sendWindow=16777216,recvWindow=524288,reset=false/true,CLOSED,age=324671,attachment=ServerHttpChannelOverHTTP2#7f28ff8d{s=HttpChannelState#3a9673c7{s=WAITING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WAITING,uri=https://{HOST}:7171/{ENDPOINT}=1633&type=sub,age=324672}#3}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.HTTP2Connection - Processing failure on HTTP2Stream#728ac07f#3{sendWindow=16777216,recvWindow=524288,reset=false/true,CLOSED,age=324672,attachment=ServerHttpChannelOverHTTP2#7f28ff8d{s=HttpChannelState#3a9673c7{s=WAITING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WAITING,uri=https://{HOST}:7171/{ENDPOINT}=1633&type=sub,age=324672}#3}: {}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.j.h.s.HttpTransportOverHTTP2 - HTTP2 Response #3/724f0d35 flush ignored
org.eclipse.jetty.io.EofException: Reset cancel_stream_error
at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onReset(HTTP2ServerConnectionFactory.java:157)
at org.eclipse.jetty.http2.HTTP2Stream.notifyReset(HTTP2Stream.java:574)
at org.eclipse.jetty.http2.HTTP2Stream.onReset(HTTP2Stream.java:343)
at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:252)
at org.eclipse.jetty.http2.HTTP2Session.onReset(HTTP2Session.java:295)
at org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onReset(Parser.java:372)
at org.eclipse.jetty.http2.parser.BodyParser.notifyReset(BodyParser.java:144)
at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:97)
at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:66)
at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:198)
at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:127)
at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:115)
at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
and this too
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG org.eclipse.jetty.http2.HTTP2Session - Disconnecting HTTP2ServerSession#1a9813fd{l:/10.16.0.6:7171 <-> r:/152.32.102.11:43890,sendWindow=16777216,recvWindow=1048437,streams=0,CLOSED,null}
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.AbstractEndPoint - close DecryptedEndPoint#1456064f{/152.32.102.11:43890<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=115/30000}
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.AbstractEndPoint - close(null) DecryptedEndPoint#1456064f{/152.32.102.11:43890<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=115/30000}
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.ssl.SslConnection - shutdownOutput: SslConnection#4d317d32::SocketChannelEndPoint#31397a33{/152.32.102.11:43890<->/10.16.0.6:7171,ISHUT,fill=-,flush=-,to=115/30000}{io=1/0,kio=1,kro=1}->SslConnection#4d317d32{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint#1456064f{/152.32.102.11:43890<->/10.16.0.6:7171,CLOSED,fill=-,flush=-,to=115/30000}=>HTTP2ServerConnection#7ebcadd4 oshut=false, ishut=true
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.ssl.SslConnection - >flush SslConnection#4d317d32::SocketChannelEndPoint#31397a33{/152.32.102.11:43890<->/10.16.0.6:7171,ISHUT,fill=-,flush=-,to=115/30000}{io=1/0,kio=1,kro=1}->SslConnection#4d317d32{NEED_WRAP,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint#1456064f{/152.32.102.11:43890<->/10.16.0.6:7171,CLOSED,fill=-,flush=-,to=115/30000}=>HTTP2ServerConnection#7ebcadd4
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.ssl.SslConnection - flush b[0]=HeapByteBuffer#17a64e32[p=0,l=0,c=0,r=0]={<<<>>>}
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.ssl.SslConnection - flush NEED_WRAP
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.ssl.SslConnection - wrap Status = CLOSED HandshakeStatus = NEED_UNWRAP bytesConsumed = 0 bytesProduced = 31 sequenceNumber = 3 [p=0,l=31,c=17408,r=31] ioDone=false/true
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG org.eclipse.jetty.io.ChannelEndPoint - flushed 31 SocketChannelEndPoint#31397a33{/152.32.102.11:43890<->/10.16.0.6:7171,ISHUT,fill=-,flush=-,to=115/30000}{io=1/0,kio=1,kro=1}->SslConnection#4d317d32{NEED_UNWRAP,eio=-1/0,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint#1456064f{/152.32.102.11:43890<->/10.16.0.6:7171,CLOSED,fill=-,flush=-,to=115/30000}=>HTTP2ServerConnection#7ebcadd4
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.ssl.SslConnection - net flushed=true, ac=true
2020-05-20 11:56:10.528 [qtp897848096-2875] DEBUG org.eclipse.jetty.io.WriteFlusher - ignored: WriteFlusher#2fa70bfd{IDLE}->null
java.nio.channels.ClosedChannelException: null
at org.eclipse.jetty.io.WriteFlusher.onClose(WriteFlusher.java:521)
at org.eclipse.jetty.io.AbstractEndPoint.onClose(AbstractEndPoint.java:354)
at org.eclipse.jetty.io.ChannelEndPoint.onClose(ChannelEndPoint.java:214)
at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:225)
at org.eclipse.jetty.io.AbstractEndPoint.shutdownOutput(AbstractEndPoint.java:157)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1081)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.doShutdownOutput(SslConnection.java:1278)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.doClose(SslConnection.java:1343)
at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:220)
at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:192)
at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:175)
at org.eclipse.jetty.http2.HTTP2Session.disconnect(HTTP2Session.java:1023)
at org.eclipse.jetty.http2.HTTP2Session.terminate(HTTP2Session.java:1045)
at org.eclipse.jetty.http2.HTTP2Session.access$1200(HTTP2Session.java:72)
at org.eclipse.jetty.http2.HTTP2Session$TerminateCallback.complete(HTTP2Session.java:1647)
at org.eclipse.jetty.http2.HTTP2Session$TerminateCallback.succeeded(HTTP2Session.java:1628)
at org.eclipse.jetty.util.Callback$Nested.succeeded(Callback.java:266)
at org.eclipse.jetty.util.CountingCallback.succeeded(CountingCallback.java:68)
at org.eclipse.jetty.http2.server.HttpChannelOverHTTP2$FailureTask.run(HttpChannelOverHTTP2.java:407)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
These are DEBUG logs, so you may find logged there exceptions that are not "crashes" but just logging what happened in the implementation.
The first and third exceptions reported are a TLS problem, but since the log is incomplete (lacks what happens before the channel is closed), it's difficult to tell what's going on.
A common problem is that the client does not connect with a sufficiently strong cipher, and the server rejects the connection as required by HTTP/2.
I would review what the client sends (what TLS protocol version, what cipher, etc.) and modern clients should work fine.

Error thrown when trying to set SES to publish events to SNS

I'm trying to run set-identity-notification-topic on SES in my localstack and an error gets thrown. Any workaround or a solution is kindly appreciated.
I have verified my email address with SES on localstack. However I get an error when trying to run,
aws --endpoint-url=http://localhost:4579 ses set-identity-notification-topic --identity myname#domain.com --notification-type Bounce --sns-topic arn:aws:sns:us-east-1:000000000000:LOCAL_ses_responses --debug
stacktrace:
......
......
> 2019-08-07 17:04:39,539 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ses.SetIdentityNotificationTopic: calling handler 2019-08-07 17:04:39,540 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth. 2019-08-07 17:04:39,540 - MainThread - botocore.auth - DEBUG - CanonicalRequest: POST / 
 content-type:application/x-www-form-urlencoded; charset=utf-8 host:localhost:4579 x-amz-date:20190807T070439Z 
 content-type;host;x-amz-date e19fa7707096ec4f34644c4d08291160421d771325cc9d8b696cb376f4f39749 2019-08-07 17:04:39,540 - MainThread - botocore.auth - DEBUG - StringToSign: AWS4-HMAC-SHA256 20190807T070439Z 20190807/us-east-1/ses/aws4_request 4922714082f96a39b9065b91093231d793cdec8af0da19698616090b17501d28 2019-08-07 17:04:39,540 - MainThread - botocore.auth - DEBUG - Signature: 06afe49a26eb61b6420bc726da1eb79b4e0a0011fe4a246ec2e9541bed2b0c3c 2019-08-07 17:04:39,540 - MainThread - botocore.endpoint - DEBUG - Sending http request: http://localhost:4579/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/1.16.210 Python/3.7.4 Darwin/18.5.0 botocore/1.12.200', 'X-Amz-Date': b'20190807T070439Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAII5V24FP6F2TGDUQ/20190807/us-east-1/ses/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=06afe49a26eb61b6420bc726da1eb79b4e0a0011fe4a246ec2e9541bed2b0c3c', 'Content-Length': '194'}> 2019-08-07 17:04:39,541 - MainThread - urllib3.util.retry - DEBUG - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None) 2019-08-07 17:04:39,541 - MainThread - urllib3.connectionpool - DEBUG
- Starting new HTTP connection (1): localhost:4579 2019-08-07 17:04:39,550 - MainThread - urllib3.connectionpool - DEBUG
- http://localhost:4579 "POST / HTTP/1.1" 500 290 2019-08-07 17:04:39,551 - MainThread - botocore.parsers - DEBUG - Response headers: {'Content-Type': 'text/html', 'Content-Length': '290', 'Server': 'Werkzeug/0.15.5 Python/3.6.8', 'Date': 'Wed, 07 Aug 2019 07:04:39 GMT'} 2019-08-07 17:04:39,551 - MainThread - botocore.parsers
- DEBUG - Response body: b'\n500 Internal Server Error\nInternal Server Error\nThe server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.\n' 2019-08-07 17:04:39,551 - MainThread - awscli.clidriver - DEBUG - Exception caught in main() Traceback (most recent call last):   File "/Users/danuka.aluthage/.virtualenvs/myvenv/lib/python3.7/site-packages/botocore/parsers.py", line 432, in _parse_xml_string_to_dom
    parser.feed(xml_string)   File "", line None xml.etree.ElementTree.ParseError: syntax error: line 1, column 54 
 During handling of the above exception, another exception occurred: 
 Traceback (most recent call last):   File "/Users/danuka.aluthage/.virtualenvs/myvenv/lib/python3.7/site-packages/awscli/clidriver.py", line 207, in main
    return command_table[parsed_args.command](remaining, parsed_args)   File "/Users/danuka.aluthage/.virtualenvs/myvenv/lib/python3.7/site-packages/awscli/clidriver.py", line 348, in call
.....
.....
Unable to parse response (syntax error: line 1, column 54), invalid XML received:
b'\n500 Internal Server Error\nInternal Server Error\nThe server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.\n'

AWS CLI access to S3 returns error 104

AWS reports that all their services are operating normally, but I get Error 104 'Connection reset by peer' on all my S3 requests via the AWS CLI. Yesterday it was working fine. Do you have any suggestions for how to fix this?
$ aws --version
aws-cli/1.11.32 Python/2.6.6 Linux/2.6.32-642.11.1.el6.x86_64 botocore/1.4.89
$ aws s3 ls s3://
('Connection aborted.', error(104, 'Connection reset by peer'))
$ aws s3api list-buckets
('Connection aborted.', error(104, 'Connection reset by peer'))
Here is the list-buckets command with debugging info:
$ aws s3api list-buckets --debug
2016-12-21 09:25:11,926 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.11.32 Python/2.6.6 Linux/2.6.32-642.11.1.el6.x86_64 botocore/1.4.89
2016-12-21 09:25:11,927 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3api', 'list-buckets', '--debug']
2016-12-21 09:25:11,927 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2016-12-21 09:25:11,927 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2016-12-21 09:25:11,929 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.6/site-packages/botocore/data/s3/2006-03-01/service-2.json
2016-12-21 09:25:11,988 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.s3: calling handler
2016-12-21 09:25:11,988 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: s3
2016-12-21 09:25:11,990 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3api: calling handler
2016-12-21 09:25:11,992 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.6/site-packages/botocore/data/s3/2006-03-01/waiters-2.json
2016-12-21 09:25:11,993 - MainThread - awscli.clidriver - DEBUG - OrderedDict()
2016-12-21 09:25:11,994 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.list-buckets: calling handler
2016-12-21 09:25:11,994 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.list-buckets: calling handler
2016-12-21 09:25:11,994 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.list-buckets: calling handler
2016-12-21 09:25:11,996 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.6/site-packages/botocore/data/s3/2006-03-01/paginators-1.json
2016-12-21 09:25:11,996 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.list-buckets: calling handler
2016-12-21 09:25:11,996 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.s3api.list-buckets: calling handler >
2016-12-21 09:25:11,997 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.s3api.list-buckets: calling handler >
2016-12-21 09:25:11,997 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.list-buckets.cli-input-json: calling handler
2016-12-21 09:25:11,997 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.list-buckets.generate-cli-skeleton: calling handler
2016-12-21 09:25:11,997 - MainThread - botocore.hooks - DEBUG - Event calling-command.s3api.list-buckets: calling handler >
2016-12-21 09:25:11,997 - MainThread - botocore.hooks - DEBUG - Event calling-command.s3api.list-buckets: calling handler >
2016-12-21 09:25:11,998 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2016-12-21 09:25:11,998 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2016-12-21 09:25:11,998 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.6/site-packages/botocore/data/endpoints.json
2016-12-21 09:25:12,030 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2016-12-21 09:25:12,033 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler
2016-12-21 09:25:12,033 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler
2016-12-21 09:25:12,034 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2016-12-21 09:25:12,037 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2016-12-21 09:25:12,037 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListBuckets: calling handler
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListBuckets: calling handler >
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListBuckets: calling handler
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListBuckets: calling handler
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListBuckets: calling handler >
2016-12-21 09:25:12,038 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListBuckets) (verify_ssl=True) with params: {'body': '', 'url': u'https://s3.amazonaws.com/', 'headers': {'User-Agent': 'aws-cli/1.11.32 Python/2.6.6 Linux/2.6.32-642.11.1.el6.x86_64 botocore/1.4.89'}, 'context': {'client_region': 'us-east-1', 'signing': {'bucket': None}, 'has_streaming_input': False, 'client_config': }, 'query_string': '', 'url_path': u'/', 'method': u'GET'}
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListBuckets: calling handler >
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListBuckets: calling handler
2016-12-21 09:25:12,039 - MainThread - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2016-12-21 09:25:12,039 - MainThread - botocore.auth - DEBUG - HTTP request method: GET
2016-12-21 09:25:12,039 - MainThread - botocore.auth - DEBUG - StringToSign:
GET
Wed, 21 Dec 2016 14:25:12 GMT
/
2016-12-21 09:25:12,044 - MainThread - botocore.endpoint - DEBUG - Sending http request:
2016-12-21 09:25:12,044 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): s3.amazonaws.com
2016-12-21 09:25:12,078 - MainThread - botocore.endpoint - DEBUG - ConnectionError received when sending HTTP request.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/botocore/endpoint.py", line 204, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/usr/lib/python2.6/site-packages/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.6/site-packages/botocore/vendored/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer'))
I experienced this too, but when I specify the region where my S3 bucket reside, the problem solved.
aws s3 --region ap-southeast-1 cp file100Mb.zip s3://mybucket.domain.xyz/path/
You can try and let me know how it does for you.
For me, I installed the awscli via the apt-get command (Ubuntu 16). It was working fine for us-east-1 region. But when I tried to upload files to different regions to a newly created bucket it started giving me this error. By removing the aws cli and installing via pip resolved this issue. It seems the version which is installed via apt-get is not updated for ubuntu 16.
sudo apt-get install pip
pip install awscli
I switched my VM networking from "NAT" to "bridged" and S3 magically started working. When it stopped working, I had not change anything manually; I hope VMware was not changing the network by itself. But if you get error 104, check your network settings.