Jetty is throwing exceptions on ktor i think issue with http2

Jetty is throwing exceptions on ktor i think issue with http2 - jetty

I'm getting these crashes on server using ktor with jetty. On start of jetty server it show exception of idle timeout then after sometime it throw too many other exceptions which i mentioned and server stop working please tell help me to findout this problem thanks.
2020-05-20 11:56:10.566 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.AbstractEndPoint - close DecryptedEndPoint#1066e08c{/152.32.102.11:14898<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=152/30000}
2020-05-20 11:56:10.566 [qtp897848096-2713] DEBUG org.eclipse.jetty.server.HttpChannel - handle https://{HOST}:7171/{ENDPOINT} ServerHttpChannelOverHTTP2#35396382{s=HttpChannelState#715f84bd{s=WOKEN rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WOKEN,uri=https://{HOST}:7171/{ENDPOINT},age=52973}#3
2020-05-20 11:56:10.566 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.AbstractEndPoint - close(null) DecryptedEndPoint#1066e08c{/152.32.102.11:14898<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=152/30000}
2020-05-20 11:56:10.566 [qtp897848096-2713] DEBUG o.e.jetty.server.HttpChannelState - handling HttpChannelState#715f84bd{s=WOKEN rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0}
2020-05-20 11:56:10.566 [qtp897848096-2713] DEBUG org.eclipse.jetty.server.HttpChannel - action ASYNC_ERROR ServerHttpChannelOverHTTP2#35396382{s=HttpChannelState#715f84bd{s=HANDLING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=HANDLING,uri=https://{HOST}:7171/{ENDPOINT},age=52973}#3
2020-05-20 11:56:10.566 [qtp897848096-2875] DEBUG o.e.j.u.t.ReservedThreadExecutor - ReservedThreadExecutor#6a969fb8{s=1/2,p=0}#2b7501a0 task=EatWhatYouKill#47733c6f/HTTP2Producer#53a8567/IDLE/p=true/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=6,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=1/2,p=0}][pc=0,pic=0,pec=0,epc=2]#2020-05-20T11:56:10.566514Z
2020-05-20 11:56:10.566 [qtp897848096-2713] WARN org.eclipse.jetty.server.HttpChannel - handleException /{ENDPOINT}
java.nio.channels.ClosedChannelException: null
at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:926)
at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:283)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Suppressed: java.lang.Throwable: HttpInput failure
at org.eclipse.jetty.server.HttpInput.failed(HttpInput.java:823)
at org.eclipse.jetty.http2.server.HttpChannelOverHTTP2.onFailure(HttpChannelOverHTTP2.java:323)
at org.eclipse.jetty.http2.server.HTTP2ServerConnection.onStreamFailure(HTTP2ServerConnection.java:221)
at org.eclipse.jetty.http2.server.HTTP2ServerConnection.onSessionFailure(HTTP2ServerConnection.java:261)
at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onFailure(HTTP2ServerConnectionFactory.java:128)
at org.eclipse.jetty.http2.HTTP2Session.notifyFailure(HTTP2Session.java:1156)
at org.eclipse.jetty.http2.HTTP2Session.abort(HTTP2Session.java:1060)
... 23 common frames omitted
and also getting these crashes
2020-05-20 11:56:04.979 [Connector-Scheduler-74f6c5d8-1] DEBUG org.eclipse.jetty.io.IdleTimeout - SocketChannelEndPoint#1aea799a{/110.54.160.90:41470<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=30000/30000}{io=1/1,kio=1,kro=1}->SslConnection#662a1f2b{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#4724da13{/110.54.160.90:41470<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=540007/30000}=>HTTP2ServerConnection#4fc8491c idle timeout check, elapsed: 30000 ms, remaining: 0 ms
2020-05-20 11:56:04.979 [Connector-Scheduler-74f6c5d8-1] DEBUG org.eclipse.jetty.io.IdleTimeout - SocketChannelEndPoint#1aea799a{/110.54.160.90:41470<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=30000/30000}{io=1/1,kio=1,kro=1}->SslConnection#662a1f2b{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#4724da13{/110.54.160.90:41470<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=540007/30000}=>HTTP2ServerConnection#4fc8491c idle timeout expired
2020-05-20 11:56:04.979 [Connector-Scheduler-74f6c5d8-1] DEBUG o.e.jetty.http2.HTTP2Connection - Ignored idle timeout on HTTP2ServerSession#720ff58e{l:/10.16.0.6:7171 <-> r:/110.54.160.90:41470,sendWindow=16777216,recvWindow=1047530,streams=3,NOT_CLOSED,null}: {}
2020-05-20 11:56:05.049 [Connector-Scheduler-74f6c5d8-1] DEBUG org.eclipse.jetty.io.IdleTimeout - HTTP2Stream#ad94434#11{sendWindow=16777216,recvWindow=524288,reset=false/false,REMOTELY_CLOSED,age=450006,attachment=ServerHttpChannelOverHTTP2#7dbb8795{s=HttpChannelState#6d3b17f6{s=WAITING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WAITING,uri=https://{HOST}:7171/{ENDPOINT}=11335&type=sub,age=450007}#11} idle timeout check, elapsed: 30000 ms, remaining: 0 ms
2020-05-20 11:56:05.049 [Connector-Scheduler-74f6c5d8-1] DEBUG org.eclipse.jetty.io.IdleTimeout - HTTP2Stream#ad94434#11{sendWindow=16777216,recvWindow=524288,reset=false/false,REMOTELY_CLOSED,age=450006,attachment=ServerHttpChannelOverHTTP2#7dbb8795{s=HttpChannelState#6d3b17f6{s=WAITING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WAITING,uri=https://{HOST}:7171/{ENDPOINT}=11335&type=sub,age=450007}#11} idle timeout expired
2020-05-20 11:56:05.049 [Connector-Scheduler-74f6c5d8-1] DEBUG org.eclipse.jetty.http2.HTTP2Stream - Idle timeout 30000ms expired on HTTP2Stream#ad94434#11{sendWindow=16777216,recvWindow=524288,reset=false/false,REMOTELY_CLOSED,age=450006,attachment=ServerHttpChannelOverHTTP2#7dbb8795{s=HttpChannelState#6d3b17f6{s=WAITING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WAITING,uri=https://{HOST}:7171/{ENDPOINT}=11335&type=sub,age=450007}#11}
2020-05-20 11:56:05.049 [Connector-Scheduler-74f6c5d8-1] DEBUG o.e.j.h.s.HttpTransportOverHTTP2 - HTTP2 Response #11/11a5a716 idle timeout ignored
java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-05-20 11:56:05.049 [Connector-Scheduler-74f6c5d8-1] DEBUG o.e.jetty.http2.HTTP2Connection - Ignored idle timeout on HTTP2Stream#ad94434#11{sendWindow=16777216,recvWindow=524288,reset=false/false,REMOTELY_CLOSED,age=450006,attachment=ServerHttpChannelOverHTTP2#7dbb8795{s=HttpChannelState#6d3b17f6{s=WAITING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WAITING,uri=https://{HOST}:7171/{ENDPOINT}=11335&type=sub,age=450007}#11}: {}
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl#1530ee6d woken up from select, 1/1/156 selected
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl#1530ee6d processing 1 keys, 0 updates
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.ManagedSelector - selected 1 channel=java.nio.channels.SocketChannel[connected local=/10.16.0.6:7171 remote=/194.208.155.11:40676], selector=sun.nio.ch.EPollSelectorImpl#1530ee6d, interestOps=1, readyOps=1 SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}{io=1/1,kio=1,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.ChannelEndPoint - onSelected 1->0 r=true w=false for SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}{io=1/0,kio=1,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.ChannelEndPoint - task CEP:SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}{io=1/0,kio=1,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d:runFillable:EITHER
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG o.e.j.u.t.ReservedThreadExecutor - ReservedThreadExecutor#6a969fb8{s=2/2,p=0} tryExecute EatWhatYouKill#771db12c/SelectorProducer#522ba524/PRODUCING/p=false/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=8,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=2/2,p=0}][pc=9,pic=39,pec=56,epc=12927]#2020-05-20T11:56:05.101877Z
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG o.e.j.u.t.ReservedThreadExecutor - ReservedThreadExecutor#6a969fb8{s=1/2,p=0}#e0a082e offer EatWhatYouKill#771db12c/SelectorProducer#522ba524/PRODUCING/p=false/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=8,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=1/2,p=0}][pc=9,pic=39,pec=56,epc=12927]#2020-05-20T11:56:05.101908Z
2020-05-20 11:56:05.101 [qtp897848096-2890] DEBUG o.e.j.u.t.strategy.EatWhatYouKill - EatWhatYouKill#771db12c/SelectorProducer#522ba524/IDLE/p=true/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=8,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=1/2,p=0}][pc=9,pic=39,pec=56,epc=12927]#2020-05-20T11:56:05.10194Z m=EXECUTE_PRODUCE_CONSUME t=CEP:SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}{io=1/0,kio=1,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d:runFillable:EITHER/EITHER
2020-05-20 11:56:05.101 [qtp897848096-2875] DEBUG o.e.j.u.t.ReservedThreadExecutor - ReservedThreadExecutor#6a969fb8{s=1/2,p=0}#e0a082e task=EatWhatYouKill#771db12c/SelectorProducer#522ba524/IDLE/p=true/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=8,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=1/2,p=0}][pc=9,pic=39,pec=56,epc=12927]#2020-05-20T11:56:05.101972Z
2020-05-20 11:56:05.102 [qtp897848096-2875] DEBUG o.e.j.u.t.strategy.EatWhatYouKill - EatWhatYouKill#771db12c/SelectorProducer#522ba524/IDLE/p=true/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=8,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=1/2,p=0}][pc=9,pic=39,pec=56,epc=12928]#2020-05-20T11:56:05.102015Z tryProduce true
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.FillInterest - fillable FillInterest#6289cab9{SSLC.NBReadCB#51776ee4{SslConnection#51776ee4::SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}{io=1/0,kio=1,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d}}
2020-05-20 11:56:05.102 [qtp897848096-2875] DEBUG org.eclipse.jetty.io.ManagedSelector - updateable 0
2020-05-20 11:56:05.102 [qtp897848096-2875] DEBUG org.eclipse.jetty.io.ManagedSelector - updates 0
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - >c.onFillable SslConnection#51776ee4::SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}{io=1/0,kio=1,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.102 [qtp897848096-2875] DEBUG org.eclipse.jetty.io.ChannelEndPoint - Key interests updated 1 -> 0 on SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}{io=0/0,kio=0,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.102 [qtp897848096-2875] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl#1530ee6d waiting with 156 keys
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - onFillable SslConnection#51776ee4::SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}{io=0/0,kio=0,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=FI,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.FillInterest - fillable FillInterest#2e4aca68{org.eclipse.jetty.http2.HTTP2Connection$FillableCallback#3c01c9c5}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.HTTP2Connection - HTTP2 onFillable HTTP2ServerConnection#5269686d::DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.HTTP2Connection - HTTP2 produce HTTP2ServerConnection#5269686d::DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.j.u.t.strategy.EatWhatYouKill - EatWhatYouKill#3c801a15/HTTP2Producer#a602086/IDLE/p=false/QueuedThreadPool[qtp897848096]#35841320{STARTED,8<=12<=200,i=8,r=2,q=0}[ReservedThreadExecutor#6a969fb8{s=1/2,p=0}][pc=0,pic=0,pec=0,epc=3]#2020-05-20T11:56:05.102181Z tryProduce false
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.HTTP2Connection - Dequeued task null
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.HTTP2Connection - Acquired NetworkBuffer#2e9c5b61{HeapByteBuffer#7e4d63f9[p=0,l=0,c=17408,r=0]={<<<>>>\x00\x00"\x01\x05\x00\x00\x00\x0b...\x00\x00\x00\x00\x00\x00\x00},r=1}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - >fill SslConnection#51776ee4::SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}{io=0/0,kio=0,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - fill NOT_HANDSHAKING
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG org.eclipse.jetty.io.ChannelEndPoint - filled 42 HeapByteBuffer#7f14d7ed[p=0,l=42,c=17408,r=42]={<<<\x17\x03\x03\x00%\x00\x00\x00\x00\x00\x00\x00\x08\x17i0\x0f\x02\xB2\x9ac\xD0U99\xDa-q\xFaXT\x07\x90R2\x9c\x13\xE720\x10E>>>\xCe\xC4~ez\x0b\x93\xDa\xAb...\x00\x00\x00\x00\x00\x00\x00}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - net filled=42
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - unwrap net_filled=42 Status = OK HandshakeStatus = NOT_HANDSHAKING bytesConsumed = 42 bytesProduced = 13 encryptedBuffer=[p=42,l=42,c=17408,r=0] unwrapBuffer=HeapByteBuffer#7e4d63f9[p=0,l=13,c=17408,r=13]={<<<\x00\x00\x04\x03\x00\x00\x00\x00\x03\x00\x00\x00\x08>>>\x93\xC6j\n\xC0\xEb\xAd\x89\x8a...\x00\x00\x00\x00\x00\x00\x00} appBuffer=HeapByteBuffer#7e4d63f9[p=0,l=13,c=17408,r=13]={<<<\x00\x00\x04\x03\x00\x00\x00\x00\x03\x00\x00\x00\x08>>>\x93\xC6j\n\xC0\xEb\xAd\x89\x8a...\x00\x00\x00\x00\x00\x00\x00}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.io.ssl.SslConnection - <fill f=13 uf=false SslConnection#51776ee4::SocketChannelEndPoint#6a358a13{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->SslConnection#51776ee4{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint#345f0a02{/194.208.155.11:40676<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=589/30000}=>HTTP2ServerConnection#5269686d
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.HTTP2Connection - Filled 13 bytes in NetworkBuffer#2e9c5b61{HeapByteBuffer#7e4d63f9[p=0,l=13,c=17408,r=13]={<<<\x00\x00\x04\x03\x00\x00\x00\x00\x03\x00\x00\x00\x08>>>\x93\xC6j\n\xC0\xEb\xAd\x89\x8a...\x00\x00\x00\x00\x00\x00\x00},r=1}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.parser.ServerParser - Parsing java.nio.HeapByteBuffer[pos=0 lim=13 cap=17408]
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.eclipse.jetty.http2.parser.Parser - Parsed [RST_STREAM|4|0|3] frame header from java.nio.HeapByteBuffer[pos=9 lim=13 cap=17408]
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG org.eclipse.jetty.http2.HTTP2Session - Received ResetFrame#6859fe12#3{cancel_stream_error}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG org.eclipse.jetty.http2.HTTP2Session - Removed remote HTTP2Stream#728ac07f#3{sendWindow=16777216,recvWindow=524288,reset=false/true,CLOSED,age=324671,attachment=ServerHttpChannelOverHTTP2#7f28ff8d{s=HttpChannelState#3a9673c7{s=WAITING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WAITING,uri=https://{HOST}:7171/{ENDPOINT}=1633&type=sub,age=324672}#3}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.jetty.http2.HTTP2Connection - Processing failure on HTTP2Stream#728ac07f#3{sendWindow=16777216,recvWindow=524288,reset=false/true,CLOSED,age=324672,attachment=ServerHttpChannelOverHTTP2#7f28ff8d{s=HttpChannelState#3a9673c7{s=WAITING rs=ASYNC os=OPEN is=IDLE awp=false se=false i=false al=0},r=1,c=false/false,a=WAITING,uri=https://{HOST}:7171/{ENDPOINT}=1633&type=sub,age=324672}#3}: {}
2020-05-20 11:56:05.102 [qtp897848096-2890] DEBUG o.e.j.h.s.HttpTransportOverHTTP2 - HTTP2 Response #3/724f0d35 flush ignored
org.eclipse.jetty.io.EofException: Reset cancel_stream_error
at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onReset(HTTP2ServerConnectionFactory.java:157)
at org.eclipse.jetty.http2.HTTP2Stream.notifyReset(HTTP2Stream.java:574)
at org.eclipse.jetty.http2.HTTP2Stream.onReset(HTTP2Stream.java:343)
at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:252)
at org.eclipse.jetty.http2.HTTP2Session.onReset(HTTP2Session.java:295)
at org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onReset(Parser.java:372)
at org.eclipse.jetty.http2.parser.BodyParser.notifyReset(BodyParser.java:144)
at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:97)
at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:66)
at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:198)
at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:127)
at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:115)
at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
and this too
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG org.eclipse.jetty.http2.HTTP2Session - Disconnecting HTTP2ServerSession#1a9813fd{l:/10.16.0.6:7171 <-> r:/152.32.102.11:43890,sendWindow=16777216,recvWindow=1048437,streams=0,CLOSED,null}
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.AbstractEndPoint - close DecryptedEndPoint#1456064f{/152.32.102.11:43890<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=115/30000}
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.AbstractEndPoint - close(null) DecryptedEndPoint#1456064f{/152.32.102.11:43890<->/10.16.0.6:7171,OPEN,fill=-,flush=-,to=115/30000}
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.ssl.SslConnection - shutdownOutput: SslConnection#4d317d32::SocketChannelEndPoint#31397a33{/152.32.102.11:43890<->/10.16.0.6:7171,ISHUT,fill=-,flush=-,to=115/30000}{io=1/0,kio=1,kro=1}->SslConnection#4d317d32{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint#1456064f{/152.32.102.11:43890<->/10.16.0.6:7171,CLOSED,fill=-,flush=-,to=115/30000}=>HTTP2ServerConnection#7ebcadd4 oshut=false, ishut=true
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.ssl.SslConnection - >flush SslConnection#4d317d32::SocketChannelEndPoint#31397a33{/152.32.102.11:43890<->/10.16.0.6:7171,ISHUT,fill=-,flush=-,to=115/30000}{io=1/0,kio=1,kro=1}->SslConnection#4d317d32{NEED_WRAP,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint#1456064f{/152.32.102.11:43890<->/10.16.0.6:7171,CLOSED,fill=-,flush=-,to=115/30000}=>HTTP2ServerConnection#7ebcadd4
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.ssl.SslConnection - flush b[0]=HeapByteBuffer#17a64e32[p=0,l=0,c=0,r=0]={<<<>>>}
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.ssl.SslConnection - flush NEED_WRAP
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.ssl.SslConnection - wrap Status = CLOSED HandshakeStatus = NEED_UNWRAP bytesConsumed = 0 bytesProduced = 31 sequenceNumber = 3 [p=0,l=31,c=17408,r=31] ioDone=false/true
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG org.eclipse.jetty.io.ChannelEndPoint - flushed 31 SocketChannelEndPoint#31397a33{/152.32.102.11:43890<->/10.16.0.6:7171,ISHUT,fill=-,flush=-,to=115/30000}{io=1/0,kio=1,kro=1}->SslConnection#4d317d32{NEED_UNWRAP,eio=-1/0,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint#1456064f{/152.32.102.11:43890<->/10.16.0.6:7171,CLOSED,fill=-,flush=-,to=115/30000}=>HTTP2ServerConnection#7ebcadd4
2020-05-20 11:56:10.528 [qtp897848096-2877] DEBUG o.eclipse.jetty.io.ssl.SslConnection - net flushed=true, ac=true
2020-05-20 11:56:10.528 [qtp897848096-2875] DEBUG org.eclipse.jetty.io.WriteFlusher - ignored: WriteFlusher#2fa70bfd{IDLE}->null
java.nio.channels.ClosedChannelException: null
at org.eclipse.jetty.io.WriteFlusher.onClose(WriteFlusher.java:521)
at org.eclipse.jetty.io.AbstractEndPoint.onClose(AbstractEndPoint.java:354)
at org.eclipse.jetty.io.ChannelEndPoint.onClose(ChannelEndPoint.java:214)
at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:225)
at org.eclipse.jetty.io.AbstractEndPoint.shutdownOutput(AbstractEndPoint.java:157)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1081)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.doShutdownOutput(SslConnection.java:1278)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.doClose(SslConnection.java:1343)
at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:220)
at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:192)
at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:175)
at org.eclipse.jetty.http2.HTTP2Session.disconnect(HTTP2Session.java:1023)
at org.eclipse.jetty.http2.HTTP2Session.terminate(HTTP2Session.java:1045)
at org.eclipse.jetty.http2.HTTP2Session.access$1200(HTTP2Session.java:72)
at org.eclipse.jetty.http2.HTTP2Session$TerminateCallback.complete(HTTP2Session.java:1647)
at org.eclipse.jetty.http2.HTTP2Session$TerminateCallback.succeeded(HTTP2Session.java:1628)
at org.eclipse.jetty.util.Callback$Nested.succeeded(Callback.java:266)
at org.eclipse.jetty.util.CountingCallback.succeeded(CountingCallback.java:68)
at org.eclipse.jetty.http2.server.HttpChannelOverHTTP2$FailureTask.run(HttpChannelOverHTTP2.java:407)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)

These are DEBUG logs, so you may find logged there exceptions that are not "crashes" but just logging what happened in the implementation.
The first and third exceptions reported are a TLS problem, but since the log is incomplete (lacks what happens before the channel is closed), it's difficult to tell what's going on.
A common problem is that the client does not connect with a sufficiently strong cipher, and the server rejects the connection as required by HTTP/2.
I would review what the client sends (what TLS protocol version, what cipher, etc.) and modern clients should work fine.

Related

How to find the reason for a jetty 400 error?

I have a jetty server (version 9.4.11. It is used in combination with org.restlet-2.4.3). A client is sending a POST message to an api on my server (route "/pos/v1/purchasable_items") and jetty responds with 400 code. I cannot figure out the reason. I would really appreciate some help.
Here is the full log of jetty:
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Queued change org.eclipse.jetty.io.ManagedSelector$Accept#616b6a22 on org.eclipse.jetty.io.ManagedSelector#79c26be5 id=1 keys=0 selected=0 updates=0
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - wakeup on submit org.eclipse.jetty.io.ManagedSelector#79c26be5 id=1 keys=0 selected=0 updates=1
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e woken with none selected
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e woken up from select, 0/0/0 selected
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e processing 0 keys, 1 updates
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - updateable 1
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - update org.eclipse.jetty.io.ManagedSelector$Accept#616b6a22
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - queue org.eclipse.jetty.io.ManagedSelector$Accept#616b6a22
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - updates 0
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e waiting with 1 keys
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - run org.eclipse.jetty.io.ManagedSelector$Accept#616b6a22
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.IdleTimeout - SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=8/100000}{io=0/0,kio=0,kro=0}-><null> idle timeout check, elapsed: 7 ms, remaining: 99993 ms
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.PreEncodedHttpField - HttpField encoders loaded: [org.eclipse.jetty.http.Http1FieldPreEncoder#1525e6cb]
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - new HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0} -> SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=72/100000}{io=0/0,kio=0,kro=0}-><null>,null,HttpChannelState#7ae84ea1{s=IDLE a=NOT_ASYNC i=true r=IDLE w=false}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - New HTTP Connection HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=80/100000}{io=0/0,kio=0,kro=0}-><null>
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.AbstractEndPoint - onOpen SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=81/100000}{io=0/0,kio=0,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.AbstractConnection - onOpen HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=81/100000}{io=0/0,kio=0,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.AbstractConnection - fillInterested HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=81/100000}{io=0/0,kio=0,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.FillInterest - interested FillInterest#72454313{AC.ReadCB#30023a7c{HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=0/100000}{io=0/0,kio=0,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}}}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - changeInterests p=false 0->1 for SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=0/100000}{io=0/1,kio=0,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Queued change org.eclipse.jetty.io.ChannelEndPoint$1#528ccae5 on org.eclipse.jetty.io.ManagedSelector#79c26be5 id=1 keys=1 selected=0 updates=0
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - wakeup on submit org.eclipse.jetty.io.ManagedSelector#79c26be5 id=1 keys=1 selected=0 updates=1
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e woken with none selected
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e woken up from select, 0/0/1 selected
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e processing 0 keys, 1 updates
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - updateable 1
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - update org.eclipse.jetty.io.ChannelEndPoint$1#528ccae5
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Created SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=0/100000}{io=0/1,kio=0,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - ran org.eclipse.jetty.io.ManagedSelector$Accept#616b6a22
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - Key interests updated 0 -> 1 on SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=1/100000}{io=1/1,kio=1,kro=0}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - updates 0
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e waiting with 1 keys
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e woken up from select, 1/1/1 selected
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e processing 1 keys, 0 updates
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - selected 1 channel=java.nio.channels.SocketChannel[connected local=/127.0.0.1:8182 remote=/127.0.0.1:50589], selector=sun.nio.ch.WindowsSelectorImpl#5c1f2b6e, interestOps=1, readyOps=1 SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=1/100000}{io=1/1,kio=1,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - onSelected 1->0 r=true w=false for SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=1/100000}{io=1/0,kio=1,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - task CEP:SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=1/100000}{io=1/0,kio=1,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
12:56:05 [36mDEBUG[0;39m o.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor#77a9ea73{s=0/8,p=0} tryExecute EatWhatYouKill#218cf600/SelectorProducer#4aecb548/PRODUCING/p=false/QueuedThreadPool[qtp103099676]#6252d1c{STARTED,8<=8<=300,i=3,q=0}[ReservedThreadExecutor#77a9ea73{s=0/8,p=0}][pc=0,pic=0,pec=0,epc=0]#2021-06-17T12:56:05.8701778+02:00
12:56:05 [36mDEBUG[0;39m o.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor#77a9ea73{s=0/8,p=1} startReservedThread p=1
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - queue ReservedThreadExecutor#77a9ea73{s=0/8,p=1}#73ca3be9
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - run ReservedThreadExecutor#77a9ea73{s=0/8,p=1}#73ca3be9
12:56:05 [36mDEBUG[0;39m o.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor#77a9ea73{s=1/8,p=1}#73ca3be9 started
12:56:05 [36mDEBUG[0;39m o.e.jetty.util.thread.strategy.EatWhatYouKill - EatWhatYouKill#218cf600/SelectorProducer#4aecb548/PRODUCING/p=false/QueuedThreadPool[qtp103099676]#6252d1c{STARTED,8<=8<=300,i=2,q=0}[ReservedThreadExecutor#77a9ea73{s=0/8,p=1}][pc=0,pic=0,pec=0,epc=0]#2021-06-17T12:56:05.8721778+02:00 m=PRODUCE_EXECUTE_CONSUME t=CEP:SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=3/100000}{io=1/0,kio=1,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING/BLOCKING
12:56:05 [36mDEBUG[0;39m o.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor#77a9ea73{s=1/8,p=0}#73ca3be9 waiting
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - queue CEP:SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=3/100000}{io=1/0,kio=1,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - updateable 0
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - updates 0
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - Key interests updated 1 -> 0 on SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=3/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.WindowsSelectorImpl#5c1f2b6e waiting with 1 keys
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.util.thread.QueuedThreadPool - run CEP:SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=3/100000}{io=0/0,kio=1,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.FillInterest - fillable FillInterest#72454313{AC.ReadCB#30023a7c{HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=FI,flush=-,to=4/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}}}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=4/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0} onFillable enter HttpChannelState#7ae84ea1{s=IDLE a=NOT_ASYNC i=true r=IDLE w=false} null
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - filled 181 SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=6/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=3/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0} filled 181 HeapByteBuffer#25fdf14c[p=0,l=181,c=8192,r=181]={<<<POST /pos/v1/purc...00-continue\r\n\r\n>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=3/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=START,0 of -1},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=0,c=false,a=IDLE,uri=null,age=0} parse HeapByteBuffer#25fdf14c[p=0,l=181,c=8192,r=181]={<<<POST /pos/v1/purc...00-continue\r\n\r\n>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} {}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - parseNext s=START HeapByteBuffer#25fdf14c[p=0,l=181,c=8192,r=181]={<<<POST /pos/v1/purc...00-continue\r\n\r\n>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - START --> SPACE1
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - SPACE1 --> URI
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - URI --> SPACE2
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - SPACE2 --> REQUEST_VERSION
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - REQUEST_VERSION --> HEADER
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Content-Type: application/x-www-form-urlencoded --> IN_VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Content-Type: application/x-www-form-urlencoded --> FIELD
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Host --> VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Host --> IN_VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Host --> FIELD
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Content-Length --> VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Content-Length --> IN_VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Content-Length --> FIELD
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Expect --> VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Expect --> IN_VALUE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER:Expect --> FIELD
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - HEADER --> CONTENT
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - REQUEST for //127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05 on HttpChannelOverHttp#3b503229{r=1,c=false,a=IDLE,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=1}
POST //127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05 HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: 127.0.0.1:8182
Content-Length: 228
Expect: 100-continue
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=IDLE,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=2}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=13/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=IDLE,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=2} parsed true HttpParser{s=CONTENT,0 of 228}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - releaseRequestBuffer HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=IDLE,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=2}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=13/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=IDLE,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=2}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp#3b503229{r=1,c=false,a=IDLE,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=3} handle //127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannelState - handling HttpChannelState#7ae84ea1{s=IDLE a=NOT_ASYNC i=true r=IDLE w=false}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=4} action DISPATCH
12:56:05 [36mTRACE[0;39m org.restlet.Component.ServerRouter - Call score for the "org.restlet.routing.VirtualHost#24386839" host: 1.0
12:56:05 [36mDEBUG[0;39m org.restlet.Component.ServerRouter - Default virtual host selected
12:56:05 [36mDEBUG[0;39m org.restlet.Component.ServerRouter - Base URI: "http://127.0.0.1:8182". Remaining part: "/pos/v1/purchasable_items?d=17.06.2021%2012:56:05"
12:56:05 [36mTRACE[0;39m org.restlet.VirtualHost - Call score for the "" URI pattern: 0.5
12:56:05 [36mDEBUG[0;39m org.restlet.VirtualHost - Selected route: "" -> de.sage.scheduler.access.server.AccessServerImpl#440e3ce6
12:56:05 [36mTRACE[0;39m org.restlet.VirtualHost - No characters were matched
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - filled 0 SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=47/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=36}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - filled 0 SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=47/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=36}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=36}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=47/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=36} filled 0 HeapByteBuffer#25fdf14c[p=0,l=0,c=8192,r=0]={<<<>>>POST /pos/v1/purc...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=36}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=48/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=37} parse HeapByteBuffer#25fdf14c[p=0,l=0,c=8192,r=0]={<<<>>>POST /pos/v1/purc...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} {}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.http.HttpParser - parseNext s=CONTENT HeapByteBuffer#25fdf14c[p=0,l=0,c=8192,r=0]={<<<>>>POST /pos/v1/purc...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=37}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=48/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=37} parsed false HttpParser{s=CONTENT,0 of 228}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - releaseRequestBuffer HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=37}<-SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=-,to=48/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=START}]=>HttpChannelOverHttp#3b503229{r=1,c=false,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=37}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - sendResponse info=HTTP/1.1{s=100,h=-1,cl=-1} content=null complete=false committing=true callback=Blocker#5a2856fc{null}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - COMMIT for /pos/v1/purchasable_items on HttpChannelOverHttp#3b503229{r=1,c=true,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=37}
100 null HTTP/1.1
null
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback#364501c7[PROCESSING][i=HTTP/1.1{s=100,h=-1,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$Commit100Callback#d25f545] generate: NEED_HEADER (null,null,false)#START
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback#364501c7[PROCESSING][i=HTTP/1.1{s=100,h=-1,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$Commit100Callback#d25f545] generate: FLUSH ([p=0,l=25,c=8192,r=25],null,false)#COMPLETING_1XX
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.WriteFlusher - write: WriteFlusher#786527a2{IDLE}->null [HeapByteBuffer#25fdf14c[p=0,l=25,c=8192,r=25]={<<<HTTP/1.1 100 Continue\r\n\r\n>>>items?d=17.06.202...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}]
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.WriteFlusher - update WriteFlusher#786527a2{WRITING}->null:IDLE-->WRITING
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.ChannelEndPoint - flushed 25 SocketChannelEndPoint#54fedde4{/127.0.0.1:50589<->/127.0.0.1:8182,OPEN,fill=-,flush=W,to=55/100000}{io=0/0,kio=0,kro=1}->HttpConnection#30023a7c[p=HttpParser{s=CONTENT,0 of 228},g=HttpGenerator#7f9bfe40{s=COMPLETING_1XX}]=>HttpChannelOverHttp#3b503229{r=1,c=true,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=44}
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.WriteFlusher - Flushed=true written=25 remaining=0 WriteFlusher#786527a2{WRITING}->null
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.io.WriteFlusher - update WriteFlusher#786527a2{IDLE}->null:WRITING-->IDLE
12:56:05 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback#364501c7[PROCESSING][i=HTTP/1.1{s=100,h=-1,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$Commit100Callback#d25f545] generate: DONE ([p=25,l=25,c=8192,r=0],null,false)#START
12:56:09 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - sendResponse info=null content=DirectByteBuffer#3b173fda[p=0,l=84,c=32768,r=84]={<<<{"DESC":"The requ...formed syntax"}>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} complete=false committing=true callback=Blocker#5a2856fc{null}
12:56:09 [36mDEBUG[0;39m org.eclipse.jetty.server.HttpChannel - COMMIT for /pos/v1/purchasable_items on HttpChannelOverHttp#3b503229{r=1,c=true,a=DISPATCHED,uri=//127.0.0.1:8182/pos/v1/purchasable_items?d=17.06.2021%2012:56:05,age=3888}
400 null HTTP/1.1
Content-Type: application/json;charset=utf-8
Date: Thu, 17 Jun 2021 10:56:09 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.4.3
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Authorization, Content-Type
Access-Control-Allow-Methods: DELETE, POST, GET, OPTIONS, PUT
I have also tried to debug it with Wireshark without any success. This is the overview of the communication (Port 8182 im by server):
This is the POST request in detail from Wireshark:
I would be happy about anyone giving me a hint.

Your client is sending a POST with an Expect: 100-Continue header (a normal enough concept).
It's a bit hard to tell from your wireshark, but it looks like your client isn't handling the HTTP/1.1 100 Continue response properly.
Your client should wait on sending the body of the request until it receives the HTTP/1.1 100 Continue response.
And when it sends the request body, it should only contain the request body, it should not send the request headers again. (which your logs on jetty, and your wireshark both seem to show).
These headers do not conform to the Content-Type: application/x-www-form-urlencoded and result in a 400 Bad Request response.
Advice, Upgrade both your Restlet version, and your Jetty version.
Judging by the headers sent on the request, you have a very old version of restlet (there are some headers that i'm expecting to see, but do not).
Your Jetty version 9.4.11 is quite old and is missing many important fixes (one of which is more clear error messages around 400 Bad Request, which would have helped you figure this out yourself). You are 3,081 updates (yes, three thousand eighty one updates) behind version 9.4.42.v20210604.

The problem turned out to be a user-agent black list which blocked the request because it had no user-agent. It had nothing to do with jetty.

Is anyone else have issue deleting AWS Glacier archives?

I've been following the instructions on the AWS Docs https://docs.aws.amazon.com/amazonglacier/latest/dev/getting-started-delete-archive-cli.html and it works up until the last command where I delete the archive by its ID.
The command executes fine but the archive still in the vault, is there a time delay between aws glacier delete-archive and the archive actually getting deleted?
aws --debug glacier delete-archive --vault-name test-vault --account-id 1234567890 --archive-id "XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw"
2021-02-20 16:37:15,290 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=DeleteArchive) with params: {'url_path': '/1234567890/vaults/test-vault/archives/XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw', 'query_string': {}, 'method': 'DELETE', 'headers': {'User-Agent': 'aws-cli/2.0.38 Python/3.7.4 Darwin/20.3.0 exe/x86_64 command/glacier.delete-archive', 'x-amz-glacier-version': '2012-06-01'}, 'body': b'', 'url': ' 'context': {'client_region': 'eu-west-1', 'client_config': <botocore.config.Config object at 0x7fd5e7a9d8d0>, 'has_streaming_input': False, 'auth_type': None}}
2021-02-20 16:37:15,290 - MainThread - botocore.hooks - DEBUG - Event request-created.glacier.DeleteArchive: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fd5e7a9d790>>
2021-02-20 16:37:15,291 - MainThread - botocore.hooks - DEBUG - Event choose-signer.glacier.DeleteArchive: calling handler <function set_operation_specific_signer at 0x7fd5e67118c0>
2021-02-20 16:37:15,291 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-02-20 16:37:15,291 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
DELETE
/1234567890/vaults/test-vault/archives/XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw
host:glacier.eu-west-1.amazonaws.com
x-amz-date:20210220T163715Z
x-amz-glacier-version:2012-06-01
host;x-amz-date;x-amz-glacier-version
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2021-02-20 16:37:15,291 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20210220T163715Z
20210220/eu-west-1/glacier/aws4_request
3ac54f35e66fba13673a078b721c5db02a9f00c538bd96fc5b2d5b5709926b75
2021-02-20 16:37:15,292 - MainThread - botocore.auth - DEBUG - Signature:
1a09aef04ae7830ae5a7623cedb6ca69bf8cdef7349e3f3493a116967ba03110
2021-02-20 16:37:15,292 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=DELETE, url=https://glacier.eu-west-1.amazonaws.com/1234567890/vaults/test-vault/archives/XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw, headers={'User-Agent': b'aws-cli/2.0.38 Python/3.7.4 Darwin/20.3.0 exe/x86_64 command/glacier.delete-archive', 'x-amz-glacier-version': b'2012-06-01', 'X-Amz-Date': b'20210220T163715Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=<Access Key Removed>/20210220/eu-west-1/glacier/aws4_request, SignedHeaders=host;x-amz-date;x-amz-glacier-version, Signature=1a09aef04ae7830ae5a7623cedb6ca69bf8cdef7349e3f3493a116967ba03110', 'Content-Length': '0'}>
2021-02-20 16:37:15,294 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): glacier.eu-west-1.amazonaws.com:443
2021-02-20 16:37:15,510 - MainThread - urllib3.connectionpool - DEBUG - https://glacier.eu-west-1.amazonaws.com:443 "DELETE /1234567890/vaults/test-vault/archives/XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw HTTP/1.1" 204 0
2021-02-20 16:37:15,511 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': 'mR_DR8mtULOgCU0FDDv_aec7eYp34-QtXBONWBtWXjH1yts', 'Date': 'Sat, 20 Feb 2021 16:37:15 GMT'}
2021-02-20 16:37:15,511 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2021-02-20 16:37:15,511 - MainThread - botocore.hooks - DEBUG - Event needs-retry.glacier.DeleteArchive: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fd5e7aef190>>
2021-02-20 16:37:15,512 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2021-02-20 16:37:15,512 - MainThread - botocore.hooks - DEBUG - Event after-call.glacier.DeleteArchive: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fd5e7a9dc50>>
2021-02-20 16:37:15,521 - MainThread - awscli.formatter - DEBUG - RequestId: mR_DR8mtULOgCU0FDDv_aec7eYp34-QtXBONWBtWXjH1yts
It's been over an hour since I executed the delete command and the vault still have the archive on it.
Has anyone had this issue? or does anyone know how I can delete a vault fully?
Update
The console allows me to delete but fails with the vault not being empty.
ps: I've posted this on AWS Forum as well but that's always slow at getting answers, so I'm trying my luck here. https://forums.aws.amazon.com/thread.jspa?threadID=335827

Per Deleting an Archive in Amazon S3 Glacier:
After you delete an archive, if you immediately download the vault
inventory, it might include the deleted archive in the list because S3
Glacier prepares vault inventory only about once a day.

Why do I get metadata service request related HTTP exceptions when using AWS CLI (in debug mode) to list a publicly available bucket?

I'm using the AWS CLI on my server (my server isn't an EC2 instance) to list the contents of an open data set on an AWS S3, and it works fine:
$ aws --no-sign-request s3 ls s3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2013-20/
PRE subset=warc/
But when I use --debug to see what's happening behind the scenes, I see exceptions caused by timeouts, such as:
2020-06-02 14:08:14,945 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
File "urllib3/connection.py", line 160, in _new_conn
File "urllib3/util/connection.py", line 84, in create_connection
File "urllib3/util/connection.py", line 74, in create_connection
socket.timeout: timed out
and:
2020-06-02 14:08:15,948 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/placement/availability-zone/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
Traceback (most recent call last):
File "urllib3/connection.py", line 160, in _new_conn
File "urllib3/util/connection.py", line 84, in create_connection
File "urllib3/util/connection.py", line 74, in create_connection
socket.timeout: timed out
It's working, in the sense that it's returning the proper response from the S3 bucket, but why do I see those exceptions in the bucket mode? Why does boto complain for the
http://169.254.169.254/latest/api/token: Connect timeout on endpoint
http://169.254.169.254/latest/meta-data/placement/availability-zone/: Connect timeout on endpoint
The version of AWS CLI that I use:
$ aws --version
aws-cli/2.0.17 Python/3.7.3 Linux/5.3.0-45-generic botocore/2.0.0dev21
I run AWS CLI on an Ubuntu GNU/Linux server 19.10.
Is there a way to prevent those exceptions? I'm asking this because I'm using the same aws cli utility to connect to non-AWS object stores, too. In this case there's no concept of "IAM" with respect to such S3-compatible object storage. This is why I'm trying to understand if I can make aws cli skip that part.
The full debug output:
$ aws --debug --no-sign-request s3 ls s3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2013-20/
2020-06-02 14:24:31,488 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.0.17 Python/3.7.3 Linux/5.3.0-45-generic botocore/2.0.0dev21
2020-06-02 14:24:31,489 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', '--no-sign-request', 's3', 'ls', 's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2013-20/']
2020-06-02 14:24:31,489 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7ff2e8f5a598>
2020-06-02 14:24:31,489 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7ff2e98a6268>
2020-06-02 14:24:31,489 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7ff2e8f1da60>
2020-06-02 14:24:31,489 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7ff2e985f840>
2020-06-02 14:24:31,491 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7ff2e90a40d0>
2020-06-02 14:24:31,491 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7ff2e90d89d8>
2020-06-02 14:24:31,491 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7ff2e8f619d8>
2020-06-02 14:24:31,492 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,492 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ls: calling handler <function add_waiters at 0x7ff2e8f619d8>
2020-06-02 14:24:31,493 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,493 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,493 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,494 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,494 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ff2e9830940>
2020-06-02 14:24:31,494 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,494 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ff2e9830940>
2020-06-02 14:24:31,494 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff2e8e9f400>
2020-06-02 14:24:31,496 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2020-06-02 14:24:32,498 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
File "urllib3/connection.py", line 160, in _new_conn
File "urllib3/util/connection.py", line 84, in create_connection
File "urllib3/util/connection.py", line 74, in create_connection
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "botocore/httpsession.py", line 263, in send
File "urllib3/connectionpool.py", line 725, in urlopen
File "urllib3/util/retry.py", line 379, in increment
File "urllib3/packages/six.py", line 735, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1229, in request
File "botocore/awsrequest.py", line 92, in _send_request
File "http/client.py", line 1275, in _send_request
File "http/client.py", line 1224, in endheaders
File "botocore/awsrequest.py", line 119, in _send_output
File "botocore/awsrequest.py", line 203, in send
File "http/client.py", line 956, in send
File "urllib3/connection.py", line 187, in connect
File "urllib3/connection.py", line 167, in _new_conn
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x7ff2e8e3b470>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "botocore/utils.py", line 221, in _fetch_metadata_token
File "botocore/httpsession.py", line 287, in send
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
2020-06-02 14:24:32,499 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (2): 169.254.169.254:80
2020-06-02 14:24:33,500 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/placement/availability-zone/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
Traceback (most recent call last):
File "urllib3/connection.py", line 160, in _new_conn
File "urllib3/util/connection.py", line 84, in create_connection
File "urllib3/util/connection.py", line 74, in create_connection
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "botocore/httpsession.py", line 263, in send
File "urllib3/connectionpool.py", line 725, in urlopen
File "urllib3/util/retry.py", line 379, in increment
File "urllib3/packages/six.py", line 735, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1229, in request
File "botocore/awsrequest.py", line 92, in _send_request
File "http/client.py", line 1275, in _send_request
File "http/client.py", line 1224, in endheaders
File "botocore/awsrequest.py", line 119, in _send_output
File "botocore/awsrequest.py", line 203, in send
File "http/client.py", line 956, in send
File "urllib3/connection.py", line 187, in connect
File "urllib3/connection.py", line 167, in _new_conn
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x7ff2e8e3b748>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "botocore/utils.py", line 264, in _get_request
File "botocore/httpsession.py", line 287, in send
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
2020-06-02 14:24:33,501 - MainThread - awscli.utils - DEBUG - Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2020-06-02 14:24:33,501 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2020-06-02 14:24:33,502 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2020-06-02 14:24:33,502 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2020-06-02 14:24:33,502 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2020-06-02 14:24:33,502 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2020-06-02 14:24:33,503 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2020-06-02 14:24:33,503 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.0.17/dist/botocore/data/endpoints.json
2020-06-02 14:24:33,510 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ff2ea856bf8>
2020-06-02 14:24:33,517 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.0.17/dist/botocore/data/s3/2006-03-01/service-2.json
2020-06-02 14:24:33,527 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7ff2ea8a4bf8>
2020-06-02 14:24:33,527 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7ff2ea8a49d8>
2020-06-02 14:24:33,532 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2020-06-02 14:24:33,539 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.0.17/dist/botocore/data/s3/2006-03-01/paginators-1.json
2020-06-02 14:24:33,540 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.s3.ListObjectsV2: calling handler <function base64_decode_input_blobs at 0x7ff2e8f222f0>
2020-06-02 14:24:33,541 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function set_list_objects_encoding_type_url at 0x7ff2ea878158>
2020-06-02 14:24:33,541 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function validate_bucket_name at 0x7ff2ea874ea0>
2020-06-02 14:24:33,541 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ff2e89b66d8>>
2020-06-02 14:24:33,541 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7ff2e89b6a20>>
2020-06-02 14:24:33,541 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function generate_idempotent_uuid at 0x7ff2ea874ae8>
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function add_expect_header at 0x7ff2ea877268>
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ff2e89b66d8>>
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function inject_api_version_header_if_needed at 0x7ff2ea878510>
2020-06-02 14:24:33,542 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '/commoncrawl?list-type=2', 'query_string': {'prefix': 'cc-index/table/cc-main/warc/crawl=CC-MAIN-2013-20/', 'delimiter': '/', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.0.17 Python/3.7.3 Linux/5.3.0-45-generic botocore/2.0.0dev21'}, 'body': b'', 'url': 'https://s3.amazonaws.com/commoncrawl?list-type=2&prefix=cc-index%2Ftable%2Fcc-main%2Fwarc%2Fcrawl%3DCC-MAIN-2013-20%2F&delimiter=%2F&encoding-type=url', 'context': {'client_region': 'aws-global', 'client_config': <botocore.config.Config object at 0x7ff2e89b62b0>, 'has_streaming_input': False, 'auth_type': None, 'encoding_type_auto_set': True, 'signing': {'bucket': 'commoncrawl'}}}
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjectsV2: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff2e89b6240>>
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <function set_operation_specific_signer at 0x7ff2ea8749d8>
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <function disable_signing at 0x7ff2ea8771e0>
2020-06-02 14:24:33,542 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7ff2e89b6a90>>
2020-06-02 14:24:33,543 - MainThread - botocore.utils - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2020-06-02 14:24:33,543 - MainThread - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/commoncrawl?list-type=2&prefix=cc-index%2Ftable%2Fcc-main%2Fwarc%2Fcrawl%3DCC-MAIN-2013-20%2F&delimiter=%2F&encoding-type=url
2020-06-02 14:24:33,543 - MainThread - botocore.utils - DEBUG - URI updated to: https://commoncrawl.s3.amazonaws.com/?list-type=2&prefix=cc-index%2Ftable%2Fcc-main%2Fwarc%2Fcrawl%3DCC-MAIN-2013-20%2F&delimiter=%2F&encoding-type=url
2020-06-02 14:24:33,543 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://commoncrawl.s3.amazonaws.com/?list-type=2&prefix=cc-index%2Ftable%2Fcc-main%2Fwarc%2Fcrawl%3DCC-MAIN-2013-20%2F&delimiter=%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/2.0.17 Python/3.7.3 Linux/5.3.0-45-generic botocore/2.0.0dev21'}>
2020-06-02 14:24:33,544 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): commoncrawl.s3.amazonaws.com:443
2020-06-02 14:24:33,988 - MainThread - urllib3.connectionpool - DEBUG - https://commoncrawl.s3.amazonaws.com:443 "GET /?list-type=2&prefix=cc-index%2Ftable%2Fcc-main%2Fwarc%2Fcrawl%3DCC-MAIN-2013-20%2F&delimiter=%2F&encoding-type=url HTTP/1.1" 200 None
2020-06-02 14:24:33,989 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'd83WTlBDKacMBr+0ReHHYaZmErezzMJxNvTChfrzCeF8gc9SFPcUGX59R7Qk3KSG3kpaK+OM/jI=', 'x-amz-request-id': 'AF67B94BB8A77A9F', 'Date': 'Tue, 02 Jun 2020 14:24:34 GMT', 'x-amz-bucket-region': 'us-east-1', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'}
2020-06-02 14:24:33,989 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>commoncrawl</Name><Prefix>cc-index/table/cc-main/warc/crawl%3DCC-MAIN-2013-20/</Prefix><KeyCount>1</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><CommonPrefixes><Prefix>cc-index/table/cc-main/warc/crawl%3DCC-MAIN-2013-20/subset%3Dwarc/</Prefix></CommonPrefixes></ListBucketResult>'
2020-06-02 14:24:33,990 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7ff2e89b69e8>>
2020-06-02 14:24:33,990 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2020-06-02 14:24:33,990 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ff2e89b66d8>>
2020-06-02 14:24:33,990 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x7ff2ea878268>
2020-06-02 14:24:33,991 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function enhance_error_msg at 0x7ff2e8f5a1e0>
2020-06-02 14:24:33,991 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7ff2e89b6630>>
PRE subset=warc/

Error thrown when trying to set SES to publish events to SNS

I'm trying to run set-identity-notification-topic on SES in my localstack and an error gets thrown. Any workaround or a solution is kindly appreciated.
I have verified my email address with SES on localstack. However I get an error when trying to run,
aws --endpoint-url=http://localhost:4579 ses set-identity-notification-topic --identity myname#domain.com --notification-type Bounce --sns-topic arn:aws:sns:us-east-1:000000000000:LOCAL_ses_responses --debug
stacktrace:
......
......
> 2019-08-07 17:04:39,539 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ses.SetIdentityNotificationTopic: calling handler 2019-08-07 17:04:39,540 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth. 2019-08-07 17:04:39,540 - MainThread - botocore.auth - DEBUG - CanonicalRequest: POST /   content-type:application/x-www-form-urlencoded; charset=utf-8 host:localhost:4579 x-amz-date:20190807T070439Z   content-type;host;x-amz-date e19fa7707096ec4f34644c4d08291160421d771325cc9d8b696cb376f4f39749 2019-08-07 17:04:39,540 - MainThread - botocore.auth - DEBUG - StringToSign: AWS4-HMAC-SHA256 20190807T070439Z 20190807/us-east-1/ses/aws4_request 4922714082f96a39b9065b91093231d793cdec8af0da19698616090b17501d28 2019-08-07 17:04:39,540 - MainThread - botocore.auth - DEBUG - Signature: 06afe49a26eb61b6420bc726da1eb79b4e0a0011fe4a246ec2e9541bed2b0c3c 2019-08-07 17:04:39,540 - MainThread - botocore.endpoint - DEBUG - Sending http request: http://localhost:4579/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/1.16.210 Python/3.7.4 Darwin/18.5.0 botocore/1.12.200', 'X-Amz-Date': b'20190807T070439Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAII5V24FP6F2TGDUQ/20190807/us-east-1/ses/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=06afe49a26eb61b6420bc726da1eb79b4e0a0011fe4a246ec2e9541bed2b0c3c', 'Content-Length': '194'}> 2019-08-07 17:04:39,541 - MainThread - urllib3.util.retry - DEBUG - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None) 2019-08-07 17:04:39,541 - MainThread - urllib3.connectionpool - DEBUG
- Starting new HTTP connection (1): localhost:4579 2019-08-07 17:04:39,550 - MainThread - urllib3.connectionpool - DEBUG
- http://localhost:4579 "POST / HTTP/1.1" 500 290 2019-08-07 17:04:39,551 - MainThread - botocore.parsers - DEBUG - Response headers: {'Content-Type': 'text/html', 'Content-Length': '290', 'Server': 'Werkzeug/0.15.5 Python/3.6.8', 'Date': 'Wed, 07 Aug 2019 07:04:39 GMT'} 2019-08-07 17:04:39,551 - MainThread - botocore.parsers
- DEBUG - Response body: b'\n500 Internal Server Error\nInternal Server Error\nThe server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.\n' 2019-08-07 17:04:39,551 - MainThread - awscli.clidriver - DEBUG - Exception caught in main() Traceback (most recent call last):   File "/Users/danuka.aluthage/.virtualenvs/myvenv/lib/python3.7/site-packages/botocore/parsers.py", line 432, in _parse_xml_string_to_dom
    parser.feed(xml_string)   File "", line None xml.etree.ElementTree.ParseError: syntax error: line 1, column 54   During handling of the above exception, another exception occurred:   Traceback (most recent call last):   File "/Users/danuka.aluthage/.virtualenvs/myvenv/lib/python3.7/site-packages/awscli/clidriver.py", line 207, in main
    return command_table[parsed_args.command](remaining, parsed_args)   File "/Users/danuka.aluthage/.virtualenvs/myvenv/lib/python3.7/site-packages/awscli/clidriver.py", line 348, in call
.....
.....
Unable to parse response (syntax error: line 1, column 54), invalid XML received:
b'\n500 Internal Server Error\nInternal Server Error\nThe server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.\n'

AWS CLI access to S3 returns error 104

AWS reports that all their services are operating normally, but I get Error 104 'Connection reset by peer' on all my S3 requests via the AWS CLI. Yesterday it was working fine. Do you have any suggestions for how to fix this?
$ aws --version
aws-cli/1.11.32 Python/2.6.6 Linux/2.6.32-642.11.1.el6.x86_64 botocore/1.4.89
$ aws s3 ls s3://
('Connection aborted.', error(104, 'Connection reset by peer'))
$ aws s3api list-buckets
('Connection aborted.', error(104, 'Connection reset by peer'))
Here is the list-buckets command with debugging info:
$ aws s3api list-buckets --debug
2016-12-21 09:25:11,926 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.11.32 Python/2.6.6 Linux/2.6.32-642.11.1.el6.x86_64 botocore/1.4.89
2016-12-21 09:25:11,927 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3api', 'list-buckets', '--debug']
2016-12-21 09:25:11,927 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2016-12-21 09:25:11,927 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2016-12-21 09:25:11,929 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.6/site-packages/botocore/data/s3/2006-03-01/service-2.json
2016-12-21 09:25:11,988 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.s3: calling handler
2016-12-21 09:25:11,988 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: s3
2016-12-21 09:25:11,990 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3api: calling handler
2016-12-21 09:25:11,992 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.6/site-packages/botocore/data/s3/2006-03-01/waiters-2.json
2016-12-21 09:25:11,993 - MainThread - awscli.clidriver - DEBUG - OrderedDict()
2016-12-21 09:25:11,994 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.list-buckets: calling handler
2016-12-21 09:25:11,994 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.list-buckets: calling handler
2016-12-21 09:25:11,994 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.list-buckets: calling handler
2016-12-21 09:25:11,996 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.6/site-packages/botocore/data/s3/2006-03-01/paginators-1.json
2016-12-21 09:25:11,996 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.list-buckets: calling handler
2016-12-21 09:25:11,996 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.s3api.list-buckets: calling handler >
2016-12-21 09:25:11,997 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.s3api.list-buckets: calling handler >
2016-12-21 09:25:11,997 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.list-buckets.cli-input-json: calling handler
2016-12-21 09:25:11,997 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.list-buckets.generate-cli-skeleton: calling handler
2016-12-21 09:25:11,997 - MainThread - botocore.hooks - DEBUG - Event calling-command.s3api.list-buckets: calling handler >
2016-12-21 09:25:11,997 - MainThread - botocore.hooks - DEBUG - Event calling-command.s3api.list-buckets: calling handler >
2016-12-21 09:25:11,998 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2016-12-21 09:25:11,998 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2016-12-21 09:25:11,998 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.6/site-packages/botocore/data/endpoints.json
2016-12-21 09:25:12,030 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2016-12-21 09:25:12,033 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler
2016-12-21 09:25:12,033 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler
2016-12-21 09:25:12,034 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2016-12-21 09:25:12,037 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2016-12-21 09:25:12,037 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListBuckets: calling handler
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListBuckets: calling handler >
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListBuckets: calling handler
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListBuckets: calling handler
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListBuckets: calling handler >
2016-12-21 09:25:12,038 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListBuckets) (verify_ssl=True) with params: {'body': '', 'url': u'https://s3.amazonaws.com/', 'headers': {'User-Agent': 'aws-cli/1.11.32 Python/2.6.6 Linux/2.6.32-642.11.1.el6.x86_64 botocore/1.4.89'}, 'context': {'client_region': 'us-east-1', 'signing': {'bucket': None}, 'has_streaming_input': False, 'client_config': }, 'query_string': '', 'url_path': u'/', 'method': u'GET'}
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListBuckets: calling handler >
2016-12-21 09:25:12,038 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListBuckets: calling handler
2016-12-21 09:25:12,039 - MainThread - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2016-12-21 09:25:12,039 - MainThread - botocore.auth - DEBUG - HTTP request method: GET
2016-12-21 09:25:12,039 - MainThread - botocore.auth - DEBUG - StringToSign:
GET
Wed, 21 Dec 2016 14:25:12 GMT
/
2016-12-21 09:25:12,044 - MainThread - botocore.endpoint - DEBUG - Sending http request:
2016-12-21 09:25:12,044 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): s3.amazonaws.com
2016-12-21 09:25:12,078 - MainThread - botocore.endpoint - DEBUG - ConnectionError received when sending HTTP request.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/botocore/endpoint.py", line 204, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/usr/lib/python2.6/site-packages/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.6/site-packages/botocore/vendored/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer'))

I experienced this too, but when I specify the region where my S3 bucket reside, the problem solved.
aws s3 --region ap-southeast-1 cp file100Mb.zip s3://mybucket.domain.xyz/path/
You can try and let me know how it does for you.

For me, I installed the awscli via the apt-get command (Ubuntu 16). It was working fine for us-east-1 region. But when I tried to upload files to different regions to a newly created bucket it started giving me this error. By removing the aws cli and installing via pip resolved this issue. It seems the version which is installed via apt-get is not updated for ubuntu 16.
sudo apt-get install pip
pip install awscli

I switched my VM networking from "NAT" to "bridged" and S3 magically started working. When it stopped working, I had not change anything manually; I hope VMware was not changing the network by itself. But if you get error 104, check your network settings.

We Keep Coding

c++ django amazon-web-services regex python-2.7 google-cloud-platform list unit-testing opengl ember.js

Jetty is throwing exceptions on ktor i think issue with http2 - jetty

Related

How to find the reason for a jetty 400 error?

Is anyone else have issue deleting AWS Glacier archives?

Why do I get metadata service request related HTTP exceptions when using AWS CLI (in debug mode) to list a publicly available bucket?

Error thrown when trying to set SES to publish events to SNS

AWS CLI access to S3 returns error 104

Categories

Resources