SdkClientException: Unable to execute HTTP request: Connection reset - amazon-web-services

I'm trying to upload file to AWS s3 bucket. I'm continuously getting this exception "SdkClientException: Unable to execute HTTP request: Connection reset".
I'm trying to upload by inputstream
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(Regions.US_EAST_2)
.withCredentials(new AWSStaticCredentialsProvider(cred))
.build();
try {
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(bytesArray.length);
metadata.setContentType("text/csv");
s3Client.putObject(new PutObjectRequest(appConfig.getBucketName(), fileName, inStream, metadata).withCannedAcl(CannedAccessControlList.PublicRead));
}
----
Below is the exception in detail. What could be the reason? My S3 bucket has IAM user with S3 full access policy. Is this issue related to network or any other settings?
com.amazonaws.SdkClientException: Unable to execute HTTP request: Connection reset
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1136)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1082)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
-------
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:142)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142

I found it is occurring due to the firewall. If you set logging.level.com.amazonaws at DEBUG then it will show SSL handshake failing.

If you are running the application behind a proxy, this may happens. You can set the proxy as,
public static void setProxy() {
System.setProperty("http.proxyHost", "<http-proxy-url>");
System.setProperty("http.proxyPort", "<port>");
System.setProperty("https.proxyHost", "<https-proxy-url>");
System.setProperty("https.proxyPort", "<port>");
}

Related

Getting access denied error while uploading file to amazon s3 using tranferutility

Getting the following error while running the application in logcat
D/com.amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: null; Status Code: 403; Error Code: AccessDenied; Request ID: ZFW34X6QRZEBNR23), S3 Extended Request ID: cCcs8EOdirUSRRd9MyoUcMWDTjEvFLN/cqSYbAT3NCLR/hDSjxI/ATNX63Y9cbyDNABi8MnTauY=
Without using tranfer utility code works fine. When using transfer utility error occurs. Kindly find the code here
getApplicationContext().startService(new Intent(getApplicationContext(), TransferService.class));
AWSMobileClient.getInstance().initialize(getApplicationContext(), new Callback<UserStateDetails>() {
#Override
public void onResult(UserStateDetails result) {
Log.i("AWSMobile Clinet","initialised awsmobile client");
TransferUtility transferUtility=TransferUtility.builder().context(getApplicationContext())
.awsConfiguration(AWSMobileClient.getInstance().getConfiguration())
.s3Client(new AmazonS3Client(AWSMobileClient.getInstance()))
.build();
try {
File exampleFile = new File("/data/data/com.dieto.app/databases/user.db");
TransferObserver uploadObserver=transferUtility.upload( "dietodata144832-
dev",number+".db",exampleFile);
}
catch (Exception e){
Log.e("File load","File load failed"+e.toString());
}
Check if the user has access to the that specific S3 bucket or not. You can do this by navigating to IAM on AWS console, check the policies attached to your user and check if the user has AWSS3FullAccess policy attached or not.
After you attach this policy, you shouldn't get access denied error.

Jenkins Unable to load AWS credentials from any provider in the chain

I'm trying to send emails using AWS SES Simple Email Service in my SpringBoot project
Knowing that I verified my email on AWS, and also created an AWS_ACCESS_KEY and AWS_SECRET_KEY.
Here's my code:
public static void amazonSimpleEmailService(String to, String subject, String htmlBody, String textBody) {
AmazonSimpleEmailService client = AmazonSimpleEmailServiceClientBuilder.standard()
.withCredentials(
new AWSStaticCredentialsProvider(new BasicAWSCredentials(AWS_ACCESS_KEY, AWS_SECRET_KEY)))
.withRegion(Regions.EU_WEST_1).build();
SendEmailRequest request = new SendEmailRequest().withDestination(new Destination().withToAddresses(to))
.withMessage(new Message()
.withBody(new Body().withHtml(new Content().withCharset("UTF_8").withData(htmlBody))
.withText(new Content().withCharset("UTF-8").withData(textBody)))
.withSubject(new Content().withCharset("UTF-8").withData(subject)))
.withSource(MAIL_ADMIN);
client.sendEmail(request);
}
When Building my code on Jenkins (EC2 instance) , this error occurs
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper#55fe41ea: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/, com.amazonaws.auth.profile.ProfileCredentialsProvider#2aceadd4: profile file cannot be null]
I doubt that am missing something but I cannot find what it is.

AWSLambdaClient invoke giving exception: com.amazonaws.SdkClientException: Unable to execute HTTP request: The target server failed to respond

I have an existing aws lambda function and I am trying to invoke it using AWSLambdaClient (library: aws-java-sdk-lambda). I get the response if the runtime for lambda is less than 5 mins. But if the lambda runtime is more than 5 mins I get the exception:
com.amazonaws.SdkClientException: Unable to execute HTTP request: The target server failed to respond
What configuration I am missing while setting up lambda client ?
Full stack trace:
Exception in thread "main" com.amazonaws.SdkClientException: Unable to execute HTTP request: The target server failed to respond
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1163)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1109)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:758)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:732)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:714)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:674)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:656)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:520)
at com.amazonaws.services.lambda.AWSLambdaClient.doInvoke(AWSLambdaClient.java:3154)
at com.amazonaws.services.lambda.AWSLambdaClient.invoke(AWSLambdaClient.java:3121)
at com.amazonaws.services.lambda.AWSLambdaClient.invoke(AWSLambdaClient.java:3110)
at com.amazonaws.services.lambda.AWSLambdaClient.executeInvoke(AWSLambdaClient.java:1735)
at com.amazonaws.services.lambda.AWSLambdaClient.invoke(AWSLambdaClient.java:1707)
at LambdaTest.main(LambdaTest.java:48)
Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:82)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1285)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1101)
... 12 more
I have already increased my lambda max timeout to 15 mins on aws lambda console. Also I have made code changes and set ClientExecutionTimeout, ConnectionTimeout, RequestTimeout and SocketTimeout to 15 mins (900 sec) in AWSLambdaClient configuration.
ClientConfiguration configuration = new ClientConfiguration()
.withMaxErrorRetry(0)
// These is measured in milliseconds.
.withClientExecutionTimeout(900 * 1000)
.withConnectionTimeout(900 * 1000)
.withRequestTimeout(900 * 1000)
.withSocketTimeout(900 * 1000);
AWSLambda lambdaClient = AWSLambdaClientBuilder
.standard()
.withRegion(region)
.withClientConfiguration(configuration)
.withCredentials(new AWSStaticCredentialsProvider(creds)).build();
InvokeRequest request = new InvokeRequest()
.withFunctionName(functionName)
.withInvocationType(InvocationType.RequestResponse)
.withLogType(LogType.Tail);
InvokeResult result = lambdaClient.invoke(request);
String responseLog = new String(Base64.getDecoder().decode(result.getLogResult()), StandardCharsets.UTF_8);
System.out.println(responseLog);
I am using apache HttpClient: 4.5.5
I saw some question https://stackoverflow.com/a/44703601/9669982 I tried some configurations also:
configuration.setValidateAfterInactivityMillis(900 * 1000);
Also to try more I also used:
configuration.setConnectionMaxIdleMillis(900*1000);
configuration.setUseTcpKeepAlive(true);
But no luck.
On AWS Cloudwatch I see that my lambda function is running fine and full task was executed and no trace of any kind of error was there on cloudwatch but I am getting the above exception.
I can't get what I am missing.
As you already know issue
I get the response if the runtime for lambda is less than 5 mins. But
if the lambda runtime is more than 5 mins I get the exception:
Your connections that are kept alive by the connection manager become stale.That is, the target server shuts down the connection on its end without HttpClient being able to react to that event.
You need to add setHttpRequestRetryHandler for your HTTP client as answered in following link.
get NoHttpResponseException for load testing

Downloading file from s3 freezes and throws exception while network goes down temporary

I am using AWS S3 SDK to download a file from S3 Bucket via TransferManager.
Cases
1. File download is successful if there is no interruption in internet connectivity.
2. However in case of network failure file downloading freezes for some time but resumes successfully if Internet is up before socket timeout by default(50 sec).
3.On increasing socket time out from my side
#Bean
public ClientConfiguration clientConfiguration() {
this.clientConfiguration = new ClientConfiguration();
clientConfiguration.setSocketTimeout(600*1000);
return this.clientConfiguration;
}
file downloading freezes,downloads some bytes and freezes again after internet is up after hiccup. Later it throws connection reset exception.
com.amazonaws.SdkClientException: Unable to store object contents to disk: Connection reset at com.amazonaws.services.s3.internal.ServiceUtils.downloadToFile(ServiceUtils.java:313)
at com.amazonaws.services.s3.internal.ServiceUtils.downloadObjectToFile(ServiceUtils.java:270)
at com.amazonaws.services.s3.internal.ServiceUtils.retryableDownloadS3ObjectToFile(ServiceUtils.java:402)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1455)
at com.amazonaws.services.s3.transfer.internal.DownloadPartCallable.call(DownloadPartCallable.java:59)
at com.amazonaws.services.s3.transfer.internal.DownloadPartCallable.call(DownloadPartCallable.java:31)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593)
at sun.security.ssl.InputRecord.read(InputRecord.java:532)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198)
at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:118)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at com.amazonaws.services.s3.internal.ServiceUtils.downloadToFile(ServiceUtils.java:307)
My question is how can I configure my client so that one can resume download of a file at anytime even if the network breaks down?

AWS JAVA SDK CLI : java.net.SocketTimeoutException

I am making an API call , and I am trying to do that through sts assume-role API.
My Java program is :
STSAssumeRoleSessionCredentialsProvider = new STSAssumeRoleSessionCredentialsProvider(credentials,"arn:aws:iam::093937234853:role/CapOne-CrossAccount-CustomRole-ReadOnly","123");
ec2Client.setRegion(getRegion());
this.setDescribeInstancesResult(ec2Client.describeInstances());
In credentials variable above I am passing my secret id and key.
The error I am getting is :
Sep 23, 2015 3:25:48 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Unable to execute HTTP request: connect timed out
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
Untill I was using the STSAssume role , It was working fine . But due to a requiremnt I have to use roles. So I am sure , my credentials variables is being set properly.
I was not setting my proxy parameter properly , after setting that , it started working :
ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setProxyHost("proxy.kdc.capitalone.com");
clientConfiguration.setProxyPort(8099);
clientConfiguration.setProxyUsername("EID");
clientConfiguration.setProxyPassword("PWD");
clientConfiguration.setPreemptiveBasicProxyAuth(false);