Amazon S3 service ConnectionPoolTimeoutException: Timeout waiting for connection from pool - amazon-web-services

Question

When i work with s3 on Amazon, I keep getting the ugly ConnectionPoolTimeoutException.
The problem is that due to the front end of the application, i can not close the opened s3 objects before front end is done with them so i have implemented this solution :
#Autowired
private AmazonS3 s3client; // credentials are set properly.
private static List<S3Object> openedObjects = new ArrayList<S3Object>();
// initialize bucket :
private String bucketName = "myShinyNewBucket";
private synchronized boolean initBucket(){
try{
Boolean exists = null;
try{
exists = s3client.doesBucketExist(bucketName);
}catch(Exception e1){
System.out.println("\n\n\tToo many opened objects ; closing...\n\n");
deleteOpenedS3Objects();
exists = s3client.doesBucketExist(bucketName);
}
if(exists!=null){
if(!exists){
s3client.createBucket(new CreateBucketRequest(bucketName));
}
return true;
}
}
catch(Exception e){
System.out.println("\n\n\tFailed to initialize bucket.\n");
e.printStackTrace();
}
return false;
}
private synchronized void deleteOpenedS3Objects(){
System.out.println("\n\tClosing opened objects...");
try{
for(int i=0 ; i<openedObjects.size() ; i++){
openedObjects.get(i).close();
openedObjects.remove(i);
}
}catch(Exception e1){
System.out.println("\tCould not close all opened s3 objects, only the first "+i);
}
System.out.println("\tTrying again :\n\n");
}
// GET :
public final String getFromAWS(final String amazonName){
S3Object s3object = null;
if(initBucket()){
try{
try{
s3object = s3client.getObject(new GetObjectRequest(bucketName, amazonName));
}catch(AmazonClientException e){
deleteOpenedS3Objects();
s3object = s3client.getObject(new GetObjectRequest(bucketName, amazonName));
}
openedObjects.add(s3object);
return s3object.getObjectContent().getHttpRequest().getURI().toString();
}catch(Exception e1){
if (((AmazonS3Exception)e1).getStatusCode() == HttpStatus.SC_NOT_FOUND){
System.out.println("\n\nNo such object in bucket.\n");
}
else{
System.out.println("\n\n\tCould not read bject from bucket.\n\n");
e1.printStackTrace();
}
}
}
return null;
}
Yet, the exception is still happening.
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:286) ~[httpclient-4.5.1.jar!/:4.5.1]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263) ~[httpclient-4.5.1.jar!/:4.5.1]
at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70) ~[aws-java-sdk-core-1.11.8.jar!/:na]
at com.amazonaws.http.conn.$Proxy188.get(Unknown Source) ~[na:na]
...
Only when i do ctrl+c in console, does it get to the part where it is closing the opened s3 connections :
... Caused by: java.lang.InterruptedException: Operation interrupted
at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:142) ~[httpcore-4.4.4.jar!/:4.4.4]
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306) ~[httpcore-4.4.4.jar!/:4.4.4]
at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64) ~[httpcore-4.4.4.jar!/:4.4.4]
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192) ~[httpcore-4.4.4.jar!/:4.4.4]
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185) ~[httpcore-4.4.4.jar!/:4.4.4]
at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107) ~[httpcore-4.4.4.jar!/:4.4.4]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276) ~[httpclient-4.5.1.jar!/:4.5.1]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263) ~[httpclient-4.5.1.jar!/:4.5.1]
at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70) ~[aws-java-sdk-core-1.11.8.jar!/:na]
at com.amazonaws.http.conn.$Proxy188.get(Unknown Source) ~[na:na]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) ~[httpclient-4.5.1.jar!/:4.5.1]
... 148 common frames omitted
Too many opened objects. // <-- This is where it catches it.
Closing opened objects...
Could not close all opened s3 objects.
Trying again :
Failed to initialize bucket.
Again, I am unfortunately not in a position where i can close opened s3objects before i leave the functions in my s3-client class. The only hope i have had, was to wait until TimeoutException happens, catch it, then closing all opened objects and trying again.
However, i can't seem to catch it in the right place.
Please, help.
Thank you.

Answer 1

We must use try with resources whenever we are dealing with any type of file read or write. This will auto close the resources used, even if we face any sort of exception.
In My case : I was getting 404 while reading from S3 for bulk operation, which i ignored while writing script by logging the error and forget to close the connection in finally block.
Example snippet:
static String readFirstLineFromFile(String path) throws IOException {
try (BufferedReader br =
new BufferedReader(new FileReader(path))) {
return br.readLine();
}
}
Note:
Try with resources have support from JDK 8 and above.
All the resources declared in try() must implement the AutoCloseable interface.

Answer 2

I think the sollution is to catch and process the ugly TimeoutException in my #ControllerAdvice class. I have done so and so far i have not had it happen in the app. I'll post a confirmation once i'm sure.