Unable to listObjects from GCS bucket - amazon-web-services

Question

I am trying to listObjects from GCS bucket using latest aws-sdk java library.
Refer code snippet here
ClientConfiguration clientConfiguration = new ClientConfiguration();
// Solution is update the Signer Version.
clientConfiguration.setSignerOverride("S3SignerType");
AWSCredentials awsCredentials = new BasicAWSCredentials("XXX","XXX");
AmazonS3 amazonS3Client = AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(awsCredentials))
.withClientConfiguration(clientConfiguration)
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("https://storage.googleapis.com","Multi-Regional")).build();
String bucketName = "bucket_name";
// List Objects
amazonS3Client.listObject(bucketName);
But receiving invalid arguments.Refer Error and DEBUG logs
Also I am able to do with getObjects and putObjects with above amazonS3Client.
Any ideas?
2017-11-13 17:54:15,360 [main] DEBUG com.amazonaws.request - Sending Request: GET https://bucket_name.storage.googleapis.com / Parameters: ({"encoding-type":["url"]}Headers: (User-Agent: aws-sdk-java/1.11.158 Linux/4.10.0-38-generic Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11/1.8.0_131, amz-sdk-invocation-id: 121cd76e-1374-4e5d-9e68-be22ee2ad17a, Content-Type: application/octet-stream, )
2017-11-13 17:54:16,316 [main] DEBUG com.amazonaws.request - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid argument. (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: null), S3 Extended Request ID: null
Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Invalid argument. (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: null), S3 Extended Request ID: null
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
at

Answer 1

S3 has a parameter to its object list call called "encodingtype" that, when set to "url", encodes characters that can't be rendered natively in XML 1.0 using URL encoding. The client library appears to be using that flag. I don't believe that GCS's XML API supports that parameter, and so the call will fail with an InvalidArgument error.
You can probably avoid this by using a ListObjectRequest and calling setEncodingType(null), although I haven't tried.

Answer 2

I'm in the same boat but with the aws-sdk php library. I tracked down the issue to the constructor for the S3Client class where a bunch of middleware are added, one of which is setting the encoding type. Commenting out this line allows me to successfully perform the request, so I know I'm on the right track.
$stack->appendSign(PutObjectUrlMiddleware::wrap(), 's3.put_object_url');
$stack->appendSign(PermanentRedirectMiddleware::wrap(), 's3.permanent_redirect');
$stack->appendInit(Middleware::sourceFile($this->getApi()), 's3.source_file');
$stack->appendInit($this->getSaveAsParameter(), 's3.save_as');
$stack->appendInit($this->getLocationConstraintMiddleware(), 's3.location');
// $stack->appendInit($this->getEncodingTypeMiddleware(), 's3.auto_encode');
$stack->appendInit($this->getHeadObjectMiddleware(), 's3.head_object');
See if aws-sdk for java at least gives you some options to conditionally apply the middleware, but it appears that it's no dice with the php version.

Answer 3

As Brandon and Pez have noticed, GCS does not like the EncodingType header that's being added by the S3Client natively.
Fortunately, there's an easy way to fix this using a piece of middleware. This avoids edits to the vendor folder which should generally be avoided.
use Aws\Middleware;
$client = new S3Client([
'credentials' => [
'key' => 'XXXX',
'secret' => 'YYYY'
],
'region' => 'europe',
'endpoint' => 'https://storage.googleapis.com',
'version' => 'latest'
]);
$middleware = Middleware::tap(function ($command, $request = null) {
unset($command['EncodingType']);
});
$client->getHandlerList()->appendInit($middleware, 'encode-type-interceptor');
See also: https://blog.bandhosting.nl/blog/avoid-listobjects-invalid-query-parameter-s-encoding-type-errors-when-using-s3-with-google-cloud-storage

Answer 4

I ran into the same issue and found that Amazon Java API includes several hooks during the S3 calls that can be used to remove the encoding-type from the HTTP request.
{
return AmazonS3ClientBuilder.standard()//
.withCredentials(getAwsCredentialsProvider())//
.withEndpointConfiguration(getEndpointConfiguration(regionName))//
.withRequestHandlers(new GoogleRequestHandler()) // IMPORTANT PART
.build();
}
public class GoogleRequestHandler extends RequestHandler2 {
#Override
public void beforeRequest(Request<?> request) {
// google does not support the encoding-type parameter so just remove it from the request
// This appears to be only true for ListObjects
if (request.getOriginalRequest() instanceof ListObjectsRequest) {
Map<String, List<String>> params = request.getParameters();
params.remove("encoding-type");
}
}
}
See RequestHandler2 for more documentation.