I'm refactoring a bunch of code from AWS V1 sdk to V2 and am stuck creating the proxy configuration for a GlueClient.
V1 code:
ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setProxyPort(8443);
clientConfiguration.setProtocol(Protocol.HTTPS);
V2 code:
ClientOverrideConfiguration.builder(). ??? .build();
I can't find any good examples of how to do this. There do not seem to be any proxy options in the replacement class for V2
I'm going to assume you're using the Java SDK based on classnames, in which case you set the proxy configuration on the HTTPClientBuilder. Here's an example with the default (Apache) client-builder, taken from this gist, which shows a complete program and compares it to the V1 proxy configuration.
ProxyConfiguration config = ProxyConfiguration.builder()
.endpoint(new URI("http://localhost:3128"))
.addNonProxyHost("169.254.169.254")
.useSystemPropertyValues(Boolean.FALSE)
.build();
ApacheHttpClient.Builder clientBuilder = ApacheHttpClient.builder()
.proxyConfiguration(config);
StsClient client = StsClient.builder()
.httpClientBuilder(clientBuilder)
.build();
You might also be interested in the ways to automatically configure a proxy via environment variables or system properties, which I've documented here. TL;DR: it's a hot mess.
Related
I am trying to use my code to access S3 using aws sdk and in a java service. But I receiving following exception.
Code I use to as follows.
configure the builder.
ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setRetryPolicy( PredefinedRetryPolicies.getDefaultRetryPolicyWithCustomMaxRetries(5));
AmazonS3ClientBuilder s3ClientCommonBuilder = AmazonS3ClientBuilder.standard()
.withClientConfiguration(clientConfiguration)
.withForceGlobalBucketAccessEnabled(true)
.withPathStyleAccessEnabled(true);
s3ClientCommonBuilder.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(awsKey, awsSecret)));
return s3ClientCommonBuilder;
create the AwsS3Client
AmazonS3ClientBuilder internalS3ClientBuilder =
createS3ClientCommonBuilder(s3bucket, awsKey, awsSecret);
if (internalServiceEndpoint != null && !internalServiceEndpoint.isEmpty()) {
internalS3ClientBuilder.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(internalServiceEndpoint, Region.EU_Ireland.getFirstRegionId()));
}
And then at run time when the client is attempting to connect to s3 following exception is thrown.
1) Error in custom provider, com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
while locating AwsS3ClientProvider
at org.test.ApiServerBootstrap.configure(ApiServerBootstrap.java:103)
while locating org.test.AWSS3Client
for the 3rd parameter of org.testDataUploader.<init>(DataUploader.java:60)
at org.test.DataUploader.class(DataUploader.java:49)
while locating org.test.DataUploader
for the 3rd parameter of org.test.StoreService.<init>(StoreService.java:56)
at org.test.services.v3.StoreService.class(StoreService.java:53)
while locating org.test.StoreService
for the 1st parameter of org.test.verticles.EventBusConsumerVerticle.<init>
(EventBusConsumerVerticle.java:35)
at
org.test.verticles.EventBusConsumerVerticle.class(EventBusConsumerVerticle.java:35)
while locating org.test.verticles.EventBusConsumerVerticle
for the 3rd parameter of org.test.ApiServer.<init>(ApiServer.java:52)
while locating org.test.ApiServer
1 error
at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:226)
at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1053)
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1086)
at org.test.ApiServer.main(ApiServer.java:46)
Caused by: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:462)
at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)
at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
at org.test.AWSS3Client.create(AWSS3Client.java:99)
I have tested this with some other computers to verify it is my aws client configuration that is is wrong.
My .aws/config file looks as follows
$ cat .aws/config
[default]
region = eu-west-1
output = json
my .aws/credintials look like follows.
$ cat .aws/credentials
[default]
aws_access_key_id=XXXXXXXX.....
aws_secret_access_key=XXXXXXXX
my aws client verison is as follows.
$ aws --version
aws-cli/2.2.40 Python/3.8.8 Linux/5.11.0-34-generic exe/x86_64.ubuntu.20 prompt/off
I appreciate if some one can figure out what am I missing for the aws client to connect to s3.
You want to make sure you are using the DefaultAWSCredentialsProviderChain() class when building your S3Client.
Also you can create an environment variable AWS_REGION=eu-west-1
I have an application instance running in EKS with the following variables set:
declare -x AWS_DEFAULT_REGION="us-west-2"
declare -x AWS_REGION="us-west-2"
declare -x AWS_ROLE_ARN="xxxxx"
declare -x AWS_WEB_IDENTITY_TOKEN_FILE="/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
As I understand there is a default Java SDK authorization chain that contains com.amazonaws.auth.WebIdentityTokenCredentialsProvider which builds com.amazonaws.services.securitytoken.AWSSecurityTokenService under the hood.
But I can't realize how this circular dependency is solved? I mean you need to specify credentials during creation of AWSSecurityTokenService but credentials create service itself.
I have practical requirements to do that, I want to customize endpoint in sts client but can't since circular dependency.
AWSSecurityTokenServiceClientBuilder.standard()
.withCredentials(new STSAssumeRoleWithWebIdentitySessionCredentialsProvider.Builder(
"arn",
"session",
"tokenfile")
.withStsClient(xxxx)
.build())
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://localhost:4566", null))
.build()
It was easy. It's just done with anonymous auth (https://github.com/aws/aws-sdk-java/blob/1.11.792/aws-java-sdk-sts/src/main/java/com/amazonaws/auth/STSAssumeRoleWithWebIdentitySessionCredentialsProvider.java#L122-L125)
return AWSSecurityTokenServiceClientBuilder.standard()
.withClientConfiguration(clientConfiguration)
.withCredentials(new AWSStaticCredentialsProvider(new AnonymousAWSCredentials()))
.build();
I want to test my AWS code locally so I have to set a proxy to a AWS client.
There is a proxy host (http://user#pass:my-corporate-proxy.com:8080) set in my environment via a variable HTTPS_PROXY.
I didn't find a way how to set the proxy as whole so I came up with this code:
AmazonSNS sns = AmazonSNSClientBuilder.standard()
.withClientConfiguration(clientConfig(System.getenv("HTTPS_PROXY")))
.withRegion(Regions.fromName(System.getenv("AWS_REGION")))
.withCredentials(new DefaultAWSCredentialsProviderChain())
.build();
ClientConfiguration clientConfig(String proxy) {
ClientConfiguration configuration = new ClientConfiguration();
if (proxy != null && !proxy.isEmpty()) {
Matcher matcher = Pattern.compile("(\\w{3,5})://((\\w+):(\\w+)#)?(.+):(\\d{1,5})").matcher(proxy);
if (!matcher.matches()) {
throw new IllegalArgumentException("Proxy not valid: " + proxy);
}
configuration.setProxyHost(matcher.group(5));
configuration.setProxyPort(Integer.parseInt(matcher.group(6)));
configuration.setProxyUsername(matcher.group(3));
configuration.setProxyPassword(matcher.group(4));
}
return configuration;
}
The whole method clientConfig is only boilerplate code.
Is there any elegant way how to achieve this?
As far as I can tell while using AWS SDK V1 (1.11.840), if you have environment variables such as HTTP(S)_PROXY or http(s)_proxy set at runtime, or properties like http(s).proxyHost, proxyPort, proxyUser, and proxyPassword passed to your application, you don't have to set any of that. It gets automatically read into the newly created ClientConfigiration.
As, such you'd only want to set the ProxyAuthenticationMethod, if needed.
ClientConfiguration clientConfig(ProxyAuthenticationMethod authMethod) {
ClientConfiguration conf = new ClientConfiguration();
List<ProxyAuthenticationMethod> proxyAuthentication = new ArrayList<>(1);
proxyAuthentication.add(authMethod);
conf.setProxyAuthenticationMethods(proxyAuthentication);
return conf;
}
ProxyAuthenticationMethod can be ProxyAuthenticationMethod.BASIC or DIGEST or KERBEROS or NTLM or SPNEGO
I can confirm setting the parameters "http(s).proxyHost" (and others) work out of the box, you need however to specify a port, otherwise AWS SDK (1) will not pick it up.
java -Dhttps.proxyHost=proxy.company.com -Dhttps.proxyPort=8080 -Dhttps.proxyUser=myUsername -Dhttps.proxyPassword=myPassword <app>
Username & passsword are optional.
See for more info:
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/doc-files/net-properties.html
and
What Java properties to pass to a Java app to authenticate with a http proxy
I enabled "Transfer acceleration" on my bucket. But I dont see any improvement in speed of Upload in my C++ application. I have waited for more than 20 minutes that is mentioned in AWS Documentation.
Does the SDK support "Transfer acceleration" by default or is there a run time flag or compiler flag? I did not spot anything in the SDK code.
thanks
Currently, there isn't a configuration option that simply turns on transfer acceleration. You can however, use endpoint override in the client configuration to set the accelerated endpoint.
What I did to enable a (working) transfer acceleration:
set in the bucket configuration on the AWS panel "Transfer Acceleration" to enabled.
add to the IAM user that I use inside my C++ application the permission s3::PutAccelerateConfiguration
Add the following code to the s3 transfer configuration (bucket_ is your bucket name, the final URL must match the one shown in the AWS panel "Transfer Acceleration"):
Aws::Client::ClientConfiguration config;
/* other configuration options */
config.endpointOverride = bucket_ + ".s3-accelerate.amazonaws.com";
Ask for acceleration to the bucket before transfer... (docs in here )
auto s3Client = Aws::MakeShared<Aws::S3::S3Client>("Uploader",
Aws::Auth::AWSCredentials(id_, key_), config);
Aws::S3::Model::PutBucketAccelerateConfigurationRequest bucket_accel;
bucket_accel.SetAccelerateConfiguration(
Aws::S3::Model::AccelerateConfiguration().WithStatus(
Aws::S3::Model::BucketAccelerateStatus::Enabled));
bucket_accel.SetBucket(bucket_);
s3Client->PutBucketAccelerateConfiguration(bucket_accel);
You can check in the detailed logs of the AWS sdk that your code is using the accelerated entrypoint and you can also check that before the transfer start there is a call to /?accelerate (info)
What worked for me:
Enabling S3 Transfer Acceleration within AWS console
When configuring the client, only utilize the accelerated endpoint service:
clientConfig->endpointOverride = "s3-accelerate.amazonaws.com";
#gabry - your solution was extremely close, I think the reason it wasn't working for me was perhaps due to SDK changes since originally posted as the change is relatively small. Or maybe because I am constructing put object templates for requests used with the transfer manager.
Looking through the logs (Debug level) the SDK automatically concatenates the bucket used in transferManager::UploadFile() with the overridden endpoint. I was getting unresolved host errors as the requested host looked like:
[DEBUG] host: myBucket.myBucket.s3-accelerate.amazonaws.com
This way I could still keep the same S3_BUCKET macro name while only selectively calling this when instantiating a new configuration for upload.
e.g.
<<
...
auto putTemplate = new Aws::S3::Model::PutObjectRequest();
putTemplate->SetStorageClass(STORAGE_CLASS);
transferConfig->putObjectTemplate = *putTemplate;
auto multiTemplate = new Aws::S3::Model::CreateMultipartUploadRequest();
multiTemplate->SetStorageClass(STORAGE_CLASS);
transferConfig->createMultipartUploadTemplate = *multiTemplate;
transferMgr = Aws::Transfer::TransferManager::Create(*transferConfig);
auto transferHandle = transferMgr->UploadFile(localFile, S3_BUCKET, s3File);
transferMgr = Aws::Transfer::TransferManager::Create(*transferConfig);
...
>>
I am trying to automate the process of creating application version for an existing elastic beanstalk application through java api and command line arguments.
while implementing createApplicationVersion() of AWSElasticBeanstalkClient I am getting error for the below code snipplet.
Note: I am passing the endpoint for AWSElasticBeanstalkClient as US East-1 (N.Virginia) or the environment url for the existing application.
ArrayList<String> s3SourceBundleList = AmazonS3BucketUploadApp.doBucketUploadFromLocal(sourceLocation);
String bucketName = s3SourceBundleList.get(0);
String keyName = java.net.URLEncoder.encode(s3SourceBundleList.get(1), "UTF-8");
//String keyName = s3SourceBundleList.get(1);
S3Location s3SourceBundle = new S3Location();
s3SourceBundle.setS3Bucket(bucketName);
s3SourceBundle.setS3Key(keyName);
createApplicationVersionRequest.setSourceBundle(s3SourceBundle);
createApplicationVersionRequest.setDescription("New version");
appVersionResultObject = awsBeanstalkclient.createApplicationVersion(createApplicationVersionRequest);
Error:
com.amazonaws.AmazonClientException: Unable to unmarshall response (ParseError at [row,col]:[6,1]
and one more error is
AWS service: AmazonElasticBeanstalk AWS Request ID: null AWS service unavailable.
Please suggest any solution for this.
How are you initializing the client (check logs output - enabling logger org.apache.http.wire to TRACE could help)?
If you want an idea, peek at this source:
https://github.com/jenkinsci/awseb-deployment-plugin/blob/master/src/main/java/br/com/ingenieux/jenkins/plugins/awsebdeployment/Deployer.java
It contains all you need to build and deploy into AWS EB :)