Typically emrfs consistency is enabled via emrfs-site.xml
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emrfs-configure-consistent-view.html
Does anyone know if these setting can be accessed via the SDK?
To enable EMRFS with the Java SDK, an "emrfs-site" configuration needs to be added to the RunJobFlowRequest and the fs.s3.consistent property must be set to true. Like this:
Map<String, String> emrfsProperties = new HashMap<>();
emrfsProperties.put("fs.s3.consistent", "true");
RunJobFlowRequest request = new RunJobFlowRequest()
....
.withServiceRole(SERVICE_ROLE)
.withJobFlowRole(JOB_FLOW_ROLE)
.withConfigurations(
new Configuration().withClassification("yarn-site").withProperties(yarnProperties),
new Configuration().withClassification("emrfs-site").withProperties(emrfsProperties)
)
.withInstances(new JobFlowInstancesConfig()
.withEc2KeyName(EC2_KEYPAIR)
....
A full list of EMRFS configuration parameters can be found here
Yes, you have a full documentation here:
http://docs.aws.amazon.com/ElasticMapReduce/latest/API/Welcome.html
You need to authorize connection to your AWS first, than you can configure you application to your needs, using API.
Look also here:
http://docs.aws.amazon.com/ElasticMapReduce/latest/API/CommonParameters.html
http://docs.aws.amazon.com/ElasticMapReduce/latest/API/EmrConfigurations.html
Related
In the AWS SDK https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/kms/package-summary.html the KMS has two different types of clients. A regular client and a client builder. What is the purpose of them both? When do you choose one over the other?
I'm trying to implement envelope encryption using KMS. I want to be able to hit the KMS endpoint and encrypt the payload. Which client library should I be using?
There is only one client type: KmsClient.
It can be created in 2 ways:
Using the KmsClientBuilder returned by KmsClient.builder() to modify properties and ultimately do .Build for your customised version of the client - this KmsClientBuilder is an instance of DefaultKmsClientBuilder, which is currently the only class that implements the client builder interface.
Using the KmsClient returned by KmsClient.create(), which is exactly the equivalent (and a shortcut) to new DefaultKmsClientBuilder().build() - this method returns a client set up with the region & credentials already loaded from the respective default provider chain, for applications that don't require further customisation.
This is how the above looks like in code:
final KmsClient defaultKmsClient = KmsClient.create();
final KmsClient alsoDefaultKmsClientButLonger = KmsClient.builder().build();
final KmsClient customisedKmsClient = KmsClient.builder()
.region(...)
.credentialsProvider(...)
.httpClient(...)
.endpointOverride(...)
...
In conclusion, use KmsClient.create() if you do not require a particular configuration, as the default region and creds should be sufficient in most cases.
If not, then customise it via an instance of the builder (which can only be accessed via the KmsClient.builder() method since KmsClientBuilder is an interface).
They are not 'different'.
The builder ultimately is what creates the client.
I am using the TapKey SDK in combination with a FlinkeyBox.
So far (SDK 2.12.7), I used to be able to set the BleServiceUuid in the TapkeyEnvironmentConfigBuilder.
Now I've upgraded to the newest SDK version and the method TapkeyEnvironmentConfigBuilder.setBleServiceUuid is simply gone. I can't find it in any migration guide either.
Can someone help?
Indeed this information is missing. We will cover this in the migration guide.
To change the BLE Service UUID you have now to use the TapkeyBleAdvertisingFormatBuilder.
TapkeyBleAdvertisingFormat advertisingFormat = new TapkeyBleAdvertisingFormatBuilder()
.addV1Format("[serviceUUID]")
//.addV2Format([domainID])
.build();
TapkeyServiceFactory tapkeyServiceFactory = new TapkeyServiceFactoryBuilder(this)
.setBluetoothAdvertisingFormat(advertisingFormat)
...
.build();
New hardware generations will use a new Bluetooth Advertising, which has then to be configured with the V2 format. But for now it will be sufficient just to configure the V1 format. For more informations about how to configure the TapkeyBleAdvertisingFormat please contact your service provider.
I am creating a method that needs a S3 client as a parameter. I do not know what type should I declare it to be.
this is the doc for S3Client https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3Client.html
Ignore since answered (this is the doc for AmazonS3Client
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html My question is which type is recommended and what are difference between them? Thank you! )
Update:
I find another S3 Client here: AmazonS3 interface.
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3.html
However, setObjectTagging is supported in type AmazonS3 not but in type S3Client .
Does AmazonS3 provide more functionality than S3Client?
What if I need some function in AmazonS3 not in S3Client, or some in S3Client not in AmazonS3?
The AWS SDK for Java has two versions: V1 and V2. AmazonS3Client is the older V1 version while S3Client is the newer V2 version.
Amazon recommends using V2:
The AWS SDK for Java 2.x is a major rewrite of the version 1.x code base. It’s built on top of Java 8+ and adds several frequently requested features. These include support for non-blocking I/O and the ability to plug in a different HTTP implementation at run time.
You can find Amazon S3 V2 code examples in the Java Developer V2 DEV Guide here:
Developer guide - AWS SDK for Java 2.x
(At this point, the Amazon S3 Service guide does not have V2 examples in it.)
In addition, you can find all Amazon S3 V2 code examples in AWS Github here:
https://github.com/awsdocs/aws-doc-sdk-examples/tree/master/javav2/example_code/s3
If you are not familiar developing apps by using the AWS SDK for Java V2, it's recommended that you start here:
Get started with the AWS SDK for Java 2.x
(This getting started topic happens to use the Amazon S3 Java V2 API to help get you up and running with using the AWS SDK for Java V2)
Update:
You stated: However, setObjectTagging is supported in type AmazonS3 not but in type S3Client .
The way to tag an Object in an Amazon S3 bucket by using Java V2 API is to use this code:
// First need to get existing tag set; otherwise the existing tags are overwritten.
GetObjectTaggingRequest getObjectTaggingRequest = GetObjectTaggingRequest.builder()
.bucket(bucketName)
.key(key)
.build();
GetObjectTaggingResponse response = s3.getObjectTagging(getObjectTaggingRequest);
// Get the existing immutable list - cannot modify this list.
List<Tag> existingList = response.tagSet();
ArrayList<Tag> newTagList = new ArrayList(new ArrayList<>(existingList));
// Create a new tag.
Tag myTag = Tag.builder()
.key(label)
.value(LabelValue)
.build();
// push new tag to list.
newTagList.add(myTag);
Tagging tagging = Tagging.builder()
.tagSet(newTagList)
.build();
PutObjectTaggingRequest taggingRequest = PutObjectTaggingRequest.builder()
.key(key)
.bucket(bucketName)
.tagging(tagging)
.build();
s3.putObjectTagging(taggingRequest);
I am using JDBC to connect to Athena for a specific Workgroup. But it is by default redirecting to the primary workgroup
Below is the code snippet
Properties info = new Properties();
info.put("user", "access-key");
info.put("password", "secrect-access-key");
info.put("WorkGroup","test");
info.put("schema", "testschema");
info.put("s3_staging_dir", "s3://bucket/athena/temp");
info.put("aws_credentials_provider_class","com.amazonaws.auth.DefaultAWSCredentialsProviderChain");
Class.forName("com.simba.athena.jdbc.Driver");
Connection connection = DriverManager.getConnection("jdbc:awsathena://athena.<region>.amazonaws.com:443/", info);
As you can see I am using "Workgroup" as the key for the properties. I also tried "workgroup", "work-group", "WorkGroup". It is not able to redirect to the specified Workgroup. Always going to the default one i.e primary workgroup.
Kindly help. Thanks
If you look at the release notes of Athena JDBC, the workgroup support is from v2.0.7.
If you jar is below this version, it will not work. Try to upgrade the library to 2.0.7 or above
You need to Override Client-Side Settings in workgroup.Enable below setting and rerun the query via JDBC.
Check this doc for more information.
I am trying to use AWS Powershell with Eucalyptus.
I can do this with AWS CLI with parameter --endpoint-url.
Is it possible to set endpoint url in AWS powershell?
Can I create custom region with my own endpoint URL in AWS Powershell?
--UPDATE--
The newer versions of the AWS Tools for Windows PowerShell (I'm running 3.1.66.0 according to Get-AWSPowerShellVersion), has an optional -EndpointUrl parameter for the relevant commands.
Example:
Get-EC2Instance -EndpointUrl https://somehostnamehere
Additionally, the aforementioned bug has been fixed.
Good stuff!
--ORIGINAL ANSWER--
TL;TR
Download the default endpoint config file from here: https://github.com/aws/aws-sdk-net/blob/master/sdk/src/Core/endpoints.json
Customize it. Example:
{
"version": 2,
"endpoints": {
"*/*": {
"endpoint": "your_endpoint_here"
}
}
}
After importing the AWSPowerShell module, tell the SDK to use your customized endpoint config. Example:
[Amazon.AWSConfigs]::EndpointDefinition = "path to your customized Amazon.endpoints.json here"
Note: there is a bug in the underlying SDK that causes endpoints that have a path component from being signed correctly. The bug affects this solution and the solution #HyperAnthony proposed.
Additional Info
Reading through the .NET SDK docs, I stumbled across a section that revealed that one can global set the region rules given a file: http://docs.aws.amazon.com/AWSSdkDocsNET/latest/V2/DeveloperGuide/net-dg-config-other.html#config-setting-awsendpointdefinition
Unfortunately, I couldn't find anywhere where the format of such a file is documented.
I then splunked through the AWSSDK.Core.dll code and found where the SDK loads the file (see LoadEndpointDefinitions() method at https://github.com/aws/aws-sdk-net/blob/master/sdk/src/Core/RegionEndpoint.cs).
Reading through the code, if a file isn't explicitly specified on AWSConfigs.EndpointDefinition, it ultimately loads the file from an embedded resource (i.e. https://github.com/aws/aws-sdk-net/blob/master/sdk/src/Core/endpoints.json)
I don't believe that it is. This list of common parameters (that can be used with all AWS PowerShell cmdlets) does not include a Service URL, it seems instead to opt for a simple string Region to set the Service URL based on a set of known regions.
This AWS .NET Development forum post suggests that you can set the Service URL on a .NET SDK config object, if you're interested in a possible alternative in PowerShell. Here's an example usage from that thread:
$config=New-Object Amazon.EC2.AmazonEC2Config
$config.ServiceURL = "https://ec2.us-west-1.amazonaws.com"
$client=[Amazon.AWSClientFactory]::CreateAmazonEC2Client($accessKeyID,$secretKeyID,$config)
It looks like you can use it with most config objects when setting up a client. Here's some examples that have the ServiceURL property. I would imagine that this is on most all AWS config objects:
AmazonEC2Config
AmazonS3Config
AmazonRDSConfig
Older versions of the documentation (for v1) noted that this property will be ignored if the RegionEndpoint is set. I'm not sure if this is still the case with v2.