I am trying to use Glue schema registry service in AWS with scala (or java should be useful also) and I tested two ways to assume a role but it results in an error:
"Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()])"
I don't want to use environment variables so I tried STS to assume a role with the following code:
val assumeRoleRequest = AssumeRoleRequest.builder.roleSessionName(UUID.randomUUID.toString).roleArn("roleArn").build
val stsClient = StsClient.builder.region(Region.EU_CENTRAL_1).build
val stsAssumeRoleCredentialsProvider = StsAssumeRoleCredentialsProvider.builder.stsClient(stsClient).refreshRequest(assumeRoleRequest).build
val glueClient = GlueClient
.builder()
.region(Region.EU_CENTRAL_1)
.credentialsProvider(stsAssumeRoleCredentialsProvider)
Based on https://stackoverflow.com/a/62930761/17221117
The second way I used is using the following AWS code official documentation
But it fails also... I don't understand if this generate a token that I should use or just executing this code should work.
Anyone can help me with this?
Related
We need to generate an aws rds iam auth token using the c++ sdk.
Java sdk equivalent is here: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.Java.html#UsingWithRDS.IAMDBAuth.Connecting.Java.AuthToken
Is there a way to do the above using the C++ SDK?
The following code just results in printing the connection string (replace parameters with actual values):
Aws::RDS::RDSClient* p = new Aws::RDS::RDSClient();
std::cout << p->GenerateConnectAuthToken("host", "region", 3306, "user");
Not sure if these are related:
https://github.com/aws/aws-sdk-cpp/issues/868
https://github.com/aws/aws-sdk-cpp/issues/864
Any help will be appreciated.
I am facing this weird scenario. I generate my AWS AccessKeyId, SecretAccessKey and SessionToken by running assume-role-with-saml command. After copying these values to .aws\credentials file, I try to run command "aws s3 ls" and can see all the S3 buckets. Similarly I can run any AWS command to view objects and it works perfectly fine.
However, when I write .Net Core application to list objects, it doesn't work on my computer. Same .Net application works find on other colleagues' computers. We all have access to AWS through the same role. There are no users in IAM console.
Here is the sample code, but I am not sure there is nothing wrong with the code, because it works fine on other users' computers.
var _ssmClient = new AmazonSimpleSystemsManagementClient();
var r = _ssmClient.GetParameterAsync(new Amazon.SimpleSystemsManagement.Model.GetParameterRequest
{
Name = "/KEY1/KEY2",
WithDecryption = true
}).ConfigureAwait(false).GetAwaiter().GetResult();
Any idea why running commands through CLI works and API calls don't work? Don't they both look at the same %USERPROFILE%.aws\credentials file?
I found it. Posting here since it can be useful for someone having same issue.
Go to this folder: %USERPROFILE%\AppData\Local\AWSToolkit
Take a backup of all files and folders and delete all from above location.
This solution applies only if you can run commands like "aws s3 ls" and get the results successfully, but you get error "The provided token has expired" while running the same from .Net API libraries.
I want to write a python program to check which account I am in by using account alias (I have multiple Tennants et up on AWS). I think aws iam list-account-aliases return what exactly I am looking for but it is a command line results and I am not sure what is the best to capture as variable in a python program.
Also I was reading about the aws iam list-account-aliases and they have a output section mentioned AccountAliases -> (list). (https://docs.aws.amazon.com/cli/latest/reference/iam/list-account-aliases.html)
I wonder what this AccountAliases is? an option? a command? a variable? I was a little bit confused here.
Thank you!
Use Boto3 to get the account alias in python. Your link points to aws-cli.
Here is the link for equivalent Boto3 command for python:
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam.html#IAM.Client.list_account_aliases
Sample code:
import boto3
client = boto3.client('iam')
response = client.list_account_aliases()
Response:
{
'AccountAliases': [
'myawesomeaccount',
],
'IsTruncated': True,
}
Account alias on AWS is a readable alias created against AWS user's account id. You can find more info here :
https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html
AWS has provided a very well documented library for Python called Boto3 which can be used to obtain connect to your AWS account as client, resource or session (more information on these in SO answer here: Difference in boto3 between resource, client, and session?)
For your use case you can connect to your AWS as client with iam label:
import boto3
client = boto3.client(
'iam',
region_name="region_name",
aws_access_key_id="your_aws_access_key_id",
aws_secret_access_key="your_aws_secret_access_key")
account_aliases = client. list_account_aliases()
The response is a JSON object which can be traversed to get the desired information on aliases.
I need a way to allow a 3rd party app to upload a txt file (350KB and slowly growing) to an s3 bucket in AWS. I'm hoping for a solution involving an endpoint they can PUT to with some authorization key or the like in the header. The bucket can't be public to all.
I've read this: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
and this: http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html
but can't quite seem to find the solution I'm seeking.
I'd suggests using a combination of the AWS API gateway, a lambda function and finally S3.
You clients will call the API Gateway endpoint.
The endpoint will execute an AWS lambda function that will then write out the file to S3.
Only the lambda function will need rights to the bucket, so the bucket will remain non-public and protected.
If you already have an EC2 instance running, you could replace the lambda piece with custom code running on your EC2 instance, but using lambda will allow you to have a 'serverless' solution that scales automatically and has no min. monthly cost.
I ended up using the AWS SDK. It's available for Java, .NET, PHP, and Ruby, so there's very high probability the 3rd party app is using one of those. See here: http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadObjSingleOpNET.html
In that case, it's just a matter of them using the SDK to upload the file. I wrote a sample version in .NET running on my local machine. First, install the AWSSDK Nuget package. Then, here is the code (taken from AWS sample):
C#:
var bucketName = "my-bucket";
var keyName = "what-you-want-the-name-of-S3-object-to-be";
var filePath = "C:\\Users\\scott\\Desktop\\test_upload.txt";
var client = new AmazonS3Client(Amazon.RegionEndpoint.USWest2);
try
{
PutObjectRequest putRequest2 = new PutObjectRequest
{
BucketName = bucketName,
Key = keyName,
FilePath = filePath,
ContentType = "text/plain"
};
putRequest2.Metadata.Add("x-amz-meta-title", "someTitle");
PutObjectResponse response2 = client.PutObject(putRequest2);
}
catch (AmazonS3Exception amazonS3Exception)
{
if (amazonS3Exception.ErrorCode != null &&
(amazonS3Exception.ErrorCode.Equals("InvalidAccessKeyId")
||
amazonS3Exception.ErrorCode.Equals("InvalidSecurity")))
{
Console.WriteLine("Check the provided AWS Credentials.");
Console.WriteLine(
"For service sign up go to http://aws.amazon.com/s3");
}
else
{
Console.WriteLine(
"Error occurred. Message:'{0}' when writing an object"
, amazonS3Exception.Message);
}
}
Web.config:
<add key="AWSAccessKey" value="your-access-key"/>
<add key="AWSSecretKey" value="your-secret-key"/>
You get the accesskey and secret key by creating a new user in your AWS account. When you do so, they'll generate those for you and provide them for download. You can then attach the AmazonS3FullAccess policy to that user and the document will be uploaded to S3.
NOTE: this was a POC. In the actual 3rd party app using this, they won't want to hardcode the credentials in the web config for security purposes. See here: http://docs.aws.amazon.com/AWSSdkDocsNET/latest/V2/DeveloperGuide/net-dg-config-creds.html
I tried creating a datasource using boto for machine learning but ended up with an error.
Here's my code :
import boto
bucketname = 'mybucket'
filename = 'myfile.csv'
schema = 'myfile.csv.schema'
conn = boto.connect_s3()
datasource = 'my_datasource'
ml = boto.connect_machinelearning()
#create a data source
ds = ml.create_data_source_from_s3(
data_source_id = datasource,
data_spec ={
'DataLocationS3':'s3://'+bucketname+'/'+filename,
'DataSchemaLocationS3':'s3://'+bucketname+'/'+schema},
data_source_name=None,
compute_statistics = True)
print ml.get_data_source(datasource,verbose=None)
I get this error as a result of get_data_source call:
Could not access 's3://mybucket/myfile.csv'. Either there is no file at that location, or the file is empty, or you have not granted us read permission.
I have checked and I have FULL_CONTROL as my permissions. The bucket, file and schema all are present and are non-empty.
How do I solve this?
You may have FULL_CONTROL over that S3 resource but in order for this to work you have to grant the Machine Learning service the appropriate access to that S3 resource.
I know links to answers are frowned upon but in this case I think its best to link to the definitive documentation from the Machine Learning Service since the actual steps are complicated and could change in the future.