I am trying to figure out how to pass in static IAM AWS credentials when using the AWS Data API to interact with an Aurora Serverless db.
I am using the AWS Python Boto library and I read data from a table like this (which by default uses the credentials of the default IAM user that is defined in my ~/.aws/credentials file):
rds_client = boto3.client('rds-data')
rds_client.execute_statement(
secretArn=self.db_credentials_secrets_store_arn,
database=self.database_name,
resourceArn=self.db_cluster_arn,
sql='SELECT * FROM TestTable;',
parameters=[])
This works successfully.
But I want to be able to pass in an AWS Access Key and Secret Key as parameters to the execute_statement call, something like:
rds_client.execute_statement(
accessKey='XXX',
secretKey='YYY',
secretArn=self.db_credentials_secrets_store_arn,
database=self.database_name,
resourceArn=self.db_cluster_arn,
sql='SELECT * FROM TestTable;',
parameters=[])
But that does not work.
Any ideas on how I can achieve this?
Thanks!
In order to accomplish this, you will need to create a new function that takes the access key and the secret key, create a client for that user, then make the call.
def execute_statement_with_iam_user(accessKey, secretKey):
rds_client = boto3.client(
'rds',
aws_access_key_id=accessKey,
aws_secret_access_key=secretKey
)
rds_client.execute_statement(
secretArn=self.db_credentials_secrets_store_arn,
database=self.database_name,
resourceArn=self.db_cluster_arn,
sql='SELECT * FROM TestTable;',
parameters=[])
execute_statement_with_iam_user(accessKey, secretkey)
FYI, AWS does not recommend hard coding your credentials like this. What you should be doing is assuming a role with a temporary session. For this, you would need to look into the sts client and creating roles for assumption.
Related
I'm wondering if something is possible at all, or I'm trying to build something that is not possible from the start.
Let's say within Account A there is an RDS DB Password, (can be any AWS resource ID or value) that I have stored in Secrets Manager or Parameter Store.
Now I want to use that value in AWS CDK in Account B, is this possible?
It is possible to retrieve the value based on ARN, see: https://bobbyhadz.com/blog/get-secrets-manager-values-aws-cdk#get-secrets-manager-value-by-arn---alternative but would this work cross-account?
You can attach a policy to your secret granting access to other AWS account. Check https://aws.amazon.com/premiumsupport/knowledge-center/secrets-manager-share-between-accounts/
Previously when I did not set MFA to login to AWS console I've connected to dynamodb by
dynamo = boto3.resource('dynamodb',
region_name='ap-northeast-2',
endpoint_url='http://dynamodb.ap-northeast-2.amazonaws.com')
table = dynamo.Table('tablename')
and querying to that table was perfectly fine.
response = table.query(
KeyConditionExpression =Key("user_id").eq(123123)
)
After I've set MFA for additional security to login to AWS console and now when I execute above code I get:
ClientError: An error occurred (UnrecognizedClientException) when calling the Query operation: The security token included in the request is invalid.
I use tunnel for RDB, is there something like that I could use for connecting to dynamodb or is there a permission I need in order to access dynamodb?
When you enable MFA, SDK does not automatically know how to work with it. Your regular IAM user's API and SECRET keys are no longer enough. Instead you need to use temporary credentials created only for your MFA session.
To make MFA work with boto3 you have to explicitly call get_session_token:
MFA-enabled IAM users would need to call GetSessionToken and submit an MFA code that is associated with their MFA device. Using the temporary security credentials that are returned from the call, IAM users can then make programmatic calls to API operations that require MFA authentication.
Using get_session_token you can call sts service which is going to provide you with temporary credentials based on your MFA details:
sts = boto3.client('sts')
mfa_response = sts.get_session_token(
DurationSeconds=123,
SerialNumber='string',
TokenCode='string'
)
The call will return the credentials in mfa_response which you can use to create a new boto3 session. For example:
mfa_session = boto3.session.Session(
aws_access_key_id=mfa_session['Credentials']['AccessKeyId'],
aws_secret_access_key=mfa_session['Credentials']['SecretAccessKey'],
aws_session_token=mfa_session['Credentials']['SessionToken'])
dynamo = mfa_session.resource('dynamodb', ...)
# and the rest of the code
I am using AWS Secret Manager Service to retrieve some confidential information like SMTP details or connection strings. However, to get secret value from AWS Secret Manager Service it seems like we need to pass the Access key and secret key apart from which secret we want to retrieve. So I am maintaining those values in config file.
public AwsSecretManagerService(IOptions<AwsAppSettings> settings)
{
awsAppSettings = settings.Value;
amazonSecretsManagerClient = new AmazonSecretsManagerClient
(awsAppSettings.Accesskey, awsAppSettings.SecretKey, RegionEndpoint.GetBySystemName(awsAppSettings.Region));
}
public async Task<SecretValueResponse> GetSecretValueAsync(SecretValueRequest secretValueRequest)
{
return _mapper.Map<SecretValueResponse>(await amazonSecretsManagerClient.GetSecretValueAsync(_mapper.Map<GetSecretValueRequest>(secretValueRequest)));
}
So I am thinking I am kind of defeating the whole purpose of using secret manager by maintaining the AWS credentials in app settings file. I am wondering what is the right way to do this
It is not a good practice to pass or add AWS credentials of an IAM User (access key and secret access key) in the code.
Instead, don't pass it and update your code as follows:
amazonSecretsManagerClient = new AmazonSecretsManagerClient
(RegionEndpoint.GetBySystemName(awsAppSettings.Region));
Question: Then how would it access the AWS services?
Answer: If you are going to execute your code on your local system, install and configure AWS CLI instead of passing AWS credentials via CLI or Terminal, it will use those AWS configured credentials to access the AWS services.
Reference for AWS CLI Installation: Installing the AWS CLI
Reference for AWS CLI Configuration: Configuring the AWS CLI
If you are going to execute your code on an AWS service (e.g., EC2 instance), attach an IAM role with that AWS resource (e.g., EC2 instance) having sufficient permissions, it will use that IAM role to access the AWS services.
Recently it was create the possibility to access RDS instances with IAM users and roles. I am confused about how to configure a python connection, since I would not use the database authentication data with psycopg2.
Now I am using like this:
conn = psycopg2.connect("dbname='%s' user='%s' host='%s' password='%s'" % (db_name, db_user, db_host, db_pass))
I have not idea how to use IAM credentials to connect my lambda function with IAM auth.
Please help.
First, you need to create an IAM policy and a DB user as described here:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html
Then you need to create IAM role for your Lambda function and attach the IAM policy created above to it. Your Lambda function will need to be executed with this role to be able to create a temporary DB password for the DB user.
Finally, you can generate a temporary password for your DB user (created above) within your Lambda using a code snippet like this:
from urllib.parse import quote_plus
import boto3
def get_password(rds_hostname, db_user, aws_region=None, url_encoded=True):
if (not aws_region):
aws_region = boto3.session.Session().region_name
if (not aws_region):
raise Exception("Error: no aws_region given and the default region is not set!")
rds_port = 5432
if (":" in rds_hostname):
split_hostname = rds_hostname.split(":")
rds_hostname = split_hostname[0]
rds_port = int(split_hostname[1])
rds_client = boto3.client("rds")
password = rds_client.generate_db_auth_token( Region=aws_region,
DBHostname=rds_hostname,
Port=rds_port,
DBUsername=db_user)
if url_encoded:
return quote_plus( password )
else:
return password
Do not assign the the password to a variable. Get a new password on every run, since the password has limited time validity and your Lambda container might not be recycled before it expires...
Finally, create the DB connection string for whatever python package you use (I would suggest some pure Python implementation, such as pg8000) from your RDS hostname, port, username and the temporary password obtained with the function above (<user>:<password>#<hostname>:<port>/<db_name>).
Connecting to the RDS might be a bit tricky. If you don't know how to set up VPC's properly I would suggest you run your Lambda outside of VPC and connect to the RDS over a public IP.
Additionally, you will probably need to enforce SSL connection and possibly include the RDS CA file in your Lambda deployment package. The exact way how to do this depends on what you use to connect (I could only describe how to do this with pymysql and sqlalchemy).
Each of these steps could be described in a tutorial of it's own, but knowing about them should be enough to get you started.
Good luck!
I can't use boto3 to connect to S3 with a role arn provided 100% programmatically.
session = boto3.Session(role_arn="arn:aws:iam::****:role/*****",
RoleSessionName="****")
s3_client = boto3.client('s3',
aws_access_key_id="****",
aws_secret_access_key="****")
for b in s3_client.list_buckets()["Buckets"]:
print (b["Name"])
I can't provide arn info to Session and also client and there is no assume_role() on a client based on s3.
I found a way with a sts temporary token but I don't like that.
sess = boto3.Session(aws_access_key_id="*****",
aws_secret_access_key="*****")
sts_connection = sess.client('sts')
assume_role_object = sts_connection.assume_role(RoleArn="arn:aws:iam::***:role/******",
RoleSessionName="**",
DurationSeconds=3600)
session = boto3.Session(
aws_access_key_id=assume_role_object['Credentials']['AccessKeyId'],
aws_secret_access_key=assume_role_object['Credentials']['SecretAccessKey'],
aws_session_token=assume_role_object['Credentials']['SessionToken'])
s3_client = session.client('s3')
for b in s3_client.list_buckets()["Buckets"]:
print (b["Name"])
Do you have any idea ?
You need to understand how temporary credentials are created.
First you need to create a client using your current access keys. These credentials are then used to verify that you have the permissions to call assume_role and have the rights to issue credentials from the IAM role.
If someone could do it your way, there would be a HUGE security hole with assume_role. Your rights must be validated first, then you can issue temporary credentials.
Firstly, never put an Access Key and Secret Key in your code. Always store credentials in a ~/.aws/credentials file (eg via aws configure). This avoids embarrassing situations where your credentials are accidentally released to the world. Also, if you are running on an Amazon EC2 instance, then simply assign an IAM Role to the instance and it will automatically obtain credentials.
An easy way to assume a role in boto3 is to store the role details in the credentials file with a separate profile. You can then reference the profile when creating a client and boto3 will automatically call assume-role on your behalf.
See: boto3: Assume Role Provider