I'm working on the AWS glue job, can someone please help me to give me a script in AWS Glue Job Spark that would retrieve my secrets from secret manager. Help is appreciated.
It is fairly simple (if you have all of the required IAM permissions) using boto3 and get_secret_value() function.
sm_client = boto3.client('secretsmanager')
response = sm_client.get_secret_value(
SecretId=<your_secret_id>
)
If the value is a string, you can extract it like this:
value = secret_value_response['SecretString']
If it is a binary value:
secret_value_response['SecretBinary']
Additionally, if the secret has multiple versions of the secret, you have to use VersionId and/or VersionStage as explained in the linked documentation.
Related
I want to get the bucket policy for the various buckets. I tried the following code snippet(picked from the boto3 documentation):
conn = boto3.resource('s3')
bucket_policy=conn.BucketPolicy('demo-bucket-py')
print(bucket_policy)
But here's the output I get :
s3.BucketPolicy(bucket_name='demo-bucket-py')
What shall I rectify here ? Or is there some another way to get the access policy for s3 ?
Try print(bucket_policy.policy). More information on that here.
this worked for me
import boto3
# Create an S3 client
s3 = boto3.client('s3')
# Call to S3 to retrieve the policy for the given bucket
result = s3.get_bucket_policy(Bucket='my-bucket')
print(result)
to perform this you need to configure or mention your keys like this s3=boto3.client("s3",aws_access_key_id=access_key_id,aws_secret_access_key=secret_key). BUT there is much better way to do this is by using aws configure command and enter your credentials. for setting up docs. Once you set up you wont need to enter your keys again in your code, boto3 or aws cli will automatically fetch it behind the scenes .https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html.
you can even set different profiles to work with different accounts
I am trying to fetch pre existing secrets from the aws-secretsmanager module on CDK, and from the documentation here, the suggestion is
If you need to use a pre-existing secret, the recommended way is to
manually provision the secret in AWS SecretsManager and use the
Secret.fromSecretArn or Secret.fromSecretAttributes method to make it
available in your CDK Application
However, both the methods demand the use of the arn to fetch the secrets. I am not sure if it is a good idea to hardcode arns and check them into the git repo. Instead is there a way to just fetch the secrets by just using the name, since we already have the account details available in the profile for cdk.
At least until this current version (1.38.0), it’s not possible. An alternative is to save the secret arn in the SSM parameter store and use the ssm key in the code.
Putting full ARNs in CFN should not be a concern. Since you are creating these secrets ahead of time, their name, account, and region will be know. If you wish, however, you could still use the CFN psuedo parameters for partition, region, and account (AWS::Partition, AWS::Region, AWS::AccountId or the CDK equivelent).
I am trying to figure out how to pass in static IAM AWS credentials when using the AWS Data API to interact with an Aurora Serverless db.
I am using the AWS Python Boto library and I read data from a table like this (which by default uses the credentials of the default IAM user that is defined in my ~/.aws/credentials file):
rds_client = boto3.client('rds-data')
rds_client.execute_statement(
secretArn=self.db_credentials_secrets_store_arn,
database=self.database_name,
resourceArn=self.db_cluster_arn,
sql='SELECT * FROM TestTable;',
parameters=[])
This works successfully.
But I want to be able to pass in an AWS Access Key and Secret Key as parameters to the execute_statement call, something like:
rds_client.execute_statement(
accessKey='XXX',
secretKey='YYY',
secretArn=self.db_credentials_secrets_store_arn,
database=self.database_name,
resourceArn=self.db_cluster_arn,
sql='SELECT * FROM TestTable;',
parameters=[])
But that does not work.
Any ideas on how I can achieve this?
Thanks!
In order to accomplish this, you will need to create a new function that takes the access key and the secret key, create a client for that user, then make the call.
def execute_statement_with_iam_user(accessKey, secretKey):
rds_client = boto3.client(
'rds',
aws_access_key_id=accessKey,
aws_secret_access_key=secretKey
)
rds_client.execute_statement(
secretArn=self.db_credentials_secrets_store_arn,
database=self.database_name,
resourceArn=self.db_cluster_arn,
sql='SELECT * FROM TestTable;',
parameters=[])
execute_statement_with_iam_user(accessKey, secretkey)
FYI, AWS does not recommend hard coding your credentials like this. What you should be doing is assuming a role with a temporary session. For this, you would need to look into the sts client and creating roles for assumption.
I'm running Spark 2.4 on an EC2 instance. I am assuming an IAM role and setting the key/secret key/token in the sparkSession.sparkContext.hadoopConfiguration, along with the credentials provider as "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider".
When I try to read a dataset from s3 (using s3a, which is also set in the hadoop config), I get an error that says
com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 7376FE009AD36330, AWS Error Code: null, AWS Error Message: Forbidden
read command:
val myData = sparkSession.read.parquet("s3a://myBucket/myKey")
I've repeatedly checked the S3 path and it's correct. My assumed IAM role has the right privileges on the S3 bucket. The only thing I can figure at this point is that spark has some sort of hidden credential chain ordering and even though I have set the credentials in the hadoop config, it is still grabbing credentials from somewhere else (my instance profile???). But I have no way to diagnose that.
Any help is appreciated. Happy to provide any more details.
spark-submit will pick up your env vars and set them as the fs.s3a access +secret + session key, overwriting any you've already set.
If you only want to use the IAM credentials, just set fs.s3a.aws.credentials.provider to com.amazonaws.auth.InstanceProfileCredentialsProvider; it'll be the only one used
Further Reading: Troubleshooting S3A
AWS has secret manager which stores secrets. It has the API to get individual secret. I want to fetch all the secrets related to an account at once. Any way we can achieve this?
You can use the method ListSecrets to list all secret metadata excluding SecretString or SecretBinary.
I tried to list secrets names in my secrets manager using boto3 python: using list.secrets()
secrets = secret_client.list_secrets()
secrets_manager = (secrets['SecretList'])
for secret in secrets_manager:
print ("{0}".format(secret['Name']))
The complete list was around 20, but the output was only around 5 secrets.
Updated the code to below, it worked:
secrets = secret_client.list_secrets()
secrets_manager = (secrets['SecretList'])
while "NextToken" in secrets:
secrets = secret_client.list_secrets(NextToken=secrets["NextToken"])
secrets_manager.extend(secrets['SecretList'])
for secret in secrets_manager:
print ("{0}".format(secret['Name']))
So basically, AWS secrets manager list.secrets() call paginates your output, so it is better to use 'NextToken' as mentioned in https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.list_secrets
'The encrypted fields SecretString and SecretBinary are not included in the output' in ListSecrets.
If you're trying to fetch all secret values then options might include:
1) Scripting list-secrets and get-secret-value to fetch all secret values. This example will be slow since it's using serial requests.
#!/usr/bin/env python3
import json
import subprocess
secrets = json.loads(subprocess.getoutput("aws secretsmanager list-secrets"))
for secret in secrets.values():
for s in secret:
name = s.get('Name')
data = json.loads(subprocess.getoutput("aws secretsmanager get-secret-value --secret-id {}".format(name)))
value = data.get('SecretString')
print("{}: {}".format(name, value))
2) Use a 3rd party tools such as Summon with its AWS Provider which accepts secrets.yml file and makes async calls to inject secrets into the environment of whatever command you're calling.