Getting below error while running the Spark scala code job in Databricks.
{secretsmanager, us-east-1} was not found in region metadata, trying to construct an endpoint using the standard pattern for this region: 'secretsmanager.us-east-1.amazonaws.com'
Secret Value not pulled from secret manager for tbdp-audit-api-key
Expecting the job to run successfully
Related
I have EC2 instance where I'm trying to configure PySpark to read from S3.
I set a full access IAM role to EC2 instance and used the following packages in spark:
com.amazonaws:aws-java-sdk-bundle:1.11.563,org.apache.hadoop:hadoop-aws:3.3.1
However, I'm getting some new error, and I'm not sure what does it mean:
: org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
s3a://bucket_name.stuff/mycsv.csv: com.amazonaws.SdkClientException: Unable to execute HTTP
request: Certificate for <bucket_name.stuff.s3.amazonaws.com> doesn't
match any of the subject alternative names: [*.s3.amazonaws.com,
s3.amazonaws.com]
So the issue turned out to be a version mismatch between pyspark, hadoop-aws and java-sdk (I was getting all kind of different errors until I found a proper version setup)
The combination that worked for me was:
pyspark 3.0.0
org.apache.hadoop:hadoop-aws:2.7.2
com.amazonaws:aws-java-sdk-pom:1.11.34
I seem to have problems running a command to verify that my credentials are configured correctly and that I can connect to AWS as stated here:https://docs.aws.amazon.com/cli/latest/userguide/tutorial-ec2-ubuntu.html:
When running:
$ aws ec2 describe-regions --output table
I get the following output:
An error occurred (AuthFailure) when calling the DescribeRegions
operation: AWS was not able to validate the provided access
credentials
What am I missing?
After installing the AWS CLI (on a fedora machine), I ran
$ aws configure
for AWS Access Key ID and AWS Secret Access Key:
I went to AWS website and created an IAM user.
For that user, I have gone to the security credentials tab and
I have created a new Access key, which is key value pair of Access key ID,Secret access key.
I have used those values for AWS Access Key ID and AWS Secret Access Key but I keep getting the above error message.
What am I missing? Thanks in advance.
You need to pass the profile parameter. This link from AWS has more details
In my Jenkinsfile, I am trying to push the image that I have built using the docker plugin like follows:
docker.withRegistry('https://<my-id>.dkr.ecr.us-east-1.amazonaws.com/', 'ecr:us-east-1:awscreds') {
docker.image('image').push('latest')
}
The pipeline fails every time with the message ERROR: Could not find credentials matching ecr:us-east-1:awscreds but I do have my AWS key ID and secret key in my Jenkins credentials with the ID "awscreds".
What could be a potential fix for this?
Alternatively, can I provide my credentials directly instead of mentioning the credential ID in the call?
I had the same error message. Make sure the Amazon ECR plugin is installed and up to date and that you reboot jenkins after the installation.
I have a quick question about usage of AWS SNS.
I have deployed an EC2 (t2.micro, Linux) instance in us-west-1 (N.California). I have written a python script using boto3 to send a simple text message to my phone. Later I discovered, there is no SNS service for instances deployed out of us-east-1 (N.Virginia). Till this point it made sense, because I see this below error when i execute my python script, as the region is defined as "us-west-1" in aws configure (AWS cli) and also in my python script.
botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameter) when calling the Publish operation: Invalid parameter: PhoneNumber Reason:
But to test, when I changed the "region" in aws conifgure and in my python script to "us-east-1", my script pushed a text message to my phone. Isn't it weird? Can anyone please explain why this is working just by changing region in AWS cli and in my python script, though my instance is still in us-west-1 and I dont see "Publish text message" option on SNS dashboard on N.california region?
Is redefining the aws cli with us-east-1 similar to deploying a new instance altogether in us-east-1? I dont think so. Correct me if I am wrong. Or is it like having an instance in us-west-1, but just using SNS service from us-east-1? Please shed some light.
Here is my python script, if anyone need to look at it (Its a simple snippet).
import boto3
def send_message():
# Create an SNS client
client = boto3.client("sns", aws_access_key_id="XXXX", aws_secret_access_key="XXXX", region_name="us-east-1")
# Send your sms message.
client.publish(PhoneNumber="XXXX",Message="Hello World!")
if __name__ == '__main__':
send_message()
Is redefining the aws cli with us-east-1 similar to deploying a new
instance altogether in us-east-1?
No, it isn't like that at all.
Or is it like having an instance in us-west-1, but just using SNS
service from us-east-1?
Yes, that's all you are doing. You can connect to any AWS regions' API from anywhere on the Internet. It doesn't matter that it is running on an EC2 instance in a specific region, it only matters what region you tell the SDK/CLI to use.
You could run the same code on your local computer. Obviously your local computer is not running on AWS so you would have to tell the code which AWS region to send the API calls to. What you are doing is the same thing.
Code running on an EC2 server is not limited into using the AWS API in the same region that the EC2 server is in.
Did you try creating a topic before publishing to it? You should try create a topic and then publish to that topic.
I am trying to connect to Dynamodb using federated login. I created the dynamodb with sample table.When I try to access the table using aws cli, I was able to access the tables.
aws dynamodb list-tables --profile default
When I tried to call using a java class I am getting the following exception.
Exception in thread "main" com.amazonaws.SdkClientException:
Unable to execute HTTP request: Connection reset
Any help on this is appreciated.