After setting up AWS Kinesis Firehose along with OpenSearch I've been unable to run Test with Demo data successfully.
Error log suggests Auth permission related issue but the user is an Admin with full permmission access.
{ "deliveryStreamARN": "arn:aws:firehose:us-east-1:...:deliverystream/PUT-...", "destination": "https://....us-east-1.aoss.amazonaws.com", "deliveryStreamVersionId": 1, "message": "{"type":"security_exception","reason":"Authorization failure for the following indices: [analytics-index]"}", "errorCode": "403" }
I am trying to sync two S3 buckets in different accounts. I have successfully configured the locations and created a task. However, when I run the task I get a Unable to connect to S3 endpoint error. Can anyone help?
This could have been related to the datasync's IAM role's policy (datasync IAM role) not having permission to the target S3 bucket
verify your policy and trust relationship using the below documentation
https://docs.aws.amazon.com/datasync/latest/userguide/using-identity-based-policies.html
Also turn on cloudwatch logs (like shown in the image) and view detailed log in cloudwatch. If it is permission related, add the missing policy in the Datasync role.
While creating AWS EMR cluster, always i get the issue- Service role EMR_DefaultRole has insufficient EC2 permissions
And the cluster terminates automatically, have even done steps as per aws documentation of recreating emr specific roles, but no progress please guide how to resolve the issue- Service role EMR_DefaultRole has insufficient EC2 permissions
EMR needs two roles to start the cluster 1) EC2 Instance profile role 2)EMR Service role. The service role should have enough permissions to provision new resources to start the cluster, EC2 instances, their network etc. There could be many reasons for this common error:
Verify the resources and their actions. Refer https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-iam-role.html.
Check if you are passing the tag that signifies if cluster needs to use emr managed policy.
{
"Key": "for-use-with-amazon-emr-managed-policies",
"Value": "true"
}
At last try to find out the exact reason from cloud trail. Go to aws>cloud trail. From the event history configuration enable the error code so that you can see the exact error. If you find the error code something like 'You are not authorized to perform this operation. Encoded authorization failure message'. Then open the event history details, pick up the encrypted error message and decrypt using aws cli
aws sts decode-authorization-message message. This will show you the complete role details, event, resources, action. Compare it with AWS IAM permissions and you can find out the missing permission or parameter that you need to pass while creating the job flow.
Following this guide and creating a Kinesis Firehose Stream.
I have followed the guide and when I get to creating a subscription filter (step 12), I encounter this error when trying to send to S3:
An error occurred (InvalidParameterException) when calling the PutSubscriptionFilter operation: Could not deliver test message to specified Firehose stream. Check if the given Firehose stream is in ACTIVE state.
I can confirm that the stream is active and I can send test data via the console and it arrives in S3 as expected.
This is the command I am running (changed my account id):
aws logs put-subscription-filter --log-group-name "myLogGroup" --filter-name "Destination" --filter-pattern "{$.userIdentity.type = Root}" --destination-arn "arn:aws:firehose:ap-southeast-1:1234567890:deliverystream/my-delivery-stream" --role-arn "arn:aws:iam::1234567890:role/CWLtoKinesisFirehoseRole"
I have checked the trusted entities and the role has priviliges to logs and firehose. Any ideas?
I also struggle with this for a long time, for me it was those 2 gothca's:
step 4 in the guide:
make sure to change to bucket name to you bucket:
step 8 !!!:
make sure to put your account ID it is not highlighted:
I am sure you already know how to configure logs subscription filter so not adding steps in my answer
Go to firehose and check logs is your firehose has access to execute lambda ,if not please add required role.
Now start dummy data stream using firehose test and see is your data is moving till lambda or S3.
check cloud trail and cloud watch logs and see if found any error .
Open your IAM role and check all required role added to your role , now click trust relationship add- "logs group ","IAM" and component name in my case its "Ec2"
Hope this will helpful to resolve your issue.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#FirehoseExample
I have a Glue ETL job, created by CloudFormation. This job extracts data from RDS Aurora and write to S3.
When I run this job, I get the error below.
The job has an IAM service role.
This service role allows
Glue and RDS service,
assume arn:aws:iam::aws:policy/AmazonS3FullAccess and arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole, and
has full range of rds:* , kms:* , and s3:* actions allow to the corresponding RDS, KMS, and S3 resources.
I have the same error whether the S3 bucket is encrypted with either AES256 or aws:kms.
I get the same error whether the job has a Security Configuration or not.
I have a job doing the exactly same thing that I created manually and can run successfully without a Security Configuration.
What am I missing? Here's the full error log
"/mnt/yarn/usercache/root/appcache/application_1...5_0002/container_15...45_0002_01_000001/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o145.pyWriteDynamicFrame.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 2.0 failed 4 times, most recent failure: Lost task 3.3 in stage 2.0 (TID 30, ip-10-....us-west-2.compute.internal, executor 1): com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: F...49), S3 Extended Request ID: eo...wXZw=
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588
Unfortunately the error doesn't tell us much except that it's failing during the write of your DynamicFrame.
There is only a handful of possible reasons for the 403, you can check if you have met them all:
Bucket Policy rules on the destination bucket.
The IAM Role needs permissions (although you mention having S3*)
If this is cross-account, then there is more to check with regards things like to allow-policies on the bucket and user. (In general a Trust for the Canonical Account ID is simplest)
I don't know how complicated your policy documents might be for the Role and Bucket, but remember that an explicit Deny statement takes precedence over an allow.
If the issue is KMS related, I would check to ensure your Subnet you select for the Glue Connection has a route to reach the KMS endpoints (You can add an Endpoint for KMS in VPC)
Make sure issue is not with the Temporary Directory that is also configured for your job or perhaps write-operations that are not your final.
Check that your account is the "object owner" of the location you are writing to (normally an issue when read/writing data between accounts)
If none of the above works, you can shed some more light with regards to your setup. Perhaps the code for write-operation.
In addition to Lydon's answer, error 403 is also received if your Data Source location is the same as the Data Target; defined when creating a Job in Glue. Change either of these if they are identical and the issue will be resolved.
You should add a Security configurations(mentioned under Secuity tab on Glue Console). providing S3 Encryption mode either SSE-KMS or SSE-S3.
Security Configuration
Now select the above security configuration while creating your job under Advance Properties.
Duly verify you IAM role & S3 bucket policy.
It will work
How are you providing permission for PassRole to glue role?
{
"Sid": "AllowAccessToRoleOnly",
"Effect": "Allow",
"Action": [
"iam:PassRole",
"iam:GetRole",
"iam:GetRolePolicy",
"iam:ListRolePolicies",
"iam:ListAttachedRolePolicies"
],
"Resource": "arn:aws:iam::*:role/<role>"
}
Usually we create roles using <project>-<role>-<env> e.g. xyz-glue-dev where project name is xyz and env is dev. In that case we use "Resource": "arn:aws:iam::*:role/xyz-*-dev"
For me it was two things.
Access policy for a bucket should be given correctly - bucket/*, here I was missing the * part
Endpoint in VPC must be created for glue to access S3 https://docs.aws.amazon.com/glue/latest/dg/vpc-endpoints-s3.html
After these two settings, my glue job ran successfully. Hope this helps.
Make sure you have given the right policies.
I was facing the same issue, thought I had the role configured well.
But after I erased the role and followed this step, it worked ;]