My AWS Elemental Media Convert is showing:
undefined (undefined)
Network Failure: {"message":"Network Failure","code":"NetworkingError","time":"xxx","hostname":"mediaconvert-eu-west-1.amazonaws.com","retryable":true}
The undefined network failure error is a sort of catch-all, so a closer look at the job configuration and access permissions to any specified resources (S3 buckets, IAM roles, etc.) would be required to identify the exact cause. I would recommend opening a ticket with AWS support for additional support.
When I've seen this error, the most common reason was because the user was specifying a MediaConvert queue that belonged to a different AWS account. Check to make sure you aren't referencing any resources that your account/role don't have access to, and make sure you're running the job in the correct region (if any referenced resources are region-specific).
A future update to MediaConvert will address this generalized error to make it more specific and actionable by the user.
Related
I found a really weird bug on aws js sdk.
If I try to delete/add LifecycleRules in my s3Bucket (using putBucketLifecycleConfiguration) and fetch theses rules just after this call using getBucketLifecycleConfiguration I can receive more or less rules than that I've put with putBucketLifecycleConfiguration.
If I keep calling getBucketLifecycleConfiguration I will continue to receive more or less rules that I've put, it seems to be a random behavior...
Do you know if it's a known bug or the reason of this behavior?
NOTE: It seems it has the same behavior with aws s3api get-bucket-lifecycle-configuration AND also in the AWS Management Console.
Maybe we have to wait a moment for AWS servers to replicate the data on all servers?
Thanks!
It seems to be the normal behavior of aws, look at this link:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-set-lifecycle-configuration-intro.html
Propagation delay
When you add an S3 Lifecycle configuration to a bucket, there is
usually some lag before a new or updated Lifecycle configuration is
fully propagated to all the Amazon S3 systems. Expect a delay of a
few minutes before the configuration fully takes effect.
This delay can also occur when you delete an S3 Lifecycle configuration.
I can't get rid of five buckets in S3. Every screen in the AWS console says "Error Data not found" (i.e. Overview, Properties, Permissions, Management, Access points).
I can't set lifecycle rules to delete objects, but the buckets never had anything in them and versioning was never enabled anyway.
I've also tried forcing it in my terminal...
aws s3 rb s3://bucketblah --force
...but it fails and I get remove_bucket failed: Unable to delete all objects in the bucket, bucket will not be deleted.
Help me Obi Wan...
Amazon S3 is what gives a developer their power. It's an energy field created by objects stored in the cloud. It surrounds us and penetrates us. It binds the Internet together.
Some may mock Amazon S3 because they cannot sense invisible objects in their bucket. But the wise Jedi amongst us will check whether the bucket has Versioning enabled. When attempting to rid the galaxy of their bucket, they might see messages such as:
$ aws s3 rb s3://rebel-base --force
remove_bucket failed: An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: The rebel base you tried to destroy is not empty. You must delete all versions in the bucket.
If such resistance is met, sneak into the Amazon S3 management console, select the bucket, choose Versions: Show and reach out with your mind. If any deleted versions of objects are displayed, delete them within this view until all objects cry out in terror and are suddenly silenced.
If this does not lead to the resolution you seek, then check that your Master has allocated sufficient access privileges for you to access the central computer and this particular bucket. It is also possible that these buckets have bucket policies that override the central computer via Deny policies. If so, attempt to bypass security by deleting the bucket policy first, then destroy the rebel bucket. You know better than to trust a strange computer!
I'm trying to run a simple GroundTruth labeling job with a public workforce. I upload my images to S3, start creating the labeling job, generate the manifest using their tool automatically, and explicitly specify a role that most certainly has permissions on both S3 bucket (input and output) as well as full access to SageMaker. Then I create the job (standard rest of stuff -- I just wanted to be clear that I'm doing all of that).
At first, everything looks fine. All green lights, it says it's in progress, and the images are properly showing up in the bottom where the dataset is. However, after a few minutes, the status changes to Failure and I get this: ClientError: Access Denied. Cannot access manifest file: arn:aws:sagemaker:us-east-1:<account number>:labeling-job/<job name> using roleArn: null in the reason for failure.
I also get the error underneath (where there used to be images but now there are none):
The specified key <job name>/manifests/output/output.manifest isn't present in the S3 bucket <output bucket>.
I'm very confused for a couple of reasons. First of all, this is a super simple job. I'm just trying to do the most basic bounding box example I can think of. So this should be a very well-tested path. Second, I'm explicitly specifying a role arn, so I have no idea why it's saying it's null in the error message. Is this an Amazon glitch or could I be doing something wrong?
The role must include SageMakerFullAccess and access to the S3 bucket, so it looks like you've got that covered :)
Please check that:
the user creating the labeling job has Cognito permissions: https://docs.aws.amazon.com/sagemaker/latest/dg/sms-getting-started-step1.html
the manifest exists and is at the right S3 location.
the bucket is in the same region as SageMaker.
the bucket doesn't have any bucket policy restricting access.
If that still doesn't fix it, I'd recommend opening a support ticket with the labeling job id, etc.
Julien (AWS)
There's a bug whereby sometimes the console will say something like 401 ValidationException: The specified key s3prefix/smgt-out/yourjobname/manifests/output/output.manifest isn't present in the S3 bucket yourbucket. Request ID: a08f656a-ee9a-4c9b-b412-eb609d8ce194 but that's not the actual problem. For some reason the console is displaying the wrong error message. If you use the API (or AWS CLI) to DescribeLabelingJob like
aws sagemaker describe-labeling-job --labeling-job-name yourjobname
you will see the actual problem. In my case, one of the S3 files that define the UI instructions was missing.
I had the same issue when I tried to write to a different bucket to the one that was used successfully before.
Apparently the IAM role ARN can be assigned permissions for a particular bucket only.
I would suggest to refer to CloudWatch logs and look for a CloudWatch>>CloudWatch Logs >> Log groups >> /aws/sagemaker/LabelingJobs group. I had all points ticked from another post, but my pre-processing Lambda function had wrong id for my region and the error was obvious in the logs.
Trying to put cloud watch logs into kineses firehose.
Followed below:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#FirehoseExample
Got this error
An error occurred (InvalidParameterException) when calling the PutSubscriptionFilter operation: Could not deliver test message to specified Firehose stream. Check if t
e given Firehose stream is in ACTIVE state.
aws logs put-subscription-filter --log-group-name "xxxx" --filter-name "xxx" --filter-pattern "{$.httpMethod = GET}" --destination-arn "arn:aws:firehose:us-east-1:12345567:deliverystream/xxxxx" --role-arn "arn:aws:iam::12344566:role/xxxxx"
You need to update the trust policy of your IAM role so that it gives permissions to the logs.amazonaws.com service principal to assume it, otherwise CloudWatch Logs won't be able to assume your role to publish events to your Kinesis stream. (Obviously you also need to double-check the permissions on your role to make sure it has permissions to read from your Log Group and write to your Kinesis Stream.)
It would be nice if they added this to the error message to help point people in the right direction...
The most likely problem that causes this error is a permissions issue. i.e. something wrong in the definition of the IAM role you passed to --role-arn. You may want to double check that the role and its permissions were set up properly as described in the doc.
I was getting a similar error when subscribing to a cloudwatch loggroup and publishing to a Kinesis stream. Cdk was not defining a dependency needed for the SubscriptionFilter to be created after the Policy that would allow the filtered events to be published in Kinesis. This is reported in this github cdk issue:
https://github.com/aws/aws-cdk/issues/21827
I ended up using the workaround implemented by github user AlexStasko: https://github.com/AlexStasko/aws-subscription-filter-issue/blob/main/lib/app-stack.ts
If your Firehose is active status and you can send log stream then the remaining issue is only policy.
I got the similar issue when follow the tutorial. The one confused here is Kinesis part and Firehose part, we may mixed up together. You need to recheck your: ~/PermissionsForCWL.json, with details part of:
....
"Action":["firehose:*"], *// You could confused with kinesis:* like me*
"Resource":["arn:aws:firehose:region:123456789012:*"]
....
When I did the tutorial you mentioned, it was defaulting to a different region so I had to pass --region with my region. It wasn't until I did the entire steps with the correct region that it worked.
For me I think this issue was occurring due to the time it takes for the IAM data plane to settle after new roles are created via regional IAM endpoints for regions that are geographically far away from us-east-1.
I have a custom Lambda CF resource that auto-subscribes all existing and future log groups to a Firehose via a subscription filter. The IAM role gets deployed for CW Logs then very quickly the Lambda function tries to subscribe the log groups. And on occasion this error would happen.
I added a time.sleep(30) to my code (this code only runs once a stack creation so it's not going to hurt anything to wait 30 seconds).
We are trying to get DMS set up with an S3 Source however we are unable to connect the replication instance to the Source S3 endpoint.
When we run a connection test on the source endpoint, the error we receive is:
Error Details: [errType=ERROR_RESPONSE, status=1020414, errMessage= Failed to connect to database., errDetails=]
We have followed the documentation however we are still unable to get the connection to work. The bucket is within the VPC that the replication instance has access to, and the IAM role has the GetObject, ListBucket and dms* permissions. I'm 95% sure that the JSON mapping file is set up correctly with schema and table names pointing to the right place.
Due to the lack of error messages or detailed reasons why we can't connect to the source database (the S3 bucket/CSV file), debugging this feels a tad hit and miss. We are using the Amazon Console and not the CLI, if that makes much of a difference.
I had this same error.
Check this troubleshooting guide. It covers the basic configuration problems you might run into.
My answer wasn't there, tho, and I couldn't find it anywhere, not even asking in the official forums.
In my case, for some reason I thought I should use the full bucket name in the "Bucket Name" field, like "arn:aws:s3:::my-bucket". Probably because I had to use the ARN for the role in the previous field.
And the error message when you try to connect to it will not be clear, it only says it couldn't connect to the bucket. Anyway, you don't need to provide an ARN, just the bucket's name, as in "my-bucket".