I'm try to transfer data between s3 and dynamodb with AWSDataPipeline.
error message below...
Unable to create resource for #EmrClusterForLoad_2017-05-15T18:51:19
due to: The supplied ami version is invalid. (Service:
AmazonElasticMapReduce; Status Code: 400; Error Code:
ValidationException; Request ID: 7ebf0367-399f-11e7-b1d7-29efc4730e41)
but, i cannot solve the problem.
help me
aws datapipeline error
ami 3.9.0 is not supported in all regions.
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-3x.html
Also make sure to select a supported EC2 instance type
https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-emr-supported-instance-types.html
Finally you need to set "Resize Cluster Before Running" as false in the Table Load activity.
I made it run after doing all these changes. Hopefully it will help you too.
Related
I'm trying to create ECS Fargate deployment using Cloudformation script, but the script fails during creation of ECS Cluster with error saying that unable to assume service role. I'm not able to figure out what I'm missing in the script, I have tried many ways none of them seem to be working.
Here is the link to cloud formation script as I'm not able to post it here due to character limitation.
ECS Cloudformation script
the error where the resource creation fails.
Resource handler returned message: "Invalid request provided: CreateCluster Invalid Request: Unable to assume the service linked role. Please verify that the ECS service linked role exists. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: e08ab312-4bd8-4c21-852f-ae5d49cc5932; Proxy: null)" (RequestToken: a686f226-e1d3-7b4c-13f1-66fa0a516c51, HandlerErrorCode: InvalidRequest
I'm able to get it working if I create an ECS cluster from aws console, as it creates a service liked role. But I want to work without creating the cluster manually from Console, enerything building up from Cloudformation. I tried looking over aws docs and did dig up Internet but couldn't get it working.Can anyone please help me out.
I have an AWS Glue Spark job that fails with the following error:
An error occurred while calling o362.cache. com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: ...; S3 Extended Request ID: ...; Proxy: null), S3 Extended Request ID: ...
I believe the error is thrown at line where the Spark persist() method is called on a DataFrame. The Glue job is assigned an IAM role that has full S3 access (all locations/operations allowed), yet I'm still getting the S3 exception. I tried setting the "Temporary path" for the Glue job on the AWS Console to a specific S3 bucket with full access, I also tried setting the Spark temporary directory to a specific S3 bucket with full access via:
conf = pyspark.SparkConf()
conf.set('spark.local.dir', 's3://...')
self.sc = SparkContext(conf=conf)
which didn't help. It's very strange that the job is failing even with full S3 access. Not sure what to try next, any help would be really appreciated. Thank you!
We have a CodePipeline process set up, and all stages work except the CodeDeploy stage.
Our pipeline stage is as follows:
GenerateChangeSet for CloudFormation
ExecuteChangeSet for CloudFormation
Deploy for CodeDeploy
These stages were set up and configured by CodeStar.
Our GenerateChangeSet stage tries to access s3 to get our BuildArtifact, but fails with the following error:
Action execution failed
Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 40P7HSHQGWXSRA72; S3 Extended Request ID: I6hiCC7xx+YmnQMLfUnMzZziLDz/5b8uJWzOqWNZwSiVRCS14Q6UyVfss6q80teO5MAGuR9Xft4=; Proxy: null)
This suggests that CloudFormation cannot access s3, but I've checked and rechecked the policy that it uses and it definitely has the correct permissions for accessing s3.
I'm not quite sure why this error is happening, given that the role policy does indeed have access to s3. I even went with the nuclear option of granting this role full control over s3 (with a view to reverting once I solved the issue), but to no avail, the error still occurs.
Has anyone encountered this before? Anyone know why it might be happening?
I discovered the issue. The CloudFormation template file (template.yml and template-configuration.yml) was reading the one from the repo, but that had been removed at some point prior, so I was getting access denied errors from that resource.
I wish the error message was more explicit, it would have saved hours.
Have an RDS database cluster. The deployed version in AWS has the following attributes:
Engine: aurora-postgresql
EngineVersion: '10.11'
My cloudformation template specified 'EngineVersion 10.7', but I believe the minor version was updated automatically on the deployed cluster. When I tried to deploy my Cloudformation stack, i ran into this error(Something very similar, i don't have that error message available right now):
The specified new engine version is different current version: 10.11 (Service: AmazonRDS; Status Code: 400; Error Code: InvalidParameterCombination
I tried updating my CF template to match the deployed engine version, and now I get:
The specified new engine version is same as current version: 10.11 (Service: AmazonRDS; Status Code: 400; Error Code: InvalidParameterCombination
I can't figure out what the InvalidParameterCombination means.
How do I get out of this predicament?
One option i see to work around this issue, is to attach a retention policy (retain) to the cluster, update stack, remove the cluster from the template, update stack and finally import the DB Cluster into the template with the correct version.
Can be difficult with dependencies, for those !Ref calls one could hard code the arn or custer id as a mapping, replace the references with the static mapping and finally follow the steps above. At the end replace hardcoded IDs with newly imported DB cluster and !Ref.
I have an identity TVM on Amazon Elastic Beanstalk that when I try to start it gives
014-07-07 15:29:46 UTC+0100 ERROR Stack named 'awseb-e-ybrpewdr7z-stack' aborted operation. Current state: 'CREATE_FAILED' Reason: The following resource(s) failed to create: AWSEBInstanceLaunchWaitCondition. (Service: AmazonCloudFormation; Status Code: 400; Error Code: OperationError; Request ID: null)
Now If I go to logs and click snapshot logs all that happens is it waits for a while showing processing but then I get no logs showing up. Does anyone please have an idea what the problem is so I can see either the logs and/or sort the startup problem?
This was security issue.
Amazon ECB uses a VPC, now that may already be running especially if a RDB has been created first. Now the groups on the EB instance can show acces rights reuired. But the VPC has an underlying security ACL that is not group based. When a RDB is created first that initially created the VPC the VPC instance gets the rights just to access the DB, so will not allow for example HTTP traffic through this cannot get to the EB to set it up.