CREATE_FAILED while creating the CloudFormation Stack from Hyperledger template - amazon-web-services

I am using this template to create the stack:
https://aws-blockchain-templates-us-east-1.s3.us-east-1.amazonaws.com/hyperledger/fabric/templates/simplenetwork/latest/hyperledger.template.yaml
While following this blog-post from AWS, I am getting an error.
Blog - Post Link :
https://aws.amazon.com/blockchain/templates/getting-started/
Region : us-east-1
Error Message :
The following resource(s) failed to create: [FabricEC2CommonStack]. . Rollback requested by user.
CREATE_FAILED AWS::CloudFormation::Stack FabricEC2CommonStack Embedded stack arn:aws:cloudformation:us-east-1:>:stack/FabricStack-FabricEC2CommonStack-NNFUD6RJCZB1/<> was not successfully created: The following resource(s) failed to create: [EC2InstanceForDev].
I have met all the prerequisites.
What could be the reason for this error and how to rectify it?
After this, I get ROLLBACK_IN_PROGRESS and ROLLBACK_COMPLETE.

The Official AWS Blockchain Cloud Formation Template for Hyperledger Fabric is a nested template (our base template calls another template which does all the setup on an EC2 instance which itself creates).
But the problem is it does everything on the EC2-Instance except installing docker-compose & it throws an error that docker-compose command not found at the end which causes the CloudFormation template to break(EC2InstanceForDev) and do a rollback. So instead of using CloudFormation Template, we can run the same script manually on the EC2-instance with a small change. The change is to install docker-compose beforehand. Rest setup remains the same i.e -- 1. Create a VPC, 2. Create Public Subnets, 3. Create EIP if you want to attach it later, 4. Create Key-Pair for SSH, 5. Create IAM Role & Policy, 6. Create Security Group with Inbound 8080(TCP) & 22(SSH), 7. launch an EC2 Instance with the created resources in step (1to6).
AMI which is preferred is -
ami-1853ac65 for us-east-1
ami-25615740 for us-east-2
ami-dff017b8 for us-west-2
Docker Image Repository -
354658284331 for us-east-1
763976151875 for us-east-2
712425161857 for us-west-2
SCRIPT TO RUN ON EC2 (Give chmod 777 and chmod +x for the script) -
#!/bin/bash -x
sudo curl -L https://github.com/docker/compose/releases/download/1.22.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
docker-compose --version
res=$?
echo $res
mkdir /tmp/fabric-install/
cd /tmp/fabric-install/
wget https://aws-blockchain-templates-us-east-1.s3.us-east-1.amazonaws.com/hyperledger/fabric/templates/simplenetwork/latest/HyperLedger-BasicNetwork.tgz -O /home/ec2-user/HyperLedger-BasicNetwork.tgz
cd /home/ec2-user
tar xzvf HyperLedger-BasicNetwork.tgz
rm /home/ec2-user/HyperLedger-BasicNetwork.tgz
chown -R ec2-user:ec2-user HyperLedger-BasicNetwork
chmod +x /home/ec2-user/HyperLedger-BasicNetwork/artifacts/first-run-standalone.sh
/home/ec2-user/HyperLedger-BasicNetwork/artifacts/first-run-standalone.sh us-east-1 example.com org1 org2 org3 mychannel 354658284331.dkr.ecr.us-east-1.amazonaws.com/ 354658284331
res=$?
echo $res
IAM policy which I attached to the role -
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": "*"
}
]
}
NOTE -
Please replace the appropriate AWS ECR account number for your region and appropriate AWS region in the above script and script has (example.com org1 org2 org3 mychannel), Please change this too as per requirement. Its the same RootDomain, Org1SubDomain, Org2SubDomain, Org3SubDomain, ChannelName as we enter in the CF template).
This whole process is tested in the us-east-1 region. The script can be straight deployed in the us-east-1 region. To access the Hyperledger web monitor interface (http://EC2-DNS OR EIP:8080)

You should be Checking your IAM Role and It fixed my issue.

Related

ECR image push AWS CodeBuild issue

COMMAND_EXECUTION_ERROR: Error while executing command: $(aws ecr get-login --no-include-email --region us-east-1). Reason: exit status 127
Below is my buildspec.yml file
version: 0.2
phases:
pre_build:
commands:
- echo Logging in to Amazon ECR...
- aws --version
- $(aws ecr get-login --region ***-east-*)
- REPOSITORY_URI=***********.dkr.ecr.***-east-*.amazonaws.com/repositoryname
- COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION)
- IMAGE_TAG=${COMMIT_HASH:=latest}
build:
commands:
- echo Build started on `date`
- echo Building the Docker image...
- docker build -t $REPOSITORY_URI:latest .
- docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
post_build:
commands:
- echo Build completed
- echo Pushing the Docker images...
- docker push $REPOSITORY_URI:latest
- docker push $REPOSITORY_URI:$IMAGE_TAG
- echo Writing definitions file...
- printf '[{"name":"project-container","imageUri":"%s"}]' $REPOSITORY_URI:$IMAGE_TAG > taskdefinition.json
artifacts:
files: taskdefinition.json
In case it helps someone else, for the work I'm doing inside my build scripts executed by CodeBuild. These are the IAM permissions I had to add (finding them one by one as I hit the error).
{
"Action": [
"ecr:GetAuthorizationToken",
"ecr:DescribeRepositories",
"ecr:CreateRepository",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:BatchCheckLayerAvailability",
"ecr:PutImage",
"ecs:UpdateService"
],
"Resource": "*",
"Effect": "Allow"
} '
I'm sure there are more permissions that may be required if you're doing stuff I'm not doing in your builds. I'm pushing to ECR and forcing the Service (and the related tasks) to deploy the new image.
Your post has inconsistent details, is that intentional? If not, it may be causing the problem. Your code snippet says:
$(aws ecr get-login --region ***-east-*)
Perhaps you purposely redacted the region (what's the point of that btw?) but why is it missing the --no-include-email? Higher up in your post, you do make reference to --no-include-email, so I know you're aware of it.
Run the process outside of a subshell to get a better log
Rather than running it inside a subshell (e.g. $(my command)), for troubleshooting purposes, try running taking the subshell out so you can get better output. Report the results here so we can troubleshoot the error you get.
aws ecr get-login --no-include-email --region us-east-1 <- try this temporarily
vs.
$(aws ecr get-login --no-include-email --region us-east-1)
Have you created an IAM Policy with ECR permissions for CodeBuild to use?
This is very important. CodeBuild needs permission to access ECR on your behalf. Here's an example I found on this blog article. It may need tweaking to your needs. http://beta.awsdocs.com/services/code_build/build_docker_images/
{
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:GetAuthorizationToken",
"ecr:InitiateLayerUpload",
"ecr:PutImage",
"ecr:UploadLayerPart"
],
"Resource": "*",
"Effect": "Allow"
}

Spinnaker + ECR access

I'm having trouble setting up Spinnaker with ECR access.
Background: I installed spinnaker using helm on an EKS cluster and I've confirmed that the cluster has the necessary ECR permissions (by manually running ECR commands from within the clouddriver pod). I am following the instructions here to get Spinnaker+ECR set up: https://www.spinnaker.io/setup/install/providers/docker-registry/
Issue: When I run:
hal config provider docker-registry account add my-ecr-registry \
--address $ADDRESS \
--username AWS \
--password-command "aws --region us-west-2 ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken' | base64 -d | sed 's/^AWS://'"
I get the following output:
+ Get current deployment
Success
- Add the some-ecr-registry account
Failure
Problems in default.provider.dockerRegistry.some-ecr-registry:
- WARNING Resolved Password was empty, missing dependencies for
running password command?
- WARNING You have a supplied a username but no password.
! ERROR Unable to fetch tags from the docker repository: code, 400
Bad Request
? Can the provided user access this repository?
- WARNING None of your supplied repositories contain any tags.
Spinnaker will not be able to deploy any docker images.
? Push some images to your registry.
Problems in halconfig:
- WARNING There is a newer version of Halyard available (1.28.0),
please update when possible
? Run 'sudo apt-get update && sudo apt-get install
spinnaker-halyard -y' to upgrade
- Failed to add account some-ecr-registry for provider
dockerRegistry.
I have confirmed that the aws-cli is installed on the clouddriver pod. And I've confirmed that I can the password-command directly from the clouddriver pod and it successfully returns a token.
I've also confirmed that if I manually generate an ECR token and run hal config provider docker-registry account add my-ecr-registry --address $ADDRESS --username AWS --password-command "echo $MANUALLY_GENERATED_TOKEN" everything works fine. So there is something specific to the password-command that is going wrong and I'm not sure how to debug this.
One other odd behavior: if I simplify the password command to be: hal config provider docker-registry account add some-ecr-registry --address $ADDRESS --username AWS --repositories code --password-command "aws --region us-west-2 ecr get-authorization-token" , I get an addt'l piece of output that says "- WARNING Password command returned non 0 return code stderr/stdout was:bash: aws: command not found". This output only appears for this simplified command.
Any advice on how to debug this would be much appreciated.
If like me your ECR registry is in another account, then you have to forcibly assume the role for the target account where your registry resides
passwordCommand: read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< `aws sts assume-role --role-arn arn:aws:iam::<AWS_ACCOUNT>:role/<SPINNAKER ROLE_NAME> --query "[Credentials.AccessKeyId, Credentials.SecretAccessKey, Credentials.SessionToken]" --output text --role-session-name spinnakerManaged-w2`; export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN; aws ecr get-authorization-token --region us-west-2 --output text --query 'authorizationData[].authorizationToken' --registry-ids <AWS_ACCOUNT> | base64 -d | sed 's/^AWS://'
Credits to https://github.com/spinnaker/spinnaker/issues/5374#issuecomment-607468678
I also installed Spinnaker on AKS and all i did was by using an AWS Managing User with the correct AWS IAM policy to ECR:* i have access to the ECR repositories directly.
I dont think that hal being java based will execute the Bash command in --password-command
set the AWS ECS provider in your spinnaker deployment
Use the Following AWS IAM policy (SpinnakerManagingPolicy) to be attached to the AWS MAnaging User to give access to ECR. Please replace the AWS Accounts based on your need.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:*",
"cloudformation:*",
"ecr:*"
],
"Resource": [
"*"
]
},
{
"Action": "sts:AssumeRole",
"Resource": [
"arn:aws:iam::123456789012:role/SpinnakerManagedRoleAccount1",
"arn:aws:iam::101121314157:role/SpinnakerManagedRoleAccount2",
"arn:aws:iam::202122232425:role/SpinnakerManagedRoleAccount3"
],
"Effect": "Allow"
}
]
}

Error: HEALTH_CONSTRAINTS in AWS CodeDeploy in deploying from Github to AWS CodeDeploy

I am new to AWS and I am trying to deploy using AWS CodeDeploy from Github.
For that, I created my instance named CodeDeployDemo and attached the role and policy to the instance.
Policy ARN arn:aws:iam::378939197253:policy/CE2CodeDeploy9
My policy is:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:Get*",
"s3:List*"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
and also attached policy named AmazonEC2RoleforAWSCodeDeploy
I also installed CodeDeploy agent for my ubuntu step by step as following:
$chmod 400 Code1.pem
$ssh -i "Code1.pem" ubuntu#54.183.22.255
$sudo apt-get update
$sudo apt-get install awscli
$sudo apt-get install ruby2.0
$cd /home/ubuntu
$sudo aws s3 cp s3://aws-codedeploy-us-east-1/latest/install . --region us-east-1
$sudo chmod +x ./install
$sudo ./install auto
and then I create my application and deploy from GitHub to CodeDeploy using CodeDeployDefault.OneAtATime
But at final stage it shows following error:
Deployment failed: Because too many individual instances failed deployment,
too few healthy instances are available for deployment,
or some instances in your deployment group are experiencing problems.
(Error code: HEALTH_CONSTRAINTS)
NOTE: My only one instance is running when my deployment is running.I stopped other instances.
Please help me to find solution for this. THANKS IN ADVANCE.!!
This happens because the codeDeploy checks health of the ec2 instances by hitting instances. Before deployment, you need to run below bash script on the instances and check if the script worked. httpd service must be started. Reboot the instance.
#!/bin/bash
sudo su
apt-get update -y
apt-get install apache2 -y
apt-get install ruby2.0
apt-get install awscli
cd ~
aws s3 cp s3://aws-codedeploy-us-east-1/latest/install . --region us-east-1
chmod +x ./install
./install auto
echo 'hello world' > /var/www/html/index.html
hostname >> /var/www/html/index.html
update-rc.d apache2 defaults
service apache2 start

How to make 'aws ecr get-login' across regions?

I have a docker registry in AWS ECR in region 'us-east-1'. Everything works fine on EC2 instances launched in 'us-east-1'. But when I launch an instance in 'eu-central-1' and try to run
$(aws ecr get-login --region us-east-1)
I get the following response
Error response from daemon: Get https://acc-id.dkr.ecr.us-east-1.amazonaws.com/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
If I run
aws ecr get-login --region us-east-1
I do see the following response
docker login -u AWS -p xxxx -e none https://acc_id.dkr.ecr.us-east-1.amazonaws.com
Ec2 instance has the following policy for the iam-role:
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:BatchGetImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage"
],
"Resource": "*"
}
Please tell me how can I have this cross-region ECR accessibility.
Below procedure can be used for cross region image pull from ECR:
$(aws ecr get-login --no-include-email --region <region having repository> --registry-ids <id>)
docker pull <id>.dkr.ecr.us-west-1.amazonaws.com/<image_name>:<tag>
Below sample shows instance in region us-east-1 is pulling AWS Deep Learning Container Image from ECR hosted in us-west-1
To get region from EC2 instance metadata:
curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document | grep region
"region" : "us-east-1"
ECR Login:
$(aws ecr get-login --no-include-email --region us-west-1 --registry-ids 763104351884)
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
Docker Pull:
docker pull 763104351884.dkr.ecr.us-west-1.amazonaws.com/tensorflow-training:1.13-cpu-py27-ubuntu16.04
1.13-cpu-py27-ubuntu16.04: Pulling from tensorflow-training
34667c7e4631: Already exists
d18d76a881a4: Already exists
119c7358fbfc: Already exists
2aaf13f3eff0: Already exists
7b890657bd19: Already exists
f095a52e6583: Already exists
182b3abfb706: Already exists
89cc0ffab23c: Already exists
7a24716f4857: Already exists
Digest: sha256:65c7f58ac49ed39d5b6bde4f3800dbbf8d9f99b5316292b27315fb6a4b8be56b
Status: Downloaded newer image for 763104351884.dkr.ecr.us-west-1.amazonaws.com/tensorflow-training:1.13-cpu-py27-ubuntu16.04

Ec2 calling CLI from user data

When launching an ec2 instance, how does one go about using CLI commands from within a user data shell script?
When I SSH into the instance I can run CLI commands and everything works as expected.
I'm assuming the issue is that user data is executed as root. When I SSH into the instance and run the CLI commands I do so as ec2-user.
Considering I have to launch an instance every time I want to test my new user data script (this takes 3 minutes every try), I'd really appreciate not have to guess and check my way through this one.
any help is appriciate. Thank you
You newly launched instance needs to have access to the command that you're trying to use. I suggest you to add IAM role set up and added to the instance. This will save you the setup of credential etc... Example IAM policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ec2:DescribeTags",
"ec2:CreateTags"
],
"Effect": "Allow",
"Resource": [
"*"
]
}
]
}
Ubuntu Example userdata
#!/bin/bash -x
apt-get update
apt-get install -y awscli # yum install awscli on CentOS based OS
REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed s/.$//g)
I_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
aws_p="$(which aws) --region ${REGION} --output text"
$aws_p ec2 create-tags --resources $I_ID --tags Key=Name,Value=my-test-server --region $REGION
# ............ more stuff related to your deployment ..... #
This will install awscli on the system and will tag itself with test name.
See how to add proper IAM roles