resource "aws_ecs_cluster" "demo" {
name = var.ecs_cluster_name
capacity_providers = [local.cluster_name]
default_capacity_provider_strategy {
capacity_provider = local.cluster_name
}
# We need to terminate all instances before the cluster can be destroyed.
# (Terraform would handle this automatically if the autoscaling group depended
# on the cluster, but we need to have the dependency in the reverse
# direction due to the capacity_providers field above).
provisioner "local-exec" {
when = destroy
interpreter = ["bash", "-c"]
command = <<EOT
set -e
CAP_PROVS="$(aws ecs describe-clusters --clusters "${self.arn}" \
--query 'clusters[*].capacityProviders[*]' --output text)"
ASG_ARNS="$(aws ecs describe-capacity-providers \
--capacity-providers "$CAP_PROVS" \
--query 'capacityProviders[*].autoScalingGroupProvider.autoScalingGroupArn' \
--output text)"
if [ -n "$ASG_ARNS" ] && [ "$ASG_ARNS" != "None" ]
then
for ASG_ARN in $ASG_ARNS
do
ASG_NAME=$(echo $ASG_ARN | cut -d/ -f2-)
aws autoscaling update-auto-scaling-group \
--auto-scaling-group-name "$ASG_NAME" \
--min-size 0 --max-size 0 --desired-capacity 0
INSTANCES="$(aws autoscaling describe-auto-scaling-groups \
--auto-scaling-group-names "$ASG_NAME" \
--query 'AutoScalingGroups[*].Instances[*].InstanceId' \
--output text)"
aws autoscaling set-instance-protection --instance-ids $INSTANCES \
--auto-scaling-group-name "$ASG_NAME" \
--no-protected-from-scale-in
done
fi
EOT
}
}
aws_ecs_service local-exec not working and the error say aws command not found
https://github.com/hashicorp/terraform-provider-aws/issues/11409
Error: Error deleting ECS cluster: ClusterContainsContainerInstancesException: The Cluster cannot be deleted while Container InstancesainsContainerInstancesException: The Cluster cantainer Instances are actnot be deleted while Container Instances are active or draining
Error waiting for internet gateway (igw 'detached' (last state:-0e21e59722a46f970) to detach: timeout while waiting for state to become 'detached' (last state: 'detaching', timeout: 15m0s)
aws_ecs_cluster.demo (local-exec): /bin/bash: aws: command not found
Related
I am trying to create a Fargate cluster with ecs-cli using a load balancer
I came up so far with a script to deploy it without, so far my script is
building image
pushing it to ECR
echo ""
echo "creating task execution role"
aws iam wait role-exists --role-name $task_execution_role 2>/dev/null || \ aws iam --region $REGION create-role --role-name $task_execution_role \
--assume-role-policy-document file://task-execution-assume-role.json || return 1
echo ""
echo "adding AmazonECSTaskExecutionRole Policy"
aws iam --region $REGION attach-role-policy --role-name $task_execution_role \
--policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy || return 1
echo ""
echo "creating task role"
aws iam wait role-exists --role-name $task_role 2>/dev/null || \
aws iam --region $REGION create-role --role-name $task_role \
--assume-role-policy-document file://task-role.json
echo ""
echo "adding AmazonS3ReadOnlyAccess Policy"
aws iam --region $REGION attach-role-policy --role-name $task_role \
--policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess || return 1
echo ""
echo "configuring cluster"
ecs-cli configure --cluster $CLUSTER --default-launch-type FARGATE --config-name $CLUSTER --region $REGION || return 1
ecs-cli down --force --cluster-config $CLUSTER --ecs-profile $profile_name || return 1
ecs-cli up --force --cluster-config $CLUSTER --ecs-profile $profile_name || return 1
echo ""
echo "adding ingress rules to security groups"
aws ec2 authorize-security-group-ingress --group-id $SGid --protocol tcp \
--port 80 --cidr 0.0.0.0/0 --region $REGION || return
ecs-cli compose --project-name $SERVICE_NAME service up --create-log-groups \
--cluster-config $CLUSTER --ecs-profile $profile_name
ecs-cli compose --project-name $SERVICE_NAME service ps \
--cluster-config $CLUSTER --ecs-profile $profile_name
aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId,InstanceType,PublicIpAddress,Tags[?Key==`Name`]| [0].Value]' --output table
this works. service is up and I can access it from the public ip.
I now would like to add a load balancer
so I can expose a DNS with route53
Following a few other questions’ advice (this one in particular)
I came up with this
echo ""
echo "configuring cluster"
ecs-cli compose --project-name $CLUSTER create
ecs-cli configure --cluster $CLUSTER --default-launch-type FARGATE --config-name $CLUSTER --region $REGION
echo ""
echo "creating a new AWS CloudFormation stack called amazon-ecs-cli-setup-"$CLUSTER
ecs-cli up --force --cluster-config $CLUSTER --ecs-profile $profile_name
echo "create elb & add a dns CNAME for the elb dns"
aws elb create-load-balancer --load-balancer-name $SERVICE_NAME --listeners Protocol="TCP,LoadBalancerPort=8080,InstanceProtocol=TCP,InstancePort=80" --subnets $subnet1 $subnet2 --security-groups $SGid --scheme internal
echo "create service with above created task definition & elb"
aws ecs create-service \
--cluster $CLUSTER \
--service-name ecs-simple-service-elb \
--cli-input-json file://ecs-simple-service-elb.json
ecs-cli compose --project-name $SERVICE_NAME service up --create-log-groups \
--cluster-config $CLUSTER --ecs-profile $profile_name
echo ""
echo "here are the containers that are running in the service"
ecs-cli compose --project-name $SERVICE_NAME service ps --cluster-config $CLUSTER --ecs-profile $profile_name
and I get the following error messages:
create elb & add a dns CNAME for the elb dns
An error occurred (InvalidParameterException) when calling the CreateService operation: Unable to assume role and validate the listeners configured on your load balancer. Please verify that the ECS service role being passed has the proper permissions.
INFO[0002] Using ECS task definition TaskDefinition="dashboard:4"
WARN[0003] Failed to create log group dashboard-ecs in us-east-1: The specified log group already exists
INFO[0003] Auto-enabling ECS Managed Tags
ERRO[0003] Error creating service error="InvalidParameterException: subnet cannot be blank." service=dashboard
INFO[0003] Created an ECS service service=dashboard taskDefinition="dashboard:4"
FATA[0003] InvalidParameterException: subnet cannot be blank.
here are the containers that are running in the service
Name State Ports TaskDefinition Health
dashboard/4d0ebb65b20e4010b93cb99fb5b9e21d/web STOPPED ExitCode: 137 80->80/tcp dashboard:4 UNKNOWN
My task execution role and task role have this policy attached
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
and the JSON I pass to create service is (copied from the documentation):
{
"serviceName": "dashboard",
"taskDefinition": "dashboard",
"loadBalancers": [
{
"loadBalancerName": "dashboard",
"containerName": "dashboard",
"containerPort": 80
}
],
"desiredCount": 10,
"role": "ecsTaskExecutionRole"
}
what permissions am I missing and what should I change?
IIRC, your ECS service role should have AmazonEC2ContainerServiceRole role permissions to access your ELB and validate the listeners.
See here - https://aws.amazon.com/premiumsupport/knowledge-center/assume-role-validate-listeners/
and here - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_managed_policies.html#AmazonEC2ContainerServiceRole
I'm using AWS ECS CLI within Jenkins pipeline to automate my CICD. What I'm trying to do is I want to create a service based upon the task definition if the service does not exist yet, if the service already exists I just want to update it instead. Here is the create-service cli command:
aws ecs create-service \
--cluster $stg_cluster \
--task-definition $task_def \
--service-name $ecs_service \
--desired-count 1 \
--launch-type EC2 \
--scheduling-strategy REPLICA \
--load-balancers \"targetGroupArn=$target_group_arn,containerName=$container_name,containerPort=80\" \
--deployment-configuration \"maximumPercent=200,minimumHealthyPercent=100\"
It works fine for the first time but will fail at subsequent deployments because of this error:
An error occurred (InvalidParameterException) when calling the CreateService operation: Creation of service was not idempotent.
I believe I have to use the command update-service instead but not sure how to write ECS CLI command to check if an ECS service has already existed. One way I can think of is I can check the returned code from the create-service cli command see if it equals to 0 but again not sure how to retrieve it from the pipeline. Thanks for your help.
First you can run aws ecs describe-services to check if that service is already there and the state is ACTIVE. If it's active then you can run aws ecs update-service to update existing service. If it's not ACTIVE then you can run aws ecs create-service to create the service.
This is a sample code you can use to check if a service is already created and active:
aws ecs describe-services --cluster CLUSTER_NAME --services SERVIVE_NAME | jq --raw-output 'select(.services[].status != null ) | .services[].status'
and then use an if condition to run update-service or create-service.
This is my code in case anyone is interested.
#Jenkinsfile pipeline
stage('Deploy')
{
steps {
....
script {
int count = sh(script: """
aws ecs describe-services --cluster $ecs_cluster --services $ecs_service | jq '.services | length'
""",
returnStdout: true).trim()
echo "service count: $count"
if (count > 0) {
//ECS service exists: update
echo "Updating ECS service $ecs_service..."
sh(script: """
aws ecs update-service \
--cluster $ecs_cluster \
--service $ecs_service \
--task-definition $task_def \
""")
}
else {
//ECS service does not exist: create new
echo "Creating new ECS service $ecs_service..."
sh(script: """
aws ecs create-service \
--cluster $ecs_cluster \
--task-definition $task_def \
--service-name $ecs_service \
--desired-count 1 \
--launch-type EC2 \
--scheduling-strategy REPLICA \
--load-balancers \"targetGroupArn=${target_group_arn},containerName=$app_name,containerPort=80\" \
--deployment-configuration \"maximumPercent=200,minimumHealthyPercent=100\"
""")
}
}
}
}
Use query parameter in aws cli :
#!/bin/bash
CLUSTER="test-cluster-name"
SERVICE="test-service-name"
echo "check ECS service exists"
status=$(aws ecs describe-services --cluster ${CLUSTER} --services ${SERVICE} --query 'failures[0].reason' --output text)
if [[ "${status}" == "MISSING" ]]; then
echo "ecs service ${SERVICE} missing in ${CLUSTER}"
exit 1
fi
same in jenkins pipline:
stage('Deploy')
{
steps {
....
script {
string status = sh(script: """
aws ecs describe-services --cluster $ecs_cluster --services $ecs_service --query 'failures[0].reason' --output text
""",
returnStdout: true).trim()
echo "service status: $status"
if (status == 'MISSING') {
echo "ecs service: $ecs_service does not exists in $ecs_cluster"
// put create service logic
}
}}}
I am trying to create a read replica of the production database and then I want to promote the read replica to a test database.
I have used this awscli command to create a read replica
aws rds create-db-instance-read-replica --db-instance-identifier test-database --source-db-instance-identifier production-database --region eu-central-1
I know I cannot issue a promote read replica command immediately as I would get an error.
bash-3.2$ aws rds promote-read-replica --db-instance-identifier test-database --region eu-central-1
An error occurred (InvalidDBInstanceState) when calling the PromoteReadReplica operation: DB Instance is not in an available state.
How can I check if the read replica is created successfully so I can issue a promote read replica command?
I tried to query the events for the database but it is returning empty.
bash-3.2$ aws rds describe-events --source-identifier test-database --source-type db-instance
{
"Events": []
}
I am trying to do this in the Jenkins pipeline so it has to be checked programmatically.
Kindly advise.
You can use describe-db-instances and create a simple while-based waiter for the replica to be available.
For example:
while true; do
db_status=$(aws rds describe-db-instances \
--db-instance-identifier test-database \
--query 'DBInstances[0].DBInstanceStatus' \
--output text)
[[ $db_status != "available" ]] \
&& (echo $db_status; sleep 5) \
|| break
done
echo "Finally ${db_status}"
The above will check the status of test-database every 5 seconds until its available.
i m trying to continue the above script by adding parameter group and reboot the read replica .
db_status=$(aws rds describe-db-instances \
--db-instance-identifier db-replica \
--profile xxx \
--region us-west-2 \
--query 'DBInstances[0].DBInstanceStatus' \
--output text)
[[ $db_status != "available" ]] \
&& (echo $db_status; sleep 15)\
|| break
echo "Finally ${db_status}"
if [ $db_status == "available" ]
then
echo "replica created successfully"
echo "Rebooting replica"
aws rds reboot-db-instance --db-instance-identifier db-replica
sleep 5;
else
echo "replica rebooted successfully"
fi
exit 1
In my Terraform AWS Docker Swarm module I use cloud-init to initialize the EC2 instance. However, Terraform says the resource is ready before cloud-init finishes. Is there a way of making it wait for cloud-init to finish ideally without SSHing or checking for a port to be up using a null resource.
Your managers and workers both use template_cloudinit_config. They also have ec2:CreateTags.
You can use an EC2 resource tag like trajano/terraform-docker-swarm-aws/cloudinit-complete to indicate that the cloudinit has finished.
You could add this final part to each to invoke a tagging script:
part {
filename = "tag_complete.sh"
content = local.tag_complete_script
content_type = "text/x-shellscript"
}
And declare tag_complete_script be the following:
locals {
tag_complete_script = <<-EOF
#!/bin/bash
instance_id="${TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \
&& curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/instance-id}"
aws ec2 create-tags --resources "$instance_id" --tags 'Key=trajano/terraform-docker-swarm-aws/cloudinit-complete,Value=true'
EOF
}
Then with a null_resource, you wait for the tag to appear (wrote this on my phone, so use it for a general idea, but I don't expect that it will work without testing and edits):
resource "null_resource" "wait_for_cloudinit" {
provisioner "local-exec" {
command = <<-EOF
#!/bin/bash
poll_tags="aws ec2 describe-tags --filters 'Name=resource-id,Values=${join(",", aws_instance.managers[*].id)}' 'Name=key,Values=trajano/terraform-docker-swarm-aws/cloudinit-complete' --output text --query 'Tags[*].Value'"
expected='${join(",", formatlist("true", aws_instance.managers[*].id))}'
$tags="$($poll_tags)"
while [[ "$tags" != "$expected" ]] ; do
$tags="$($poll_tags)"
done
EOF
}
}
This way you can have dependencies on null_resource.wait_for_cloudinit on any resources that need to run after cloudinit has completed.
Another possible approach is using AWS Systems Manager Run Command, if available on your AMI.
You create an SSM Document with Terraform that uses the cloud-init status --wait command, then you trigger the command from a local provisioner, and wait for it to complete. In this way, you don't have to play around with tags, and you are 100% sure cloud-init has been completed.
This is an example of the document you can create with Terraform:
resource "aws_ssm_document" "cloud_init_wait" {
name = "cloud-init-wait"
document_type = "Command"
document_format = "YAML"
content = <<-DOC
schemaVersion: '2.2'
description: Wait for cloud init to finish
mainSteps:
- action: aws:runShellScript
name: StopOnLinux
precondition:
StringEquals:
- platformType
- Linux
inputs:
runCommand:
- cloud-init status --wait
DOC
}
and then you can use a local-provisioner inside the EC2 instance block, or in a null resource, up to what you have to do with it.
The provisioner would be more or less like this:
provisioner "local-exec" {
interpreter = ["/bin/bash", "-c"]
command = <<-EOF
set -Ee -o pipefail
export AWS_DEFAULT_REGION=${data.aws_region.current.name}
command_id=$(aws ssm send-command --document-name ${aws_ssm_document.cloud_init_wait.arn} --instance-ids ${self.id} --output text --query "Command.CommandId")
if ! aws ssm wait command-executed --command-id $command_id --instance-id ${self.id}; then
echo "Failed to start services on instance ${self.id}!";
echo "stdout:";
aws ssm get-command-invocation --command-id $command_id --instance-id ${self.id} --query StandardOutputContent;
echo "stderr:";
aws ssm get-command-invocation --command-id $command_id --instance-id ${self.id} --query StandardErrorContent;
exit 1;
fi;
echo "Services started successfully on the new instance with id ${self.id}!"
EOF
}
I want to save output of an AWS CLI in a variable and use that variable in another AWS CLI, what I did is as follows:
taskarn= aws ecs list-tasks --cluster mycluster --service-name "myService" --region "eu-west-1" --output text | grep "arn" | tr -d '"'
echo $taskarn; //empty
aws ecs stop-task --cluster mycluster --task $taskarn --region "eu-west-1"
when I echo $taskarn, it is empty.
Any help would be appreciated.
I used the following command and it works fine:
taskarn=$(aws ecs list-tasks --cluster mycluster --service-name "myservice" --region "eu-west-1" | grep "arn" | tr -d '"')
echo $taskarn;
aws ecs stop-task --cluster mycluster --task $taskarn --region "eu-west-1"
Use backquote to execute the command and assign the result to the variable.
taskarn=`aws ecs list-tasks --cluster mycluster --service-name "myService" --region "eu-west-1" --output text | grep "arn" | tr -d '"'`
But the correct way is to use the --query option of the CLI to extract what you want.