Jenkins pipeline - check if an ECS service exist - amazon-web-services

I'm using AWS ECS CLI within Jenkins pipeline to automate my CICD. What I'm trying to do is I want to create a service based upon the task definition if the service does not exist yet, if the service already exists I just want to update it instead. Here is the create-service cli command:
aws ecs create-service \
--cluster $stg_cluster \
--task-definition $task_def \
--service-name $ecs_service \
--desired-count 1 \
--launch-type EC2 \
--scheduling-strategy REPLICA \
--load-balancers \"targetGroupArn=$target_group_arn,containerName=$container_name,containerPort=80\" \
--deployment-configuration \"maximumPercent=200,minimumHealthyPercent=100\"
It works fine for the first time but will fail at subsequent deployments because of this error:
An error occurred (InvalidParameterException) when calling the CreateService operation: Creation of service was not idempotent.
I believe I have to use the command update-service instead but not sure how to write ECS CLI command to check if an ECS service has already existed. One way I can think of is I can check the returned code from the create-service cli command see if it equals to 0 but again not sure how to retrieve it from the pipeline. Thanks for your help.

First you can run aws ecs describe-services to check if that service is already there and the state is ACTIVE. If it's active then you can run aws ecs update-service to update existing service. If it's not ACTIVE then you can run aws ecs create-service to create the service.
This is a sample code you can use to check if a service is already created and active:
aws ecs describe-services --cluster CLUSTER_NAME --services SERVIVE_NAME | jq --raw-output 'select(.services[].status != null ) | .services[].status'
and then use an if condition to run update-service or create-service.

This is my code in case anyone is interested.
#Jenkinsfile pipeline
stage('Deploy')
{
steps {
....
script {
int count = sh(script: """
aws ecs describe-services --cluster $ecs_cluster --services $ecs_service | jq '.services | length'
""",
returnStdout: true).trim()
echo "service count: $count"
if (count > 0) {
//ECS service exists: update
echo "Updating ECS service $ecs_service..."
sh(script: """
aws ecs update-service \
--cluster $ecs_cluster \
--service $ecs_service \
--task-definition $task_def \
""")
}
else {
//ECS service does not exist: create new
echo "Creating new ECS service $ecs_service..."
sh(script: """
aws ecs create-service \
--cluster $ecs_cluster \
--task-definition $task_def \
--service-name $ecs_service \
--desired-count 1 \
--launch-type EC2 \
--scheduling-strategy REPLICA \
--load-balancers \"targetGroupArn=${target_group_arn},containerName=$app_name,containerPort=80\" \
--deployment-configuration \"maximumPercent=200,minimumHealthyPercent=100\"
""")
}
}
}
}

Use query parameter in aws cli :
#!/bin/bash
CLUSTER="test-cluster-name"
SERVICE="test-service-name"
echo "check ECS service exists"
status=$(aws ecs describe-services --cluster ${CLUSTER} --services ${SERVICE} --query 'failures[0].reason' --output text)
if [[ "${status}" == "MISSING" ]]; then
echo "ecs service ${SERVICE} missing in ${CLUSTER}"
exit 1
fi
same in jenkins pipline:
stage('Deploy')
{
steps {
....
script {
string status = sh(script: """
aws ecs describe-services --cluster $ecs_cluster --services $ecs_service --query 'failures[0].reason' --output text
""",
returnStdout: true).trim()
echo "service status: $status"
if (status == 'MISSING') {
echo "ecs service: $ecs_service does not exists in $ecs_cluster"
// put create service logic
}
}}}

Related

how to set the parameter aws ecs update-service -DeploymentConfiguration_MinimumHealthyPercent XX?

there are 2 documents
1 https://docs.aws.amazon.com/powershell/latest/reference/items/Update-ECSService.html
2 https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ecs/update-service.html
In the first document I understand how to specify the parameter "-DeploymentConfiguration_MinimumHealthyPercent 50 "
The second document does not tell me how to do it
this example works:
Update-ECSService -Cluster $Cluster -Service $Service -ForceNewDeployment 1 -HealthCheckGracePeriodSecond 1200 -DeploymentConfiguration_MinimumHealthyPercent 50
How do I do the same thing through the AWS CLI?
aws ecs update-service --cluster $cluster --service $service --force-new-deployment --health-check-grace-period-seconds 1200 ( -DeploymentConfiguration_MinimumHealthyPercent 50 ???)
could you please tell me how to set the parameter ?
aws ecs update-service -DeploymentConfiguration_MinimumHealthyPercent XX
You can set the minimumHealthyPercent value as part of --deployment-configuration argument:
aws ecs update-service --cluster $cluster \
--service $service \
--force-new-deployment \
--health-check-grace-period-seconds 1200 \
--deployment-configuration "maximumPercent=100,minimumHealthyPercent=50,deploymentCircuitBreaker={enable=boolean,rollback=boolean}"

aws_ecs_cluster local-exec aws not found

resource "aws_ecs_cluster" "demo" {
name = var.ecs_cluster_name
capacity_providers = [local.cluster_name]
default_capacity_provider_strategy {
capacity_provider = local.cluster_name
}
# We need to terminate all instances before the cluster can be destroyed.
# (Terraform would handle this automatically if the autoscaling group depended
# on the cluster, but we need to have the dependency in the reverse
# direction due to the capacity_providers field above).
provisioner "local-exec" {
when = destroy
interpreter = ["bash", "-c"]
command = <<EOT
set -e
CAP_PROVS="$(aws ecs describe-clusters --clusters "${self.arn}" \
--query 'clusters[*].capacityProviders[*]' --output text)"
ASG_ARNS="$(aws ecs describe-capacity-providers \
--capacity-providers "$CAP_PROVS" \
--query 'capacityProviders[*].autoScalingGroupProvider.autoScalingGroupArn' \
--output text)"
if [ -n "$ASG_ARNS" ] && [ "$ASG_ARNS" != "None" ]
then
for ASG_ARN in $ASG_ARNS
do
ASG_NAME=$(echo $ASG_ARN | cut -d/ -f2-)
aws autoscaling update-auto-scaling-group \
--auto-scaling-group-name "$ASG_NAME" \
--min-size 0 --max-size 0 --desired-capacity 0
INSTANCES="$(aws autoscaling describe-auto-scaling-groups \
--auto-scaling-group-names "$ASG_NAME" \
--query 'AutoScalingGroups[*].Instances[*].InstanceId' \
--output text)"
aws autoscaling set-instance-protection --instance-ids $INSTANCES \
--auto-scaling-group-name "$ASG_NAME" \
--no-protected-from-scale-in
done
fi
EOT
}
}
aws_ecs_service local-exec not working and the error say aws command not found
https://github.com/hashicorp/terraform-provider-aws/issues/11409
Error: Error deleting ECS cluster: ClusterContainsContainerInstancesException: The Cluster cannot be deleted while Container InstancesainsContainerInstancesException: The Cluster cantainer Instances are actnot be deleted while Container Instances are active or draining
Error waiting for internet gateway (igw 'detached' (last state:-0e21e59722a46f970) to detach: timeout while waiting for state to become 'detached' (last state: 'detaching', timeout: 15m0s)
aws_ecs_cluster.demo (local-exec): /bin/bash: aws: command not found

deploy ecs-cli fargate cluster with load balancer

I am trying to create a Fargate cluster with ecs-cli using a load balancer
I came up so far with a script to deploy it without, so far my script is
building image
pushing it to ECR
echo ""
echo "creating task execution role"
aws iam wait role-exists --role-name $task_execution_role 2>/dev/null || \ aws iam --region $REGION create-role --role-name $task_execution_role \
--assume-role-policy-document file://task-execution-assume-role.json || return 1
echo ""
echo "adding AmazonECSTaskExecutionRole Policy"
aws iam --region $REGION attach-role-policy --role-name $task_execution_role \
--policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy || return 1
echo ""
echo "creating task role"
aws iam wait role-exists --role-name $task_role 2>/dev/null || \
aws iam --region $REGION create-role --role-name $task_role \
--assume-role-policy-document file://task-role.json
echo ""
echo "adding AmazonS3ReadOnlyAccess Policy"
aws iam --region $REGION attach-role-policy --role-name $task_role \
--policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess || return 1
echo ""
echo "configuring cluster"
ecs-cli configure --cluster $CLUSTER --default-launch-type FARGATE --config-name $CLUSTER --region $REGION || return 1
ecs-cli down --force --cluster-config $CLUSTER --ecs-profile $profile_name || return 1
ecs-cli up --force --cluster-config $CLUSTER --ecs-profile $profile_name || return 1
echo ""
echo "adding ingress rules to security groups"
aws ec2 authorize-security-group-ingress --group-id $SGid --protocol tcp \
--port 80 --cidr 0.0.0.0/0 --region $REGION || return
ecs-cli compose --project-name $SERVICE_NAME service up --create-log-groups \
--cluster-config $CLUSTER --ecs-profile $profile_name
ecs-cli compose --project-name $SERVICE_NAME service ps \
--cluster-config $CLUSTER --ecs-profile $profile_name
aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId,InstanceType,PublicIpAddress,Tags[?Key==`Name`]| [0].Value]' --output table
this works. service is up and I can access it from the public ip.
I now would like to add a load balancer
so I can expose a DNS with route53
Following a few other questions’ advice (this one in particular)
I came up with this
echo ""
echo "configuring cluster"
ecs-cli compose --project-name $CLUSTER create
ecs-cli configure --cluster $CLUSTER --default-launch-type FARGATE --config-name $CLUSTER --region $REGION
echo ""
echo "creating a new AWS CloudFormation stack called amazon-ecs-cli-setup-"$CLUSTER
ecs-cli up --force --cluster-config $CLUSTER --ecs-profile $profile_name
echo "create elb & add a dns CNAME for the elb dns"
aws elb create-load-balancer --load-balancer-name $SERVICE_NAME --listeners Protocol="TCP,LoadBalancerPort=8080,InstanceProtocol=TCP,InstancePort=80" --subnets $subnet1 $subnet2 --security-groups $SGid --scheme internal
echo "create service with above created task definition & elb"
aws ecs create-service \
--cluster $CLUSTER \
--service-name ecs-simple-service-elb \
--cli-input-json file://ecs-simple-service-elb.json
ecs-cli compose --project-name $SERVICE_NAME service up --create-log-groups \
--cluster-config $CLUSTER --ecs-profile $profile_name
echo ""
echo "here are the containers that are running in the service"
ecs-cli compose --project-name $SERVICE_NAME service ps --cluster-config $CLUSTER --ecs-profile $profile_name
and I get the following error messages:
create elb & add a dns CNAME for the elb dns
An error occurred (InvalidParameterException) when calling the CreateService operation: Unable to assume role and validate the listeners configured on your load balancer. Please verify that the ECS service role being passed has the proper permissions.
INFO[0002] Using ECS task definition TaskDefinition="dashboard:4"
WARN[0003] Failed to create log group dashboard-ecs in us-east-1: The specified log group already exists
INFO[0003] Auto-enabling ECS Managed Tags
ERRO[0003] Error creating service error="InvalidParameterException: subnet cannot be blank." service=dashboard
INFO[0003] Created an ECS service service=dashboard taskDefinition="dashboard:4"
FATA[0003] InvalidParameterException: subnet cannot be blank.
here are the containers that are running in the service
Name State Ports TaskDefinition Health
dashboard/4d0ebb65b20e4010b93cb99fb5b9e21d/web STOPPED ExitCode: 137 80->80/tcp dashboard:4 UNKNOWN
My task execution role and task role have this policy attached
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
and the JSON I pass to create service is (copied from the documentation):
{
"serviceName": "dashboard",
"taskDefinition": "dashboard",
"loadBalancers": [
{
"loadBalancerName": "dashboard",
"containerName": "dashboard",
"containerPort": 80
}
],
"desiredCount": 10,
"role": "ecsTaskExecutionRole"
}
what permissions am I missing and what should I change?
IIRC, your ECS service role should have AmazonEC2ContainerServiceRole role permissions to access your ELB and validate the listeners.
See here - https://aws.amazon.com/premiumsupport/knowledge-center/assume-role-validate-listeners/
and here - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_managed_policies.html#AmazonEC2ContainerServiceRole

How to identify if create read replica action is complete and finished

I am trying to create a read replica of the production database and then I want to promote the read replica to a test database.
I have used this awscli command to create a read replica
aws rds create-db-instance-read-replica --db-instance-identifier test-database --source-db-instance-identifier production-database --region eu-central-1
I know I cannot issue a promote read replica command immediately as I would get an error.
bash-3.2$ aws rds promote-read-replica --db-instance-identifier test-database --region eu-central-1
An error occurred (InvalidDBInstanceState) when calling the PromoteReadReplica operation: DB Instance is not in an available state.
How can I check if the read replica is created successfully so I can issue a promote read replica command?
I tried to query the events for the database but it is returning empty.
bash-3.2$ aws rds describe-events --source-identifier test-database --source-type db-instance
{
"Events": []
}
I am trying to do this in the Jenkins pipeline so it has to be checked programmatically.
Kindly advise.
You can use describe-db-instances and create a simple while-based waiter for the replica to be available.
For example:
while true; do
db_status=$(aws rds describe-db-instances \
--db-instance-identifier test-database \
--query 'DBInstances[0].DBInstanceStatus' \
--output text)
[[ $db_status != "available" ]] \
&& (echo $db_status; sleep 5) \
|| break
done
echo "Finally ${db_status}"
The above will check the status of test-database every 5 seconds until its available.
i m trying to continue the above script by adding parameter group and reboot the read replica .
db_status=$(aws rds describe-db-instances \
--db-instance-identifier db-replica \
--profile xxx \
--region us-west-2 \
--query 'DBInstances[0].DBInstanceStatus' \
--output text)
[[ $db_status != "available" ]] \
&& (echo $db_status; sleep 15)\
|| break
echo "Finally ${db_status}"
if [ $db_status == "available" ]
then
echo "replica created successfully"
echo "Rebooting replica"
aws rds reboot-db-instance --db-instance-identifier db-replica
sleep 5;
else
echo "replica rebooted successfully"
fi
exit 1

How to sort results from `list-tasks` in aws-cli?

I have few tasks running i want to get the latest(task which started last, with last created-at) task arn.
I am using this command
aws ecs list-tasks --cluster {cluster} --family {family} --desired-status 'STOPPED'
after exploring a little i got to know about sort_by. I tried
aws ecs list-tasks --cluster {cluster} --family {family} --desired-status 'STOPPED' --query "sort_by(taskArns, &CreatedAt)"
But this gives error
In function sort_by(), invalid type for value: <some_task_arn>, expected one of: ['string', 'number'], received: "null"
list-tasks will not give you that information. Have to use both list-tasks and describe-tasks.
I will provide example on my cluster, with RUNNING state tasks. You will have to adjust it to your need.
1. Get the list of tasks
task_arns=$(aws ecs list-tasks --cluster ${cluster[Name]} \
--desired-status 'RUNNING' \
--query 'taskArns' --output text)
echo ${task_arns}
Should give a list of Arns of your tasks, e.g.:
arn:aws:ecs:us-east-1:275795381673:task/0053c603-a6c9-4044-89f5-b0edc8f6de3f arn:aws:ecs:us-east-1:275795381673:task/0b4626ea-0f2b-4c99-9e90-010e8a0c8ad3 arn:aws:ecs:us-east-1:275795381673:task/0d4aa5f2-f547-45ad-b1f8-ed84ef1d678c arn:aws:ecs:us-east-1:275795381673:task/190a320f-4b68-497a-921e-439460447d45 arn:aws:ecs:us-east-1:275795381673:task/c979f4a2-3665-4c56-93c6-e9b88f6b3519
2. Get sorted tasks Arns
aws ecs describe-tasks --cluster ${cluster[Name]} \
--tasks ${task_arns} \
--query "tasks[] | reverse(sort_by(#, &createdAt)) | [].[createdAt,taskArn]" \
--output table
Should give, eg:
----------------------------------------------------------------------------------------------------
| DescribeTasks |
+----------------+---------------------------------------------------------------------------------+
| 1589888493.15 | arn:aws:ecs:us-east-1:275795381673:task/c979f4a2-3665-4c56-93c6-e9b88f6b3519 |
| 1589888501.348| arn:aws:ecs:us-east-1:275795381673:task/190a320f-4b68-497a-921e-439460447d45 |
| 1589888499.438| arn:aws:ecs:us-east-1:275795381673:task/0d4aa5f2-f547-45ad-b1f8-ed84ef1d678c |
| 1589888500.312| arn:aws:ecs:us-east-1:275795381673:task/0b4626ea-0f2b-4c99-9e90-010e8a0c8ad3 |
| 1589888497.701| arn:aws:ecs:us-east-1:275795381673:task/0053c603-a6c9-4044-89f5-b0edc8f6de3f |
+----------------+---------------------------------------------------------------------------------+
3. Get last element sorted tasks Arns
aws ecs describe-tasks --cluster ${cluster[Name]} \
--tasks ${task_arns} \
--query "tasks[] | reverse(sort_by(#, &createdAt)) | [-1].[taskArn]" \
--output text
Should give:
arn:aws:ecs:us-east-1:275795381673:task/0053c603-a6c9-4044-89f5-b0edc8f6de3f
Assuming you work from a Linux/Unix-Shell I would suggest sending the output to the sort command. This works by adding
| sort
to your command at the end of the line:
aws ecs list-tasks --cluster {cluster} --family {family} --desired-status 'STOPPED' | sort
I hope this helps.