Scale-in Concourse workers without killing tasks - amazon-web-services

Question

Each worker runs multiple tasks. If we have a lot of tasks we'll need multiple workers. In order to save resources we'd like to elastically scale workers in and out, according to supply (spare capacity) and demand (pending tasks).
Scaling out is easy: add more nodes, they register themselves with TSA and start working.
Scaling in is trickier: one needs to wait for its tasks to finish before killing a worker's instance. Otherwise they'll have to restart on another worker. That's fine for small tasks but for longer ones that might not be acceptable.
One possible solution on AWS would be to use Autoscaling Lifecycle Hooks to synchronously tell the worker to not accept any more tasks and return when all are finished, then kill it. The Concourse Worker API doesn't have any such operation though.
Is there a way to implement safe scaling in of Concourse workers?
If the answer is "don't worry, Bosh will take care of it" I'd like to know what those mechanics are as I probably won't be using it.

Answer 1

You have to use the concourse binary from the command-line, on the host that runs the ATC (which is the concourse scheduler and web interface):
concourse --help
Usage:
concourse [OPTIONS] <command>
Application Options:
-v, --version Print the version of Concourse and exit [$CONCOURSE_VERSION]
Help Options:
-h, --help Show this help message
Available commands:
land-worker Safely drain a worker's assignments for temporary downtime.
retire-worker Safely remove a worker from the cluster permanently.
web Run the web UI and build scheduler.
worker Run and register a worker.
So it looks like you could hook to the Autoscaling Lifecycle service something that calls land-worker and then retire-worker (not sure whether retire-worker would be enough), once you figure out which worker you want to spin down...
When you spin the same worker back, you might have to be careful with the worker name, I seems to remember that sometimes the ATC gets confused, you will have to experiment with that (whether you can keep the same name or change it).

Answer 2

You can create a Lifecyle hook on your Concourse worker ASG:
Type: AWS::AutoScaling::LifecycleHook
Properties:
AutoScalingGroupName: !Ref ConcourseWorkerASG
DefaultResult: CONTINUE / ABANDON
HeartbeatTimeout: 900 # 15 minutes for example
LifecycleHookName: lchname
LifecycleTransition: "autoscaling:EC2_INSTANCE_TERMINATING"
Use a script to retire the worker, something along the lines of
lch.sh
#!/bin/bash
TYPE=$(cat /opt/concourse/type)
tsa_host=zz
instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id/)
lifecycleState=$(aws autoscaling describe-auto-scaling-instances --instance-ids $instance_id --query 'AutoScalingInstances[0].LifecycleState' --output text --region eu-west-1)
if [ "$TYPE" == "worker" ]; then
if [ "$lifecycleState" == "Terminating:Wait" ]; then
asg=$(aws autoscaling describe-auto-scaling-instances --instance-ids $instance_id --query 'AutoScalingInstances[0].AutoScalingGroupName' --output text --region eu-west-1)
/opt/concourse/concourse/bin/concourse retire-worker \
--name $HOSTNAME \
--tsa-host ${tsa_host}:2222 \
--tsa-public-key some_tsa_host_key.pub \
--tsa-worker-private-key some_worker_key
sleep 5m
systemctl stop your_concourse_service
aws autoscaling complete-lifecycle-action \
--instance-id $instance_id \
--auto-scaling-group-name $asg \
--lifecycle-hook-name "lchname" \
--lifecycle-action-result "CONTINUE" \
--region eu-west-1
fi
fi
then schedule a cronjob, for example via Ansible:
- name: List lch.sh as cronjob
cron:
name: "check asg lch for retiring the worker"
minute: "*/5" # run every 5 minutes
job: "/opt/concourse/lch.sh"