I'm trying to run scripts as part of some of my deployment pipelines in Spinnaker. I don't want to use Jenkins to run these scripts. I would use a Kubernetes job, but these scripts need to execute prior to the Kubernetes deployment.
I was debating creating ECS tasks in AWS which I'd like to run on demand during one of the stages in my pipeline. Does anyone know if it's possible to execute an ECS task directly from Spinnaker?
If not, are there any other ways to execute a command or script in a pipeline outside of using a Kubernetes job or Jenkins server?
One way to do this is to use the Run Job ( Manifest ) stage and just point it to another kubernetes cluster for this. This approach gives you a bit of flexibility since you can monitor the pipeline stage for completion status.
You can also just create an arbitrary API endpoint and trigger via a webhook stage that monitors for completion and use whatever your preferred script execution environment (i.e, Lambda, ECS etc) behind that api endpoint.
Related
I have below 2 requirements can you please help share any suggestion
AWS Batch trigger a ECR Fargate ( On demand )
AWS Batch trigger a Spring App deployed in ECR ( running permanently)
SO here Option 1, I need to start a spring boot app which should start in ECR Fargate. this I understood from AWS batch we can specify the Cluster of the Fargate so that when the AWS batch run the app will get started.
For Option 2, I have a spring boot App deployed in ECR Fargate and it will be running, and inside spring batch is there, Now AWS batch need to trigger the spring batch. it is possible if so can you please share the implementation sample.
Also from my client App or program I need to update the AWS batch, saying the job is success or failure. can you share any sample for those as well.
AWS Batch only executes ECS tasks not ECS services. For option 1 - to launch a container (your app that does the work you want) within ECS Fargate, you would need to specify an AWS Batch compute environment as Fargate, a job queue that references the compute environment, and a job definition of the task (what container to run, what command to send, what CPU and memory resources are required). See the Learn AWS Batch workshop or the AWS Batch Getting Started documentation for more information on this.
For option 2 - AWS Batch and Spring Batch are orthogonal solutions. You should just call the Spring Batch API endpoint directly OR rely on AWS Batch. Using both is not recommended unless this is something you don't have control over.
But to answer your question - calling an non-AWS API endpoint is handled in your container and application code. AWS Batch does not prevent this but you would need to make sure that the container has secure access to the proper credentials to call the Spring boot app. Once your Batch job calls the API you have two choices:
Immediately exit and track the status of the spring batch operations elsewhere (i.e. The Batch job is to call the API and SUCCESS = "able to send the API request successfully, FAIL = "not able to call the API" )
Call the API, then enter a loop where you poll the status of the Spring batch job until it completes successfully or not, exiting the AWS Batch job with the same state as the Spring batch job did.
I have an ECS service and within that ECS service, I want to boot up 3 tasks all from the same task definition. I need each of these tasks to be on a separate EC2 instance, this seems simple enough however I want to pass a different command to each one of the running tasks to specify where their config can be found and some other options via the CLI within my running application.
For example for task 1 I want to pass run-node CONFIG_PATH="/tmp/nodes/node_0 and task 2 run-node CONFIG_PATH="/tmp/nodes/node_1" --bootnode true and task 3 run-node CONFIG_PATH="/tmp/nodes/node_0 --http true"
I'm struggling to see how I can manage individual task instances like this within a single service using Terraform, it seems really easy to manage multiple instances that are all completely equal but I can't find a way to pass custom overrides to each task that are all running off the same task definition.
I am thinking this may be a job for a different dev-ops automation tool but would love to carry on doing it in Terraform if possible.
This is not a limitation of a terraform. This is how ECS service works - runs exact copies of same task definition. Thus, you can't customize individual tasks in an ECS service as all these tasks are meant to be identical, interchangeable and disposable.
To provide overwrites you have to run the tasks outside of a service, which you can do using run-task or start-task with --overrides of AWS CLI or equivalent in any AWS SDK. Sadly there is no equivalent for that in terraform, except running local-exec with AWS CLI.
I am trying to have run the lastest task definition image built from GitHub deployment (CD). Seems like on AWS it creates a task definition for example "task-api:1", "task-api:2", on was my cluster is still running task-api: 1 even though there is the latest task as a new image has been built. So far I have to manually stop the old one and start a new one . How can I have it automated?
You must wrap your tasks in a service and use rolling updates for automated deployments.
When the rolling update (ECS) deployment type is used for your service, when a new service deployment is started the Amazon ECS service scheduler replaces the currently running tasks with new tasks.
Read: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-ecs.html
This is DevOps, so you need a CI/CD pipeline that will do the rolling updates for you. Look at CodeBuild, CodeDeploy and CodePipeline (and CodeCommit if you integrate your code repository in AWS with your CI/CD)
Read: https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-ecs-ecr-codedeploy.html
This is a complex topic, but it pays off in the end.
Judging from what you have said in the comments:
I created my task via the AWS console, I am running just the task definition on its own without service plus service with task definition launched via the EC2 not target both of them, so in the task definition JSON file on my Github both repositories they are tied to a revision of a task (could that be a problem?).
It's difficult to understand exactly how you have this set up and it'd probably be a good idea for you to go back and understand the services you are using a little better using the guide you are following or AWS documentation. Pushing a new task definition does not automatically update services to use the new definition.
That said, my guess is that you need to update the service in ECS to use the latest task definition. You can do that in many ways:
Through the console (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service-console-v2.html).
Through the CLI (https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html).
Through the IaC like the CDK (https://docs.aws.amazon.com/cdk/api/latest/docs/aws-ecs-readme.html).
This can be automated but you would need to set up a process to automate it.
I would recommend reading some guides on how you could automate deployment and updates using the CDK. Amazon provide a good guide to get you started https://docs.aws.amazon.com/cdk/latest/guide/ecs_example.html.
Currently have some task-based automation for ECS that run on a scheduled basis, however sometimes there is a need to run only run task or re-run tasks for only a certain kinds of tasks (for example sql tasks or datadog tasks).
I know this can be done via console, but it's inefficient. Was thinking of a bash script that calls to start a task from a CLI. Currently I know I can do this with the AWS CLI using '--task-definition', but it's not much better. I don't usually write scripts, so I'm basically here to help with brainstorming. I'm wondering if there is a way to make an API call to start tasks. Would I need to type in the ARN every time? Can I just list the tasks on the AWS CLI and have the exported to the script? Would network-config need to be hard-coded?
Thanks!
The AWS API calls to start a task are:
StartTask:
Starts a new task from the specified task definition on the specified container instance or instances.
RunTask:
Starts a new task using the specified task definition. You can allow Amazon ECS to place tasks for you, or you can customize how Amazon ECS places tasks using placement constraints and placement strategies.
Since this is AWS API calls, there are equivalent calls in CLI and SDK.
Currently working on an environment requirement where we are to push the same file out to multiple EC2 instances running Windows on a scheduled interval. As it stands now, I see a few options and have tried each:
Windows Task Manager: run a basic task on a set schedule invoking the S3 Sync CLI tool
Cons I can see here include: setting up the task on each EC2 instance (there are many).
Lambda: scheduled lambda job that utilizes SSM to run commands on each server in a resource group
Cons: introducing another layer required to execute this task.
Run Command: using an AWS-RunRemoteScript document, run the script (stored in S3) bucket on target instances.
Cons: I'm not positive you can automate these commands on a schedule without adding another layer.
What is the most scalable path forward? Thanks in advance for your help.
Using the Run Command feature of AWS Systems Manager together with either the Maintenance Window feature of AWS Systems Manager or using CloudWatch Events to schedule the execution of Run Command should be useful here.
If you also tag instances appropriately, you can use the tag targeting feature of Run Command to ensure that all instances run the command (including new instances launched in the future as long as they are tagged).
/Mats