I have a question regarding ArgoCD. When using argocd app sync to deploy our helm charts, we have a weird issue which we have to do this manually for changes to initContainers. The scenario is like this:
We make a change to a Deployment service to remove initContainer or edit the initContainer.
If we are to use helm upgrade --install, it works as intended; however, when using argocd app sync, the modifications do not occur on the pods that were updated.
The manual process is for me to go into the Deployment and remove the change of what was to occur.
I am sure this is a simple thing on my side but I would appreciate any feedback.
Related
I am rebuilding a system like gitlab where users can configure pipelines with individual jobs running on AWS ECS (Fargate).
One important functionality is donwloading and uploading of artefacts (files) generated by these jobs.
I want to solve this by running a sidecar with the logic responsible for the artefacts next to the actual code running the logic of the job.
one important requirement: it needs to be assumed that the "main" container runs custom code which i cannot control.
It seems however there is no clean solution in kubernetes for starting a pod with this order of containers:
start a sidecar, trigger download of artefacts
upon completion of artefacts download, start the logic in the main container alongside sidecar and run it to finish
upon finish of main container start upload of new artefacts and end sidecar.
Any suggestion is welcome
Edit:
I found the attribute of container dependencies on AWS ECS and will try it out now: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/example_task_definitions.html#example_task_definition-containerdependency
I have an ECS task which has 2 containers using 2 different images, both hosted in ECR. There are 2 GitHub repos for the two images (app and api), and a third repo for my IaC code (infra). I am managing my AWS infrastructure using Terraform Cloud. The ECS task definition is defined there using Cloudposse's ecs-alb-service-task, with the containers defined using ecs-container-definition. Presently I'm using latest as the image tag in the task definition defined in Terraform.
I am using CircleCI to build the Docker containers when I push changes to GitHub. I am tagging each image with latest and the variable ${CIRCLE_SHA1}. Both repos also update the task definition using the aws-ecs orb's deploy-service-update job, setting the tag used by each container image to the SHA1 (not latest). Example:
container-image-name-updates: "container=api,tag=${CIRCLE_SHA1}"
When I push code to the repo for e.g. api, a new version of the task definition is created, the service's version is updated, and the existing task is restarted using the new version. So far so good.
The problem is that when I update the infrastructure with Terraform, the service isn't behaving as I would expect. The ecs-alb-service-task has a boolean called ignore_changes_task_definition, which is true by default.
When I leave it as true, Terraform Cloud successfully creates a new version whenever I Apply changes to the task definition. (A recent example was to update environment variables.) BUT it doesn't update the version used by the service, so the service carries on using the old version. Even if I stop a task, it will respawn using the old version. I have to manually go in and use the Update flow, or push changes to one of the code repos. Then CircleCI will create yet aother version of the task definition and update the service.
If I instead set this to false, Terraform Cloud will undo the changes to the service performed by CircleCI. It will reset the task definition version to the last version it created itself!
So I have three questions:
How can I get Terraform to play nice with the task definitions created by CircleCI, while also updating the service itself if I ever change it via Terraform?
Is it a problem to be making changes to the task definition from THREE different places?
Is it a problem that the image tag is latest in Terraform (because I don't know what the SHA1 is)?
I'd really appreciate some guidance on how to properly set up this CI flow. I have found next to nothing online about how to use Terraform Cloud with CI products.
I have learned a bit more about this problem. It seems like the right solution is to use a CircleCI workflow to manage Terraform Cloud, instead of having the two services effectively competing with each other. By default Terraform Cloud will expect you to link a repo with it and it will auto-plan every time you push. But you can turn that off and use the terraform orb instead to run plan/apply via CircleCI.
You would still leave ignore_changes_task_definition set to true. Instead, you'd add another step to the workflow after the terraform/apply step has made the change. This would be aws-ecs/run-task, which should relaunch the service using the most recent task definition, which was (possibly) just created by the previous step. (See the task-definition parameter.)
I have decided that this isn't worth the effort for me, at least not at this time. The conflict between Terraform Cloud and CircleCI is annoying, but isn't that acute.
I'm trying to update an existing environment to immutable updates to work around flaky deployments as described by this (not my) blog post.
After reading the docs - specifically the "The aws:autoscaling:updatepolicy:rollingupdate namespace" section, it appears I should be able to add an .ebextensions folder at the same level as my .elasticbeanstalk folder, and put the following in an arbitrarily-named config file:
option_settings:
aws:autoscaling:updatepolicy:rollingupdate:
RollingUpdateType: Immutable
and issue an eb deploy.
Having done this and receiving an "Environemnt update completed successfully" result, I went to check the aws console and this environment is still set to "Deployment policy: Rolling." An eb config shows aws:autoscaling:updatepolicy:rollingupdate:RollingUpdateType is set to "Health" (which doesn't make much sense but I guess that's a labeling issue so, whatever).
Will this just not work or am I doing something obviously stupid?
Pedantic Note - I'm aware I can do this by hand editing eb config and checking a box in the console, but I want to automate the change so we can apply it to other environments. ;)
It seems you need a little more.
option_settings:
aws:autoscaling:updatepolicy:rollingupdate:
RollingUpdateType: Immutable
aws:elasticbeanstalk:command:
DeploymentPolicy: Immutable
What you did enables immutable updates for configuration changes only. For deployments you need the last two lines from the code above.
You can see more here at Example .ebextensions/immutable-all.config.
Up until today, my deploy process has worked fine. Today when I go to deploy a new revision, I get stuck at the Deploying... text with a spinning indicator, and it says One or more of the referenced revisions does not yet exist or is deleted. I've tried a number of different images and flags -- all the same.
See Viewing the list of revisions for a service, in order to undo whatever you may have done.
Probably you have the wrong project selected, if it does not know any of the revisions.
I know I provided scant information, but just to follow up with an answer: it looks like the issue was that I was deploying a revision, and then immediately trying to tag it using gcloud alpha run services update-traffic <service_name> --set-tags which looks to have caused some sort of race, where it complained that the revision was not yet deployed, and would hang indefinitely. Moving the set-tag into the gcloud alpha run deploy seemed to fix it.
The GCP command appcfg has been deprecated. appcfg used to have appcfg rollback to be used when there is a failed deployment.
What is its equivalent for gcloud (the new command)? I can't find it in Google GCP documentation.
More context:
Rolling back in appcfg was not meant for managing the traffic and going back to the previous version. It was used to remove the lock on your deploy.
If you had an unsuccessful deployment, you were not able to deploy any more. appcfg rollback was used to remove that lock and make it available for you to deploy again.
I think there is no direct command to appcfg rollback. However, I would highly recommend you to consider the Splitting the traffic option.
This will let you redirect the traffic from one of your versions to another, even between old versions of your service.
Let's imagine this:
You have version 1 of your service and it works just fine.
A couple weeks later you decide to deploy a new the version: version 2
However, the deploy fails and your app is completely down. You are loosing users and money. Everything is on fire.
You can easily switch the traffic to the trusty version 1 by redirecting 100% of the traffic to it.
Version 2 is out of the game until you deploy a new version.
The advantage of this is that you don't have to wait until the rollback is done. The traffic is automatically redirected to an old version. Additionally, it has the gcloud set traffic command for you to run it via the CLI.
Hope this is helpful!