We recently had an issue with one of our releases and had to rollback all our services manually. While doing so we had to disable the ‘auto-sync’ feature. After reverting the faulty PR, we forgot to enable the auto-sync again and the apps were out-of-sync for a day.
Is there a way we can enable a trigger or an alert that gets triggered after every duration (say one hour) and notify us on slack that the autosync feature is disabled?
We would also like to enable the notification after every PR merge, something like "The commit can not be released because auto-sync is disabled.".
The ArgoCD API has a api/v1/applications/{app_name} endpoint. You can fetch the application's settings from there. You then need to check if the spec.syncPolicy object has a key named automated. Here is a example written in bash and jq:
#!/usr/bin/env bash
set -e
USERNAME=admin
PASSWORD=yourpassword
ARGOCD_SERVER=https://yourserver
APP_NAME=yourapp
ARGOCD_TOKEN=$(curl -s $ARGOCD_SERVER/api/v1/session -d '{"username":"admin","password":"'$PASSWORD'"}' | jq -r ".token")
response=$(curl -s "$ARGOCD_SERVER/api/v1/applications/$APP_NAME" -H "Authorization: Bearer $ARGOCD_TOKEN")
echo $response | jq '.spec.syncPolicy | has("automated")'
Related
This question already has answers here:
Tell when Job is Complete
(7 answers)
Closed 3 years ago.
I'm looking for a way to wait for Job to finish execution Successfully once deployed.
Job is being deployed from Azure DevOps though CD on K8S on AWS. It is running one time incremental database migrations using Fluent migrations each time it's deployed. I need to read pod.status.phase field.
If field is "Succeeded", then CD will continue. If it's "Failed", CD stops.
Anyone have an idea how to achieve this?
I think the best approach is to use the kubectl wait command:
Wait for a specific condition on one or many resources.
The command takes multiple resources and waits until the specified
condition is seen in the Status field of every given resource.
It will only return when the Job is completed (or the timeout is reached):
kubectl wait --for=condition=complete job/myjob --timeout=60s
If you don't set a --timeout, the default wait is 30 seconds.
Note: kubectl wait was introduced on Kubernetes v1.11.0. If you are using older versions, you can create some logic using kubectl get with --field-selector:
kubectl get pod --field-selector=status.phase=Succeeded
We can check Pod status using K8S Rest API.
In order to connect to API, we need to get a token:
https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/#without-kubectl-proxy
# Check all possible clusters, as you .KUBECONFIG may have multiple contexts:
kubectl config view -o jsonpath='{"Cluster name\tServer\n"}{range .clusters[*]}{.name}{"\t"}{.cluster.server}{"\n"}{end}'
# Select name of cluster you want to interact with from above output:
export CLUSTER_NAME="some_server_name"
# Point to the API server refering the cluster name
APISERVER=$(kubectl config view -o jsonpath="{.clusters[?(#.name==\"$CLUSTER_NAME\")].cluster.server}")
# Gets the token value
TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(#.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}"|base64 -d)
From above code we have acquired TOKEN and APISERVER address.
On Azure DevOps, on your target Release, on Agent Job, we can add Bash task:
#name of K8S Job object we are waiting to finish
JOB_NAME=name-of-db-job
APISERVER=set-api-server-from-previous-code
TOKEN=set-token-from-previous-code
#log APISERVER and JOB_NAME for troubleshooting
echo API Server: $APISERVER
echo JOB NAME: $JOB_NAME
#keep calling API until you get status Succeeded or Failed.
while true; do
#read all pods and query for pod containing JOB_NAME using jq.
#note that you should not have similar pod names with job name otherwise you will get mutiple results. This script is not expecting multiple results.
res=$(curl -X GET $APISERVER/api/v1/namespaces/default/pods/ --header "Authorization: Bearer $TOKEN" --insecure | jq --arg JOB_NAME "$JOB_NAME" '.items[] | select(.metadata.name | contains($JOB_NAME))' | jq '.status.phase')
if (res=="Succeeded"); then
echo Succeeded
exit 0
elif (res=="Failed"); then
echo Failed
exit 1
else
echo $res
fi
sleep 2
done
If Failed, script will exit with code 1 and CD will stop (if configured that way).
If Succeeded, exist with code 0 and CD will continue.
In final setup:
- Script is part of artifact and I'm using it inside Bash task in Agent Job.
- I have placed JOB_NAME into Task Env. Vars so it can be used for multiple DB migrations.
- Token and API Server address are in Variable group on global level.
TODO:
curl is not existing with code 0 if URL is invalid. It needs --fail flag, but still above line exists 0.
"Unknown" Pod status should be handled as well
Currently my team is using Jenkins to manage our CI/CD workflow. As our infrastructure is entirely in AWS I have been looking into migrating to AWS CodePipeline/CodeBuild to manage this.
In current state, we are versioning our artifacts as such <major>.<minor>.<patch>-<jenkins build #> i.e. 1.1.1-987. However, CodeBuild doesn't seem to have any concept of a build number. As artifacts are stored in s3 like <bucket>/<version>/<artifact> I would really hate to lose this versioning approach.
CodeBuild does provide a few env variables that i can see here: http://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref.html#build-env-ref-env-vars
But from what is available it seems silly to try to use the build ID or anything else.
Is there anything readily available from CodeBuild that could support an incremental build #? Or is there an AWS recommended approach to semantic versioning? Searching this topic returns remarkably low results
Any help or suggestions is greatly appreciated
The suggestion to use date wasn't really going to work for our use case. We ended up creating a base version in SSM and creating a script that runs within the buildspec that grabs, increments, and updates the version back to SSM. It's easy enough to do:
Create a String/SecureString within SSM as [NAME]. For example lets say "BUILD_VERSION". The value should be in [MAJOR.MINOR.PATCH] or [MAJOR.PATCH].
Create a shell script. The one below should be taken as a basic template, you will have to modify it to your needs:
#!/bin/bash
if [ "$1" = 'next' ]; then
version=$(aws ssm get-parameter --name "BUILD_VERSION" --region 'us-east-1' --with-decryption | sed -n -e 's/.*Value\"[^\"]*//p' | sed -n -e 's/[\"\,]//gp')
majorminor=$(printf $version | grep -o ^[0-9]*\\.[0-9]*\. | tr -d '\n')
patch=$(printf $version | grep -o [0-9]*$ | tr -d '\n')
patch=$(($patch+1))
silent=$(aws ssm put-parameter --name "BUILD_VERSION" --value "$majorminor$patch" --type "SecureString" --overwrite)
echo "$majorminor$patch"
fi
Call the versioning script from within buildspec and use the output however you need.
It may be late while I post this answer, however since this feature is not yet released by AWS this may help a few people in a similar boat.
We used Jenkins build numbers for versioning and were migrating to codebuild/code-pipeline. codebuild-id did not work for us as it was very random.
So in the interim we create our own build number in buildspec file
BUILD_NUMBER=$(date +%y%m%d%H%M%S).
This way at least we are able to look at the id and know when it was deployed and have some consistency in the numbering.
So in your case, it would be 1.1.1-181120193918 instead of 1.1.1-987.
Hope this helps.
CodeBuild supports semantic versioning.
In the configuration for the CodeBuild project you need to enable semantic versioning (or set overrideArtifactName via the CLI/API).
Then in your buildspec.yml file specify a name using the Shell command language:
artifacts:
files:
- '**/*'
name: myname-$(date +%Y-%m-%d)
Caveat: I have tried lots of variations of this and cannot get it to work.
I though this would be easy but I cannot manage to find a way to get the revision string from a post deploy hook on EBS. The use case is straightforward: I want to warn rollbar of a deploy.
Here is the current script :
# Rollbar deploy notifier
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/90_notify_rollbar.sh":
mode: "000755"
content: |
#!/bin/bash
. /opt/elasticbeanstalk/support/envvars
LOCAL_USERNAME=`whoami`
REVISION=`date +%Y-%m-%d:%H:%M:%S`
curl https://api.rollbar.com/api/1/deploy/ \
-F access_token=$ROLLBAR_KEY \
-F environment=$RAILS_ENV \
-F revision=$REVISION \
-F local_username=$LOCAL_USERNAME
So far I'm using the current date as revision number, but that isn't really helpful. I tried using /opt/elasticbeanstalk/bin/get-config but I couldn't find anything relevant in the environment and container section, and couldn't read anything from meta. Plus, I found no doc about those, so...
Ideally, I would also like the username of the deployer, not the one on the local machine, but that would be the cherry on the cake.
Thanks for your time !
You can update your elastic beanstalk instance profile role (aws-elasticbeanstalk-ec2-role) to allow it to call Elastic Beanstalk APIs. In the post deploy hook you can call DescribeEnvironments with the current environment name using the aws cli or any of the AWS SDKs.
Let me know if you have any more questions about this or if this does not work for you.
I'm also looking for an easy alternative for API. For now I use bash
eb deploy && curl https://api.rollbar.com/api/1/deploy/ -F access_token=xxx -F environment=production -F revision=`git rev-parse --verify HEAD` -F rollbar_username=xxx
Replace xxx with your token and username
I did not know where else to post this as there are Informatica questions on this site. But at present, in order to abort workflows in Informatica, I must right click each workflow and manually select abort:
Given I have several hundred workflows active at any one time, to then abort them all manually takes a very long time.
Is there a programmatic or pre-defied GUI to allow for aborting all workflows at once?
You could dump the Integration Service. That would abort all workflows.
Otherwise you'd need to create a script to abort the workflows with pmcmd.
You can list all workflow names in a file (or get the workflow list from repository using repository query) and pass each workflow to the below pmcmd command which will automatically abort it using shell script.
pmcmd abortworkflow -service informatica-integration-Service -d domain-name -u user-name -p password -f folder-name -w workflow-name
pmcmd aborttask -service informatica-integration-Service -d domain-name -u user-name -p password -f folder-name -w workflow-name task-name
Let me know if you need any additional help.
I have a crontab that includes a MAILTO=my.email#example.com. My server uses msmtp to forward the email to Amazon Simple Email Service. My problem is that output from cron commands never arrives in my mailbox. This is what the msmtp log says:
Mar 06 14:26:02 host=email-smtp.us-east-1.amazonaws.com tls=on auth=on user=MY.SES.USER from=my.email#example.com recipients=my.email#example.com smtpstatus=554 smtpmsg='554 Transaction failed: User name is missing: ?Cron Daemon ?.' errormsg='the server did not accept the mail' exitcode=EX_UNAVAILABLE
What do I need to do in order to make Amazon SES accept the cron emails?
The suggested solution from the AWS Developer Forums:
It turns out that cron has the "from" address hard-coded in the source (q.v. "do_command.c" in the cron source), so one does not have influence over what cron transmits to sendmail (which in our case is symlinked to "/usr/bin/msmtp").
However, due to the magic of Linux, we do have the ability to alter the stream of text that goes into sendmail.
The way I worked around this cron limitation was to move the "msmtp" binary to"msmtp.bin" and then create "/usr/bin/msmtp" that was a shell script:
#! /bin/bash
sed -e 's/root .Cron Daemon./user#example.com/' | /usr/bin/msmtp.bin "$#"
This is also, AFAIK, the only means one has to set the "debug" flag to msmtp when used in a "global" setting (such as cron, or other cases where sendmail is invoked with arguments you don't control).
While the script above is rather simplistic, you can also conditionally alter the text by checking the input arguments for the magic "-FCronDaemon" which is also hard coded in the cron binary. I would be stunned if any other program calls sendmail with "-FCronDaemon".
Had exact same problem but the answer above didn't work for me so here's what I did...
As the problem is isolated to cron, and symlinking sendmail to msmtp works everywhere else, I didn't want to change the msmtp command globally for everything. So I first created /usr/bin/msmtp_cron.bin and made it executable.
Next I had to tell cron to use this as its mail path, by editing /etc/sysconfig/crond to read:
CRONDARGS="-m '/usr/bin/msmtp_cron.bin -t'"
Then not forgetting to restart crond (reloading alone isn't enough):
$ sudo systemctl restart crond.service
Back to /usr/bin/msmtp_cron.bin, in this file, I first had to find out what cron was actually streaming to sendmail/msmtp to know what substitution I could make:
sed '' > /tmp/cron-mail-capture.txt
This yielded these headers:
From: "(Cron Daemon)" <root>
It's slightly different from the other answer so my sed script ended up looking like so:
sed -e 's/..Cron Daemon.* <root>/me#mydomain.com/' | /usr/bin/msmtp -t "$#"
Now it successfully sends cron messages via AWS SES to my email account.