How to collect a grep and use it in a aws configset - amazon-web-services

In my aws Cloud Formation cfn configset I have a command to set an environment key to the name of the user group apache belongs to as it might be apache or www-data depending on the distro.
Something like this:
Metadata:
AWS::CloudFormation::Init:
configSets:
joomla:
- "set_permissions"
- "and_some_more..."
configure_cfn:
files:
/etc/cfn/hooks.d/cfn-auto-reloader.conf:
content: !Sub |
[cfn-auto-reloader-hook]
triggers=post.update
path=Resources.EC2.Metadata.AWS::CloudFormation::Init
action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource EC2 --configsets joomla --region ${AWS::Region}
mode: "000400"
owner: root
group: root
.....
set_permissions:
commands:
01_01_get_WebServerGroup:
env:
#webserver group might be apache or www-data depending on the distro
WebServerGp:
command: "ps -ef | egrep '(httpd|apache2|apache)' | grep -v `whoami` | grep -v root | head -n1 | awk '{print $1}'"
However, when I launch this stack, the configsets process halts at this point and I get an an error in the cfn_init.log that looks like this:
File
"/usr/lib/python2.7/dist-packages/cfnbootstrap/command_tool.py", line
80, in apply
raise ToolError(u"%s does not specify the 'command' attribute, which is required" % name) ToolError: 01_01_get_WebServerGroup does
not specify the 'command' attribute, which is required
Is this the preferred method to catch and use a grep result in a configset command? Is there a better way? What can I do to address the error thrown in the cfn_init.log?
OK, I guess I can create parameter and mapping elements to capture the distro type on launch and then set the webserver group accordingly but I am really trying to understand how to set the env: key to a response from the cli.

The problem of your code is this line WebServerGp.
Line command is must be on the same level of env, under the commands name, in your case is 01_01_get_WebServerGroup. So, it has to be like this:
commands:
01_01_get_WebServerGroup:
env: ..
command: ..
If you want to use the result of grep, you can put them on variable and use it later.
You can specify more than one command under that command line using \n for executing the command.
Please check this code below.
command: "result=(ps ef | grep ...)\n echo $result\n ..."
If you have really long command, you can use the Fn::Join to be the value of command.

Related

How to display constant values using custom-columns format of kubectl?

I have multiple clusters and I want to check which ingresses do not specify explicit certificate. Right now I use the following command:
~$ k config get-contexts -o name | grep -E 'app(5|3)41.+-admin' | xargs -n1 -I {} kubectl --context {} get ingress -A -o 'custom-columns=NS:{.metadata.namespace},NAME:{.metadata.name},CERT:{.spec.tls.*.secretName}' | grep '<none>'
argocd argo-cd-argocd-server <none>
argocd argo-cd-argocd-server <none>
reference-app reference-app-netcore-ingress <none>
argocd argo-cd-argocd-server <none>
argocd argo-cd-argocd-server <none>
test-ingress my-nginx <none>
~$
I want to improve the output by including the context name, but I can't figure out how to modify the custom-columns format to do that.
The below command would Not yield the exact desired output, but it will be close. using jsonpath, it's possible:
kubectl config get-contexts -o name | xargs -n1 -I {} kubectl get ingress -A -o jsonpath="{range .items[*]}{} {.metadata.namespace} {.metadata.name} {.spec.tls.*.secretName}{'\n'}{end}" --context {}
If the exact output is needed, then the kubectl output needs to be looped in the bash loop. Example:
kubectl config get-contexts -o name | while read context; do k get ingress -A -o 'custom-columns=NS:{.metadata.namespace},NAME:{.metadata.name},CERT:{.spec.tls.*.secretName}' --context "$context" |awk -vcon="$context" 'NR==1{$0=$0FS"CONTEXT"}NR>1{$0=$0 FS con}1'; done |column -t
NS NAME CERT CONTEXT
default tls-example-ingress testsecret-tls kubernetes-admin-istio-demo.local#istio-demo.local
default tls-example-ingress1 testsecret-tls kubernetes-admin-istio-demo.local#istio-demo.local
default tls-example-ingress2 <none> kubernetes-admin-istio-demo.local#istio-demo.local
To perform post-processing around the header and context, the awk command was used. Here is some details about it:
Command:
awk -vcon="$context" 'NR==1{$0=$0FS"CONTEXT"}NR>1{$0=$0 FS con}1'; done |column -t
-vcon="$context": This is to create a variable called con inside awk to store the value of bash variable($context).
NR==1: Here NR is the record number(in this case line number) and $0 means record/line.
NR==1{$0=$0FS"CONTEXT"}: This means, on the 1st line, reset the line to itself followed by FS(default is space) followed by a string "CONTEXT".
Similarly, NR>1{$0=$0 FS con} means, from the 2nd line onwards, append the line with FS followed by con.
1 in the end is the tell awk to do the print.

Error of the Zabbiks item when monitoring the ssl aws ALB

It is necessary with the help of Zabbiks to enable monitoring of SSL count licenses for ALB in aws.
Zabbiks can't recognize the item
UserParameter = ssl.count, aws elbv2 describe-listener-certificates --listener-arn --profile ******** arn: aws: elasticloadbalancing: eu-central - ******* ****** 1dcfc52e | jq '.Certificates | . []. CertificateArn '| wc -l
zabbix writes
Value of type "string" is not suitable for value type "Numeric (unsigned)".Value "The config profile (****) could not be found0"
If I change the Type of information to text, then everything works except for the triggers, which should return how many SSL licenses are in the ALB,
output from zabbix_agentd -t ssl.count command ssl.count [t | 26] everything works from the console, all credential are fine.
ABL in zabbix declared in macros {$ HOST.NAME} main-devs - *******. Eu-central-1.elb.amazonaws.com
Tell me who came across, what could it be?
The problems is that you are not redirecting the standard error, and your output is not a number. Since it's not a number, is not valid in a Numeric item.
Example:
$ ls -l temp.sh nonexisting.sh
ls: cannot access nonexisting.sh: No such file or directory
-rw-r--r-- 1 myuser domain users 14918 Oct 15 2019 temp.sh
$ ls -l temp.sh nonexisting.sh | wc -l
ls: cannot access nonexisting.sh: No such file or directory
1
$ ls -l temp.sh nonexisting.sh 2>/dev/null | wc -l
1
With your code:
UserParameter = ssl.count, aws elbv2 describe-listener-certificates --listener-arn --profile ******** arn: aws: elasticloadbalancing: eu-central - ******* ****** 1dcfc52e 2>/dev/null | jq '.Certificates | . []. CertificateArn '| wc -l

Ansible - Replace " and \r\ in a variable

I want to replace the " and \r\ from a variable content using Ansible.
I have the following data in a variable result thatI register the output to the variable from the previous task
curl -s -H \"Authorization: JWT eyJ4NWMiOlsiTUlJQytqQ0NBHuHO96csEQ\r\" https://hub.docker.com/v2/repositories/talasecurityinc/?page_size=10000 | jq -r '.results|.[]|.name'
In the above content I want to replace the \ and \r\ with null.
I have tried the below way but it doesn't work for me.
- set_fact: final_out="{{result | replace('\', "") | replace('\r\', '')}}"
The expected output is
curl -s -H "Authorization: JWT eyJ4NWMiOlsiTUlJQytqQ0NBHuHO96csEQ" https://hub.docker.com/v2/repositories/talasecurityinc/?page_size=10000 | jq -r '.results|.[]|.name'
The example playbook snippet would be helpful for me since I am new to ansible.
Escaping Hell....
I was not able to use replace, probably because I didn't try hard/smart enough. Meanwhile, in your specific case, you can achieve the expected result with a single regex_replace filter call so it was easier (and it worked right away :)).
I used yaml folded blocks (>) with white space control (-) to minimize the escape hassle. If you don't know what that is, have a look at a yaml doc (learn yaml in y minutes is my favorite one)
Note that the remaining backslashes in the last result below are added by ansible to escape the double quotes in the output.
---
- name: Escape chars
hosts: localhost
gather_facts: false
vars:
test: >-
curl -s -H \"Authorization: JWT eyJ4NWMiOlsiTUlJQytqQ0NBHuHO96csEQ\r\"
https://hub.docker.com/v2/repositories/talasecurityinc/?page_size=10000
| jq -r '.results|.[]|.name'
tasks:
- name: Show the untouched var
debug:
var: test
- name: Escape the var as intended
debug:
msg: >-
{{ test | regex_replace('\\r?\\?', '') }}
which results in
PLAY [Escape chars] ********************************************************************
TASK [Show the untouched var] **********************************************************
ok: [localhost] => {
"test": "curl -s -H \\\"Authorization: JWT eyJ4NWMiOlsiTUlJQytqQ0NBHuHO96csEQ\\r\\\" https://hub.docker.com/v2/repositories/talasecurityinc/?page_size=10000 | jq -r '.results|.[]|.name'"
}
TASK [Escape the var as intended] ******************************************************
ok: [localhost] => {
"msg": "curl -s -H \"Authorization: JWT eyJ4NWMiOlsiTUlJQytqQ0NBHuHO96csEQ\" https://hub.docker.com/v2/repositories/talasecurityinc/?page_size=10000 | jq -r '.results|.[]|.name'"
}
PLAY RECAP *****************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0

Environment Variables in newest AWS EC2 instance

I am trying to get ENVIRONMENT Variables into the EC2 instance (trying to run a django app on Amazon Linux AMI 2018.03.0 (HVM), SSD Volume Type ami-0ff8a91507f77f867 ). How do you get them in the newest version of amazon's linux, or get the logging so it can be traced.
user-data text (modified from here):
#!/bin/bash
#trying to get a file made
touch /tmp/testfile.txt
cat 'This and that' > /tmp/testfile.txt
#trying to log
echo 'Woot!' > /home/ec2-user/user-script-output.txt
#Trying to get the output logged to see what is going wrong
exec > >(tee /var/log/user-data.log|logger -t user-data ) 2>&1
#trying to log
echo "XXXXXXXXXX STARTING USER DATA SCRIPT XXXXXXXXXXXXXX"
#trying to store the ENVIRONMENT VARIABLES
PARAMETER_PATH='/'
REGION='us-east-1'
# Functions
AWS="/usr/local/bin/aws"
get_parameter_store_tags() {
echo $($AWS ssm get-parameters-by-path --with-decryption --path ${PARAMETER_PATH} --region ${REGION})
}
params_to_env () {
params=$1
# If .Ta1gs does not exist we assume ssm Parameteres object.
SELECTOR="Name"
for key in $(echo $params | /usr/bin/jq -r ".[][].${SELECTOR}"); do
value=$(echo $params | /usr/bin/jq -r ".[][] | select(.${SELECTOR}==\"$key\") | .Value")
key=$(echo "${key##*/}" | /usr/bin/tr ':' '_' | /usr/bin/tr '-' '_' | /usr/bin/tr '[:lower:]' '[:upper:]')
export $key="$value"
echo "$key=$value"
done
}
# Get TAGS
if [ -z "$PARAMETER_PATH" ]
then
echo "Please provide a parameter store path. -p option"
exit 1
fi
TAGS=$(get_parameter_store_tags ${PARAMETER_PATH} ${REGION})
echo "Tags fetched via ssm from ${PARAMETER_PATH} ${REGION}"
echo "Adding new variables..."
params_to_env "$TAGS"
Notes -
What i think i know but am unsure
the user-data script is only loaded when it is created, not when I stop and then start mentioned here (although it also says [i think outdated] that the output is logged to /var/log/cloud-init-output.log )
I may not be starting the instance correctly
I don't know where to store the bash script so that it can be executed
What I have verified
the user-data text is on the instance by ssh-ing in and curl http://169.254.169.254/latest/user-data shows the current text (#!/bin/bash …)
What Ive tried
editing rc.local directly to export AWS_ACCESS_KEY_ID='JEFEJEFEJEFEJEFE' … and the like
putting them in the AWS Parameter Store (and can see them via the correct call, I just can't trace getting them into the EC2 instance without logs or confirming if the user-data is getting run)
putting ENV variables in Tags and importing them as mentioned here:
tried outputting the logs to other files as suggested here (Not seeing any log files in the ssh instance or on the system log)
viewing the System Log on the aws webpage to see any errors/logs via selecting the instance -> 'Actions' -> 'Instance Settings' -> 'Get System Log' (not seeing any commands run or log statements [only 1 unrelated word of user])

Script execution using ansible [duplicate]

I am using Ansible to deploy my project and I trying to check if an specified package is installed, but I have a problem with it task, here is the task:
- name: Check if python-apt is installed
command: dpkg -l | grep python-apt
register: python_apt_installed
ignore_errors: True
And here is the problem:
$ ansible-playbook -i hosts idempotent.yml
PLAY [lxc-host] ***************************************************************
GATHERING FACTS ***************************************************************
ok: [10.0.3.240]
TASK: [idempotent | Check if python-apt is installed] *************************
failed: [10.0.3.240] => {"changed": true, "cmd": ["dpkg", "-l", "|", "grep", "python-apt"], "delta": "0:00:00.015524", "end": "2014-07-10 14:41:35.207971", "rc": 2, "start": "2014-07-10 14:41:35.192447"}
stderr: dpkg-query: error: package name in specifier '|' is illegal: must start with an alphanumeric character
...ignoring
PLAY RECAP ********************************************************************
10.0.3.240 : ok=2 changed=1 unreachable=0 failed=0
Why is illegal this character '|' .
From the doc:
command - Executes a command on a remote node
The command module takes the command name followed by a list of
space-delimited arguments. The given command will be executed on all
selected nodes. It will not be processed through the shell, so
variables like $HOME and operations like "<", ">", "|", and "&" will
not work (use the shell module if you need these features).
shell - Executes a commands in nodes
The shell module takes the command name followed by a list of space-delimited arguments.
It is almost exactly like the command module but runs the command
through a shell (/bin/sh) on the remote node.
Therefore you have to use shell: dpkg -l | grep python-apt.
read about the command module in the Ansible documentation:
It will not be processed through the shell, so .. operations like "<", ">", "|", and "&" will not work
As it recommends, use the shell module:
- name: Check if python-apt is installed
shell: dpkg -l | grep python-apt
register: python_apt_installed
ignore_errors: True
For what it's worth, you can check/confirm the installation in a debian environment using the apt command:
- name: ensure python-apt is installed
apt: name=python-apt state=present