Ansible - Replace " and \r\ in a variable - replace

I want to replace the " and \r\ from a variable content using Ansible.
I have the following data in a variable result thatI register the output to the variable from the previous task
curl -s -H \"Authorization: JWT eyJ4NWMiOlsiTUlJQytqQ0NBHuHO96csEQ\r\" https://hub.docker.com/v2/repositories/talasecurityinc/?page_size=10000 | jq -r '.results|.[]|.name'
In the above content I want to replace the \ and \r\ with null.
I have tried the below way but it doesn't work for me.
- set_fact: final_out="{{result | replace('\', "") | replace('\r\', '')}}"
The expected output is
curl -s -H "Authorization: JWT eyJ4NWMiOlsiTUlJQytqQ0NBHuHO96csEQ" https://hub.docker.com/v2/repositories/talasecurityinc/?page_size=10000 | jq -r '.results|.[]|.name'
The example playbook snippet would be helpful for me since I am new to ansible.

Escaping Hell....
I was not able to use replace, probably because I didn't try hard/smart enough. Meanwhile, in your specific case, you can achieve the expected result with a single regex_replace filter call so it was easier (and it worked right away :)).
I used yaml folded blocks (>) with white space control (-) to minimize the escape hassle. If you don't know what that is, have a look at a yaml doc (learn yaml in y minutes is my favorite one)
Note that the remaining backslashes in the last result below are added by ansible to escape the double quotes in the output.
---
- name: Escape chars
hosts: localhost
gather_facts: false
vars:
test: >-
curl -s -H \"Authorization: JWT eyJ4NWMiOlsiTUlJQytqQ0NBHuHO96csEQ\r\"
https://hub.docker.com/v2/repositories/talasecurityinc/?page_size=10000
| jq -r '.results|.[]|.name'
tasks:
- name: Show the untouched var
debug:
var: test
- name: Escape the var as intended
debug:
msg: >-
{{ test | regex_replace('\\r?\\?', '') }}
which results in
PLAY [Escape chars] ********************************************************************
TASK [Show the untouched var] **********************************************************
ok: [localhost] => {
"test": "curl -s -H \\\"Authorization: JWT eyJ4NWMiOlsiTUlJQytqQ0NBHuHO96csEQ\\r\\\" https://hub.docker.com/v2/repositories/talasecurityinc/?page_size=10000 | jq -r '.results|.[]|.name'"
}
TASK [Escape the var as intended] ******************************************************
ok: [localhost] => {
"msg": "curl -s -H \"Authorization: JWT eyJ4NWMiOlsiTUlJQytqQ0NBHuHO96csEQ\" https://hub.docker.com/v2/repositories/talasecurityinc/?page_size=10000 | jq -r '.results|.[]|.name'"
}
PLAY RECAP *****************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0

Related

How to display constant values using custom-columns format of kubectl?

I have multiple clusters and I want to check which ingresses do not specify explicit certificate. Right now I use the following command:
~$ k config get-contexts -o name | grep -E 'app(5|3)41.+-admin' | xargs -n1 -I {} kubectl --context {} get ingress -A -o 'custom-columns=NS:{.metadata.namespace},NAME:{.metadata.name},CERT:{.spec.tls.*.secretName}' | grep '<none>'
argocd argo-cd-argocd-server <none>
argocd argo-cd-argocd-server <none>
reference-app reference-app-netcore-ingress <none>
argocd argo-cd-argocd-server <none>
argocd argo-cd-argocd-server <none>
test-ingress my-nginx <none>
~$
I want to improve the output by including the context name, but I can't figure out how to modify the custom-columns format to do that.
The below command would Not yield the exact desired output, but it will be close. using jsonpath, it's possible:
kubectl config get-contexts -o name | xargs -n1 -I {} kubectl get ingress -A -o jsonpath="{range .items[*]}{} {.metadata.namespace} {.metadata.name} {.spec.tls.*.secretName}{'\n'}{end}" --context {}
If the exact output is needed, then the kubectl output needs to be looped in the bash loop. Example:
kubectl config get-contexts -o name | while read context; do k get ingress -A -o 'custom-columns=NS:{.metadata.namespace},NAME:{.metadata.name},CERT:{.spec.tls.*.secretName}' --context "$context" |awk -vcon="$context" 'NR==1{$0=$0FS"CONTEXT"}NR>1{$0=$0 FS con}1'; done |column -t
NS NAME CERT CONTEXT
default tls-example-ingress testsecret-tls kubernetes-admin-istio-demo.local#istio-demo.local
default tls-example-ingress1 testsecret-tls kubernetes-admin-istio-demo.local#istio-demo.local
default tls-example-ingress2 <none> kubernetes-admin-istio-demo.local#istio-demo.local
To perform post-processing around the header and context, the awk command was used. Here is some details about it:
Command:
awk -vcon="$context" 'NR==1{$0=$0FS"CONTEXT"}NR>1{$0=$0 FS con}1'; done |column -t
-vcon="$context": This is to create a variable called con inside awk to store the value of bash variable($context).
NR==1: Here NR is the record number(in this case line number) and $0 means record/line.
NR==1{$0=$0FS"CONTEXT"}: This means, on the 1st line, reset the line to itself followed by FS(default is space) followed by a string "CONTEXT".
Similarly, NR>1{$0=$0 FS con} means, from the 2nd line onwards, append the line with FS followed by con.
1 in the end is the tell awk to do the print.

Cypress AWS codebuild error: spec must be a string or comma-separated list

I am trying to implement parallel testing in AWS code build. I created a buildspec.yml file like this sample project:
https://github.com/cypress-io/cypress-realworld-app/blob/develop/buildspec.yml
My problem is the environments that I use during the cypress command are getting as empty.
- echo $CY_GROUP_SPEC
- CY_GROUP=$(echo $CY_GROUP_SPEC | cut -d'|' -f1)
- CY_BROWSER=$(echo $CY_GROUP_SPEC | cut -d'|' -f2)
- CY_SPEC=$(echo $CY_GROUP_SPEC | cut -d'|' -f3)
- CY_CONFIG=$(echo $CY_GROUP_SPEC | cut -d'|' -f4)
And then the cypress code build fails with this error:
Opening Cypress...
Cypress encountered an error while parsing the argument: --spec
You passed: true
The error was: spec must be a string or comma-separated list
I use this command to run cypress:
- NO_COLOR=1 ./node_modules/.bin/cypress run --browser $CY_BROWSER --spec "$CY_SPEC" --config "$CY_CONFIG" --headless. --record --key $CYPRESS_KEY --parallel --ci-build-id $CODEBUILD_INITIATOR --group "$CY_GROUP"
I defined these env variables like this on the top of the file:
batch:
build-matrix:
dynamic:
env:
image:
- ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/cypress:latest
variables:
CY_GROUP_SPEC:
- "UI - Chrome|chrome|cypress/e2e/account/*"
- "UI - Chrome|chrome|cypress/e2e/auth/*"
- "UI - Chrome|chrome|cypress/e2e/mastering/*"
- "UI - Chrome|chrome|cypress/e2e/pages/**/*"
- "UI - Chrome|chrome|cypress/e2e/user-flows/**/*"
WORKERS:
- 1
- 2
- 3
- 4
- 5
How can I fix this problem?
Thanks
The errors definitely tell you that the command is wrong. Check that carefully.

Dictionary with list remove u ticks and brackets

I'm having an issue accessing a list within a dictionay in ansible.
It's actually more complex than this, I've made a simplified version of the issue for brevity.
ansible version is 2.9
The dictionary looks like:
data_dict:
data:
arch_paths:
- /opt/data1/20201007_test1
- /opt/data1/20201013_test2
- /opt/data1/20201029_test3
bucket: host-data
path: archives/host
src_path: /opt/data1
targz:
arch_paths:
- /opt/data2/20201005.tar.gz
- /opt/data2/20201013.tar.gz
- /opt/data2/20201029.tar.gz
bucket: app-data
path: archives/app
src_path: /opt/data2
My task examples:
- name: print files to archive
debug:
msg: "{{item.value.arch_paths}}"
loop: "{{ data_dict|dict2items }}"
- name: Archive the things
shell: echo "{{item.value.bucket}} -k {{item.value.path}} -p {{item.value.src_path}} {{item.value.arch_paths}}" >> /var/log/put.log
loop: "{{ data_dict|dict2items }}"
These run as expected. But, what ends up in the log file is:
host-data -k archives/host -p /opt/data1 [u'/opt/data1/20201007_test1', u'/opt/data1/20201013_test2', u'/opt/data1/20201029_test3']
app-data -k archives/app -p /opt/data2 [u'/opt/data2/20201005.tar.gz', u'/opt/data2/20201013.tar.gz', u'/opt/data2/20201029.tar.gz']
I want to remove the u'...' and commas so I just have the values with spaces e.g.
host-data -k archives/host -p /opt/data1 /opt/data1/20201007_test1 /opt/data1/20201013_test2 /opt/data1/20201029_test3
app-data -k archives/app -p /opt/data2 /opt/data2/20201005.tar.gz /opt/data2/20201013.tar.gz /opt/data2/20201029.tar.gz
I've seen similar problems on this site but have been unable to adapt any of the solutions.
This happens because arch_paths is a list, so, you just need to join the list items: item.value.arch_paths | join(' ').
With the task:
- shell: echo "{{ item.value.bucket }} -k {{ item.value.path }} -p {{ item.value.src_path }} {{ item.value.arch_paths | join(' ') }}" >> /var/log/put.log
loop: "{{ data_dict|dict2items }}"
It gives a file /var/log/put.log containing:
host-data -k archives/host -p /opt/data1 /opt/data1/20201007_test1 /opt/data1/20201013_test2 /opt/data1/20201029_test3
app-data -k archives/app -p /opt/data2 /opt/data2/20201005.tar.gz /opt/data2/20201013.tar.gz /opt/data2/20201029.tar.gz

How to collect a grep and use it in a aws configset

In my aws Cloud Formation cfn configset I have a command to set an environment key to the name of the user group apache belongs to as it might be apache or www-data depending on the distro.
Something like this:
Metadata:
AWS::CloudFormation::Init:
configSets:
joomla:
- "set_permissions"
- "and_some_more..."
configure_cfn:
files:
/etc/cfn/hooks.d/cfn-auto-reloader.conf:
content: !Sub |
[cfn-auto-reloader-hook]
triggers=post.update
path=Resources.EC2.Metadata.AWS::CloudFormation::Init
action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource EC2 --configsets joomla --region ${AWS::Region}
mode: "000400"
owner: root
group: root
.....
set_permissions:
commands:
01_01_get_WebServerGroup:
env:
#webserver group might be apache or www-data depending on the distro
WebServerGp:
command: "ps -ef | egrep '(httpd|apache2|apache)' | grep -v `whoami` | grep -v root | head -n1 | awk '{print $1}'"
However, when I launch this stack, the configsets process halts at this point and I get an an error in the cfn_init.log that looks like this:
File
"/usr/lib/python2.7/dist-packages/cfnbootstrap/command_tool.py", line
80, in apply
raise ToolError(u"%s does not specify the 'command' attribute, which is required" % name) ToolError: 01_01_get_WebServerGroup does
not specify the 'command' attribute, which is required
Is this the preferred method to catch and use a grep result in a configset command? Is there a better way? What can I do to address the error thrown in the cfn_init.log?
OK, I guess I can create parameter and mapping elements to capture the distro type on launch and then set the webserver group accordingly but I am really trying to understand how to set the env: key to a response from the cli.
The problem of your code is this line WebServerGp.
Line command is must be on the same level of env, under the commands name, in your case is 01_01_get_WebServerGroup. So, it has to be like this:
commands:
01_01_get_WebServerGroup:
env: ..
command: ..
If you want to use the result of grep, you can put them on variable and use it later.
You can specify more than one command under that command line using \n for executing the command.
Please check this code below.
command: "result=(ps ef | grep ...)\n echo $result\n ..."
If you have really long command, you can use the Fn::Join to be the value of command.

How do I use regex with Ansible?

I am having trouble getting a regex pattern to match in Ansible 2. Could someone help me understand what I am doing wrong? Thank you.
--- # Disable auto update for Ubuntu
- hosts: nonedgeLinux
become: yes
tasks:
- name: disable auto updates
replace:
dest: /etc/apt/apt.conf.d/50unattended-upgrades
regexp: '(?:[ \t]*\"\${distro_id}:\${distro_codename}-security\";)'
replace: '// "\${distro_id}:\${distro_codename}-security\";'
I used https://regex101.com/ to validate the regex against a copy/paste of the file contents being searched. This test reports the regexp pattern is correct. Each time it is run on a Ubuntu 16.04.2 box I get the following results:
root#sbx54:/data/scripts/ansible# ansible-playbook disableAutoUpdate.yml -vvvv
Using /data/scripts/ansible/ansible.cfg as config file
Loaded callback default of type stdout, v2.0
1 plays in disableAutoUpdate.yml
PLAY ****************************************************************************
TASK [setup] *******************************************************************
ESTABLISH LOCAL CONNECTION FOR USER: root
localhost EXEC ( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1492480514.73-9504514182168 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1492480514.73-9504514182168 )" )
localhost PUT /tmp/tmpkYPc6g TO /root/.ansible/tmp/ansible-tmp-1492480514.73-9504514182168/setup
localhost EXEC LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1492480514.73-9504514182168/setup; rm -rf "/root/.ansible/tmp/ansible-tmp-1492480514.73-9504514182168/" > /dev/null 2>&1
ok: [localhost]
TASK [disable auto updates] ****************************************************
task path: /data/scripts/ansible/disableAutoUpdate.yml:15
ESTABLISH LOCAL CONNECTION FOR USER: root
localhost EXEC ( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1492480515.71-278594852314124 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1492480515.71-278594852314124 )" )
localhost PUT /tmp/tmpG3gchf TO /root/.ansible/tmp/ansible-tmp-1492480515.71-278594852314124/replace
localhost EXEC LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1492480515.71-278594852314124/replace; rm -rf "/root/.ansible/tmp/ansible-tmp-1492480515.71-278594852314124/" > /dev/null 2>&1
ok: [localhost] => {"changed": false, "invocation": {"module_args": {"backup": false, "content": null, "delimiter": null, "dest": "/etc/apt/apt.conf.d/50unattended-upgrades", "directory_mode": null, "follow": false, "force": null, "group": null, "mode": null, "owner": null, "regexp": "(?:(?!\\/\\/).*[ \\t]*\\\"\\${distro_id}:\\${distro_codename}-security\\\";)/g", "remote_src": null, "replace": "replaced", "selevel": null, "serole": null, "setype": null, "seuser": null, "src": null, "validate": null}, "module_name": "replace"}, "msg": ""}
PLAY RECAP *********************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0
Here is a snip of the file being searched:
Unattended-Upgrade::Allowed-Origins {
"${distro_id}:${distro_codename}";
"${distro_id}:${distro_codename}-system";
// "${distro_id}:${distro_codename}-updates";
// "${distro_id}:${distro_codename}-proposed";
// "${distro_id}:${distro_codename}-backports";
};
# ansible --version
ansible 2.0.0.2
config file = /data/scripts/ansible/ansible.cfg
configured module search path = Default w/o overrides
One last note: I did have success in search and replace within this file when only looking for strings. When looking for the entire line is when this fails. Additionally I have tried escaping the special characters as well, e.g. [/][/] in place of \/\/\, etc. with no luck.
On top of the mistake in trying to match security with system, you don't need to escape the double quotes in either of the arguments or the dollar sign in the replace argument (otherwise the backslash will be inserted into the file):
- name: disable auto updates
replace:
dest: /etc/apt/apt.conf.d/50unattended-upgrades
regexp: '(?:[ \t]*"\${distro_id}:\${distro_codename}-system";)'
replace: '// "${distro_id}:${distro_codename}-system";'
And you can achieve the same result using lineinfile module (which makes code a bit more readable, imho):
- name: disable auto updates
lineinfile:
dest: /etc/apt/apt.conf.d/50unattended-upgrades
regexp: '"\${distro_id}:\${distro_codename}-system"'
line: '// "${distro_id}:${distro_codename}-system";'