I'm facing 2 problems:
Problem 1.
I'm trying to filter a list with jinja2 regex_search but I alse get None matches.
Problem 2.
Each elements of the new list seems to be list of one element (sigth!!!).
My code.
- name: Regex_Search Test
hosts: localhost
vars:
my_list:
- app-be-dev01-2
- app-be-dev02-2
- app-be-dev02-3
- app-be-dev03-2
- app-foo-2
- app-be-dev04-1
- app-be-dev04-2
tasks:
- name: Varsmng
set_fact:
customer_instances: >-
{% for instance in my_list -%} {{ customer_instances | default([]) + [ instance | string | regex_search('app-be-(.*)-([0-9]*)', '\1' ) ] }}
{%- endfor %}
- name: Debug
debug:
msg:
- "customer_instances: {{ customer_instances }}"
My output.
TASK [Varsmng] ****************************************************************************************************************************************
task path: /home/cin0633a/progetti/ansible/testenv/test.yml:19
ok: [localhost] => {
"ansible_facts": {
"customer_instances": "[[u'dev01']][[u'dev02']][[u'dev02']][[u'dev03']][None][[u'dev04']][[u'dev04']] "
},
"changed": false
As you can see, each element has a double square brackets. And can I avoid None values?
You get a list for each element because regex_search returns a list when you use the replace feature with capture groups in the expression.
$ ansible localhost -m debug -e toto=bla-bli-blo -a "msg={{ toto | regex_search('(bla).*') }}"
localhost | SUCCESS => {
"msg": "bla-bli-blo"
}
$ ansible localhost -m debug -e toto=bla-bli-blo -a "msg={{ toto | regex_search('(bla).*', '\\1') }}"
localhost | SUCCESS => {
"msg": [
"bla"
]
}
And you get None values because some items do not match your regex.
You can get your result with a better approach IMO using specific filters rather than a complex jinja2 template. The following playbook:
- name: Regex_Search Test
hosts: localhost
gather_facts: false
vars:
my_list:
- app-be-dev01-2
- app-be-dev02-2
- app-be-dev02-3
- app-be-dev03-2
- app-foo-2
- app-be-dev04-1
- app-be-dev04-2
searchreg: >-
app-be-(.*)-([0-9]*)
my_filtered_list: >-
{{
my_list |
select('regex', searchreg) |
map('regex_replace', searchreg, '\1')
}}
tasks:
- debug:
var: my_filtered_list
Gives:
PLAY [Regex_Search Test] *****************************************************************************
TASK [debug] *****************************************************************************
ok: [localhost] => {
"my_filtered_list": [
"dev01",
"dev02",
"dev02",
"dev03",
"dev04",
"dev04"
]
}
PLAY RECAP *****************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Note1: you can pipe the unique filter if you want unique environment results
Note2: if you are using ansible < 2.10, you will have to add the list filter at the end of all others to get the actual result.
I want to apply filter to the following log file. But I keep missing something
task.yml
- ansible_loop_var: item
changed: false
failed: true
invocation:
module_args:
policy_package: Package1
version: null
wait_for_task: true
wait_for_task_timeout: 30
item: PackageItem1
msg: Task Verify policy operation with task id 01234567-89ab-cdef-a422-xxxxxxxxx
failed. Look at the logs for more details
- ansible_loop_var: item
changed: false
failed: true
invocation:
module_args:
policy_package: Package2
version: null
wait_for_task: true
wait_for_task_timeout: 30
item: PackageItem2
msg: Task Verify policy operation with task id 01234567-89ab-cdef-a6c4-xxxxxxxx
failed. Look at the logs for more details
Here is my playbook
filter.yml
---
- name: sftp-domain
hosts: check_point
connection: httpapi
gather_facts: False
vars_files:
- 'credentials/my_var.yml'
- 'credentials/login.yml'
tasks:
- name: set-details
set_fact:
filter: "{{ lookup('file', 'tmp/task.yml') }}"
- name: create list to loop through
set_fact:
new_list: "{{ filter | map(attribute='msg') | flatten }}"
- name: copy-file-to-log
local_action:
module: copy
content: "{{ new_list | to_nice_yaml }}"
dest: tmp/task2.yml
I get an error message saying
PLAY [sftp-domain] *******************************************************************************************************************************************
TASK [set-details] *******************************************************************************************************************************************
ok: [checkpoint]
TASK [create list to loop through] ***************************************************************************************************************************
fatal: [checkpoint]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'unicode object' has no attribute 'msg'\n\nThe error appears to be in '/home/tdeveu0/project/fwp_automation/filter.yml': line 15, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: create list to loop through\n ^ here\n"}
Here is actually the result I want after applying filter
- msg: Task Verify policy operation with task id 01234567-89ab-cdef-a6c4-xxxxxxxx
failed. Look at the logs for more details
- msg: Task Verify policy operation with task id 01234567-89ab-cdef-a422-xxxxxxxxx
failed. Look at the logs for more details
I want to get only the list of all the 'msg'
Use the filter from_yaml
- name: set-details
set_fact:
filter: "{{ lookup('file', 'tmp/task.yml')|from_yaml }}"
Let's take a simplified file to show the problem, e.g.
shell> cat task.yml
- a: 1
b: 2
- a: 3
b: 4
When you read the file into the variable the result is AnsibleUnsafeText, not a list
- set_fact:
filter: "{{ lookup('file', 'task.yml') }}"
- debug:
msg: "{{ filter|type_debug }}"
- debug:
var: filter.0
gives
msg: AnsibleUnsafeText
filter.0: '-'
The first item of the text is the dash '-'.
Use filter from_yaml to get the list
- set_fact:
filter: "{{ lookup('file', 'task.yml')|from_yaml }}"
- debug:
msg: "{{ filter|type_debug }}"
- debug:
var: filter.0
gives
msg: list
filter.0:
a: 1
b: 2
Context
I have a list of files and directories to be deleted.
This is obtained from lines starting with the word "deleting" from an rsync stdout.
rsync stdout_lines: [
"building file list ... done",
"*deleting Lab02/Ex2/Doc1.txt",
"*deleting Lab02/Ex2/",
"*deleting Lab02/Ex1/Doc1.txt",
"*deleting Lab02/Ex1/",
"*deleting Lab02/",
".d..t...... ./",
"*deleting Lab01/Ex2/Doc1.txt",
"*deleting Lab01/Ex2/",
"*deleting Lab01/Ex1/Doc2.txt",
"*deleting Lab01/Ex1/Doc1.txt",
".d..t...... Lab01/",
".d..t...... Lab01/Ex1/",
"sent 350 bytes received 191 bytes 360.67 bytes/sec",
"total size is 614 speedup is 1.13 (DRY RUN)"
]
Formatted using:
'{{ sync_return.stdout_lines | select("regex", "^[*]deleting") | map("regex_replace", "^[*]deleting", "") | map("regex_replace", " ", "") | list }}'
A primitive example of the format of this list is as follows:
formatted list: [
"Lab02/Ex2/Doc1.txt",
"Lab02/Ex2/",
"Lab02/Ex1/Doc1.txt",
"Lab02/Ex1/",
"Lab02/",
"Lab01/Ex2/Doc1.txt",
"Lab01/Ex2/",
"Lab01/Ex1/Doc2.txt",
"Lab01/Ex1/Doc1.txt"
]
In an attempt to hasten the process of deleting (by reducing the number of elements to iterate over) - I separated the list into 2 sub-lists:
A list of directories. (elements of the main list that end in '/')
'{{ items_to_delete | select("regex", "/$") | list }}'
A list of file paths. (elements who's containing directory does not get deleted)
'{{ items_to_delete | reject("match", item) | list }}'
The sub-lists for the example above would be...
directories to delete: [
"Lab02/Ex2/",
"Lab02/Ex1/",
"Lab02/",
"Lab01/Ex2/"
]
files to delete: [
"Lab01/Ex1/Doc2.txt",
"Lab01/Ex1/Doc1.txt"
]
The Problem
Whilst the current solution works, it's not the best it could be. The dream is to have a solution where the "directories to delete" list only contains the highest level directories possible. i.e. Since we know the directory "Lab02/" is being deleted, "directories to delete" will NOT contain "Lab02/Ex2/" or "Lab02/Ex1/".
I believe my goal is somewhat similar to the os.path.commonprefix python function, however must be done for a variety of file paths within the list.
I'm relatively new to Ansible, so any guidance/help with this matter would be greatly appreciated.
I won't ask why you want to implement that, and I'll take it as an exercise.
Idea is, you can sort directories alphabetically, then while looping the paths, you strip any one that starts with the previous line.
You can write your filter like this (put in filter_plugins directory):
def common_paths(paths=[]):
sorted_paths = sorted(paths)
pfx = sorted_paths[0]
for path in sorted_paths[1:]:
if re.compile("^%s.*" % pfx).match(path):
sorted_paths.remove(path)
else:
pfx = path
return sorted_paths
class FilterModule(object):
def filters(self):
return { 'common_paths': common_paths }
then:
- name: Filter
set_fact:
bar: "{{ foo | common_paths }}"
Test locally with:
---
- hosts: localhost
tasks:
- name: Test data
set_fact:
foo:
- 'Lab01/'
- 'Lab01/Ex5/'
- 'Ex2/foo3/'
- 'Ex2/foo2/'
- 'Ex2/'
- 'Lab03/Ex5/e/'
- 'Lab02/y/z/Lab01/1/'
- 'Lab02/y/z/Lab01/3/'
- 'Lab01/Ex5/Lab02/'
- 'Lab03/Ex5/d/1'
- name: Filter
set_fact:
bar: "{{ foo | common_paths }}"
Output:
$ ansible-playbook common_paths.yml -vvv
ansible-playbook 2.10.4
PLAYBOOK: common_paths.yml *********************************************************************************
1 plays in common_paths.yml
PLAY [localhost] *********************************************************************************
TASK [Gathering Facts] *********************************************************************************
ok: [localhost]
TASK [Test data] ********************************************************************************* task path: /home/guido/Development/git/ansible-local/common_paths.yml:5
ok: [localhost] => {
"ansible_facts": {
"foo": [
"Lab01/",
"Lab01/Ex5/",
"Ex2/foo3/",
"Ex2/foo2/",
"Ex2/",
"Lab03/Ex5/e/",
"Lab02/y/z/Lab01/1/",
"Lab02/y/z/Lab01/3/",
"Lab01/Ex5/Lab02/",
"Lab03/Ex5/d/1/"
]
},
"changed": false
}
TASK [Filter] *********************************************************************************
task path: /home/guido/Development/git/ansible-local/common_paths.yml:19
ok: [localhost] => {
"ansible_facts": {
"bar": [
"Ex2/",
"Lab01/",
"Lab02/y/z/Lab01/1/",
"Lab02/y/z/Lab01/3/",
"Lab03/Ex5/d/1/",
"Lab03/Ex5/e/"
]
},
"changed": false
}
PLAY RECAP *********************************************************************************
localhost : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Having the following list of dictionaries:
- set_fact:
inventory:
- dn: host1
lid: 0021-00
unit_id: 50
- dn: host2
lid: 1011-00
unit_id: 50
- dn: host2
lid: 1004-00
unit_id: 50
Iterating over that list I'd like to create a new
dictionary as like (dn becomes key and lid becomes list value):
- set_fact:
dn2lid:
host1:
- 0021-00
host2:
- 1011-00
- 1004-00
Your help is much appreciated.
The task below does the job
- set_fact:
dn2lid: "{{ dn2lid|default({})|combine({item.0: lids}) }}"
loop: "{{ inventory|groupby('dn') }}"
vars:
lids: "{{ item.1|map(attribute='lid')|list }}"
- debug:
var: dn2lid
gives
dn2lid:
host1:
- 0021-00
host2:
- 1011-00
- 1004-00
I am trying to grab the "CRC Input Error" count and assign it to a variable "crc_count" however, the value of this variable is showing as an empty string instead ("crc_count": ""). What am I missing here?
Code:
- name: Checking CRC Errors on the Interfaces
nxos_command:
commands:
- show interface ethernet 1/1 counters detailed all
register: crc_output
- debug: var=crc_output.stdout_lines[0][50]
- set_fact: crc_count= "{{ crc_output.stdout_lines[0][50] | regex_search('(\d{0,20}$)') }}"
- debug:
msg: "{{ crc_count }}"
String output that I want to run regex on:
"crc_output.stdout_lines[0][50]": " 10. Input CRC Errors: = 4272"
Output I want to grab:
crc_count = "4272"
Playbook verbose output:
TASK [debug] **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
task path: /ansible/saurasar/nxos.crc.check.yml:19
ok: [ord12-pob-c1u1-dci-1.uspp1.oraclecloud.com] => {
"crc_output.stdout_lines[0][50]": " 10. Input CRC Errors: = 4272"
}
TASK [set_fact] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
task path: /ansible/xxxx/nxos.crc.check.yml:20
ok: [cisco.nexus.switch] => {"ansible_facts": {"_raw_params": "\"4272\"", "crc_count": ""}, "changed": false}
TASK [debug] **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
task path: /ansible/xxxxx/nxos.crc.check.yml:22
ok: [cisco.nexus.switch] => {
"msg": ""
}
META: ran handlers
META: ran handlers
PLAY RECAP ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
<cisco.nexus.switch> : ok=5 changed=0 unreachable=0 failed=0
Try as below
- set_fact:
crc_count: "{{ result.stdout | regex_replace('^.*Input CRC Errors: = (.*)$', '\\1') }}"
I tried with your sample and got as below
ok: [localhost] => {
"ansible_facts": {
"crc_count": "4272"
},
"changed": false
}