I'm using Ansible to build inventories dynamically according to several parameters.
I get raw info from our CMDB's API, then use set_fact to format a list of FQDNs.
Those FQDNs always follow the same formula, so it looks easy, but Ansible only seems to apply the templating to the first item.
The code:
- name: populate list of all hosts
set_fact:
all_hosts: '{{ all_hosts + [ "{{item.value.fields.friendlyname | lower}}" "." "{{item.value.fields.friendlyname[:3]|lower}}" ".sncoia.lan" ] }}'
loop: "{{ hostname.json.objects | dict2items }}"
when:
- item.value.fields.friendlyname | length == 12
- debug: var=all_hosts
The result (abridged for brevity):
TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
"all_hosts": [
"devlappaps03.dev.sncoia.lan",
"{{item.value.fields.friendlyname | lower}}.{{item.value.fields.friendlyname[:3]|lower}}.sncoia.lan",
"{{item.value.fields.friendlyname | lower}}.{{item.value.fields.friendlyname[:3]|lower}}.sncoia.lan",
Given the data for testing
hostname:
json:
objects:
foo:
fields:
friendlyname: devlappaps03
bar:
fields:
friendlyname: devlappaps02
baz:
fields:
friendlyname: devlappaps01
Properly concatenate the FQDN. For example
- set_fact:
all_hosts: "{{ all_hosts|d([]) + [_fqdn] }}"
loop: "{{ hostname.json.objects|dict2items }}"
when: item.value.fields.friendlyname|length == 12
vars:
_fqdn_list:
- "{{ item.value.fields.friendlyname|lower }}"
- "{{ item.value.fields.friendlyname[:3]|lower }}"
- "sncoia.lan"
_fqdn: "{{ _fqdn_list|join('.') }}"
gives
all_hosts:
- devlappaps03.dev.sncoia.lan
- devlappaps02.dev.sncoia.lan
- devlappaps01.dev.sncoia.lan
Related
below is output of the "RESULT":
stdout:
- |-
ospf T1 VRF vrf1
ospf T2 VRF vrf2
ospf T3 VRF vrf3
stdout_lines:
- ospf T1 VRF vrf1
- ospf T2 VRF vrf2
- ospf T3 VRF vrf3
I want output in list and in dictionary:
1st output will be list. list will have following:
- T1
- T2
- T3
2nd output will be list like below:
ospf_vrf:
- vrf: vrf1
process: T1
- vrf: vrf2
process: T2
- vrf: vrf3
process: T3
3rd output will be dictionary.
how to do that?
I would start by parsing the data from stdout via regex. Assuming the components in your output are always: ospf <process_value> VRF <vrf_falue>, the regex looks like this:
{{ stdout | regex_findall('^ospf (.+) VRF (.+)$', multiline=True) }}
Based on this, you can then put the values into different forms and store them in variables:
- set_fact:
ospf: "{{ extracted | map('first') }}"
ospf_vrf: "{{ extracted | map('zip', ['process', 'vrf']) |
map('map', 'reverse') | map('community.general.dict') }}"
ospf_dict: "{{ dict(extracted) }}"
vrf_dict: "{{ dict(extracted | map('reverse')) }}"
vars:
extracted: "{{ stdout | regex_findall('^ospf (.+) VRF (.+)$', multiline=True) }}"
extracted is a helper with the regex values and the base for all other values
ospf is a list of the process_values
In each case the first element of the regex is taken.
ospf_vrf is a list of dicts
the keys process and vrf are assigned to the regex values
by reverse the keys are brought to the front
by community.general.dict a dict is created from the key-value pairs
ospf_dict: a dict based on process
vrf_dict: a dict based on vrf
Note: Both ospf_dict and vrf_dict will lose values if there are duplicates for the keys.
The output of the 4 variables looks like this:
TASK [debug] ***************************
ok: [localhost] => {
"ospf": [
"T1",
"T2",
"T3"
]
}
TASK [debug] ***************************
ok: [localhost] => {
"ospf_vrf": [
{
"process": "T1",
"vrf": "vrf1"
},
{
"process": "T2",
"vrf": "vrf2"
},
{
"process": "T3",
"vrf": "vrf3"
}
]
}
TASK [debug] ***************************
ok: [localhost] => {
"ospf_dict": {
"T1": "vrf1",
"T2": "vrf2",
"T3": "vrf3"
}
}
TASK [debug] ***************************
ok: [localhost] => {
"vrf_dict": {
"vrf1": "T1",
"vrf2": "T2",
"vrf3": "T3"
}
}
I have created this sample playbook for your requirements. I am not clear on third expected output. please clarify on that
- hosts: localhost
gather_facts: no
vars:
stdout_lines:
- ospf T1 VRF vrf1
- ospf T2 VRF vrf2
- ospf T3 VRF vrf3
tasks:
- set_fact:
first_output: "{{ first_output | default([]) + [ (item | split())[1] ] }}"
with_items:
"{{ stdout_lines }}"
- debug:
var: first_output
- set_fact:
second_output: "{{ second_output | default([]) + [ { 'vrf': (item | split())[3] , 'process': (item | split())[1] } ] }}"
with_items:
"{{ stdout_lines }}"
- debug:
var: second_output
I've a dictionary that I'm gathering through an API that looks like below,
ok: [localhost] => {
"filtered_inventory": [
{
"Manufacturer": "",
"Name": "daughtercard 0/0/CPU0",
"Serial_number": "serial",
"device_id": 287,
"part_id": "ASR9001-LC"
},
{
"Manufacturer": "",
"Name": "module mau 0/0/2/3",
"Serial_number": "serial",
"device_id": 287,
"part_id": "SFP-10G-SR"
},
I'm also using the below API to get the below values but it's stored as a list
ok: [localhost] => {
"devices": [
{
"device_id": 287,
"hostname": "example.test.com"
},
{
"device_id": 350,
"hostname": "example.test.se"
},
What I want to achive is, if device_id is the same in both the list and the dict then append the hostname to the dictionary. I've solved it with the below code but as I'm working with a big dictionary it's highly inefficiant, running it with the below loop takes around 1 hour and eats a ton of memory.
- name: Merge hostname with inventory
#no_log: True
set_fact:
merged_list: "{{ merged_list|default([]) +
[{'hostname': item[0].hostname|upper,
'device_id': item[0].device_id,
'serial': item[1].Serial_number,
'Name': item[1].Name,
'Manufacturer': item[1].Manufacturer,
'part_id': item[1].part_id}] }}"
when: "item[0].device_id == item[1].device_id"
loop: "{{ query('nested', devices, filtered_inventory) }}"
I've so far been thinking that this probably could be solved by only looping through the dictionary and then match it to the list but haven't figured out how to do that, any ideas?
I'm thinking the below code could be a start but I've tried a few different approaches but failed to get it to work.
- name: Merge hostname with inventory
#no_log: True
set_fact:
merged_list: "{{ merged_list|default([]) + [item|combine
({'device_id': item.device_id,
'serial': item.Serial_number,
'Name': item.Name,
'Manufacturer': item.Manufacturer,
'part_id': item.part_id,
'hostname': '' })] }}"
loop: "{{ filtered_inventory }}"
when:
- "item.device_id is in devices|json_query(query)"
vars:
query: "'[*].device_id'"
Create a dictionary of the hostnames, e.g.
- set_fact:
devices_dict: "{{ devices|items2dict(key_name='device_id', value_name='hostname') }}"
gives
devices_dict:
287: example.test.com
350: example.test.se
Then, use this dictionary to combine items of a modified list, e.g.
- set_fact:
fi2: "{{ fi2|default([]) +
[item|combine({'hostname': devices_dict[item.device_id]})] }}"
loop: "{{ filtered_inventory }}"
gives
fi2:
- Manufacturer: ''
Name: daughtercard 0/0/CPU0
Serial_number: serial
device_id: 287
hostname: example.test.com
part_id: ASR9001-LC
- Manufacturer: ''
Name: module mau 0/0/2/3
Serial_number: serial
device_id: 287
hostname: example.test.com
part_id: SFP-10G-SR
Q: "It takes around 6 minutes to run ..."
A: This is too much. Try the filter below
shell> cat filter_plugins/combine_attr.py
def combine_attr(l, d, k, v):
x = []
for i in l:
i[k] = d[i[v]]
x.append(i)
return x
class FilterModule(object):
''' Ansible filter. Add an attribute to the dictionaries in the list.'''
def filters(self):
return {
'combine_attr': combine_attr,
}
For example
- set_fact:
fi2: "{{ filtered_inventory|
combine_attr(devices_dict, 'hostname', 'device_id') }}"
My goal is to create new devices in NetBox with ansible. So, I use netbox_device for this reason. I am getting the required information (device names, types, serial numbers etc) from a json file like the following one:
{
"results":{
"infos": [
{
"device_type": "type1",
"device_name": "name1",
"serial_number": "num1"
},
{
"device_type": "type2",
"device_name": "name2",
"serial_number": "num2"
}
]
}
}
So, what I am doing is that I have this simple task to create 1 new device :
- name: Create new devices within Netbox
netbox_device:
netbox_url: http://url.goes.here
netbox_token: 7575747448..66353
data:
name: name1
device_type: type1
device_role: Core Switch
serial_number: "num1"
site: Main
state: present
This is something simple if someone wants to create one specific new device, but I want to dynamically create new devices while looping some lists with the required info.
In particular, I create from the json file one list with all devices names
ok: [localhost] => {
"ansible_facts": {
"dev_names": [
"nameisone",
"nameistwo"
]
},
"changed": false
}
Then I create another list from the json file with the device types
ok: [localhost] => {
"ansible_facts": {
"dev_types": [
"type_one",
"type_two"
]
},
"changed": false
}
So now I have a task with 1 loop regarding the above lists:
- name: Create new devices within Netbox looping the lists with the required info
netbox_device:
netbox_url: http://url.goes.here
netbox_token: 754544444444404509504959433333333
data:
name: "{{ item }}"
device_type: "{{ item }}"
device_role: "Core Switch"
site: Site
#serial_number: "number"
state: present
loop:
- "{{ dev_names }}"
- "{{ dev_types }} "
This works well only with one list, which means that if I comment out one of the two lists I get no errors.
The error I am getting when I run this task with both my lists is this:
File "/tmp/poioanni/ansible_netbox_device_payload_5W1o8U/ansible_netbox_device_payload.zip/ansible/module_utils/net_tools/netbox/netbox_utils.py", line 352, in normalize_data
AttributeError: 'list' object has no attribute 'lower'
failed: [localhost] (item=[u'nameisone', u'nameistwo']) => {
"ansible_loop_var": "item",
"changed": false,
"item": [
"nameisone",
"nameistwo"
],
Any ideas ?
If you are sure your lists are synchronized, you can use the zip filter, as demonstrated in the below test.yml MVCE playbook.
---
- name: Zip demo
hosts: localhost
gather_facts: false
vars:
"dev_names": [
"nameisone",
"nameistwo"
]
"dev_types": [
"type_one",
"type_two"
]
tasks:
- name: demonstrate how to use the zip filter with a loop
debug:
msg: "Element from first list: {{ item.0 }}. Element from second list: {{ item.1 }}"
loop: "{{ dev_names | zip(dev_types) | list }}"
Which gives:
$ ansible-playbook test.yml
PLAY [Zip demo] ************************************************************************************************************************************************************************************************************************
TASK [demonstrate how to use the zip filter with a loop] *******************************************************************************************************************************************************************************
ok: [localhost] => (item=['nameisone', 'type_one']) => {
"msg": "Element from first list: nameisone. Element from second list: type_one"
}
ok: [localhost] => (item=['nameistwo', 'type_two']) => {
"msg": "Element from first list: nameistwo. Element from second list: type_two"
}
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
I'm trying to get the value of a variable in a file into an Ansible variable so I can use it.
Here's what I've got:
- name: extract Unique Key
shell: "grep UNIQUE_KEY ../config.py | cut -d' ' -f 3"
register: command_output
- set_fact:
unique_key: x{{ command_output.stdout | regex_replace("^'", '') | regex_replace('^"', '') | regex_replace("'$", '') | regex_replace('"$', '') }}
- set_fact:
unique_key: "{{ unique_key | regex_replace('^x', '') }}"
- debug: var=unique_key
This works, but feels kludgy and looks ugly.
I've already tried to add sed to my original shell module, but I couldn't figure out how to get the quotes escaped correctly. I also couldn't figure out how to escape the regex_replace to get it to work in a single variable assignment.
Is there a simpler way to go from this:
"TEST"
or
'TEST'
to this:
TEST
in Ansible? (I'm also really new to Ansible so that's not helping either)
EDIT: After the answer by #Vladimir-Botka which I initially accepted, I found this issue:
If I don't strip the quotes and embed the variable in another variable, it keeps the quotes:
I need to use this value to construct a path:
vars:
service_location: "/opt/{{ unique_key }}-scheduler-service"
If I don't remove the quotes using my method above, The variable will contain the quotes as in this output of a debug statement:
ok: [fedorasvr1] => {
"service_location": "/opt/'TEST'-scheduler-service"
}
The short answer is "omit the first and the last character" if the quotes are part of the string
- set_fact:
unique_key: command_output.stdout[1:-1]
Internal interpretation is all the same. The quotes control the expansion of the variables. See 7.3.1. Double-Quoted Style and 7.3.2. Single-Quoted Style.
As an example. The play below
- hosts: localhost
vars:
var1: TEST
var2: 'TEST'
var3: "TEST"
tasks:
- template:
src: test.j2
dest: test
and the template
shell> cat test.j2
{{ var1 }}
{{ var2 }}
{{ var3 }}
give
shell> cat test
TEST
TEST
TEST
The quotes, if part of the string, can be removed. As an example the play below
- hosts: localhost
vars:
regex: "[`'\"]"
replace: ""
service_location: "/opt/{{ item|regex_replace(regex, replace)
}}-scheduler-service"
tasks:
- debug:
var: service_location
loop:
- '`TEST`'
- '"TEST"'
- '''TEST'''
- "'TEST'"
gives
ok: [localhost] => (item=`TEST`) =>
item: '`TEST`'
service_location: /opt/TEST-scheduler-service
ok: [localhost] => (item="TEST") =>
item: '"TEST"'
service_location: /opt/TEST-scheduler-service
ok: [localhost] => (item='TEST') =>
item: '''TEST'''
service_location: /opt/TEST-scheduler-service
ok: [localhost] => (item='TEST') =>
item: '''TEST'''
service_location: /opt/TEST-scheduler-service
It is also possible to use custom filter_plugins/string_filters.py which might be more convenient than complex escape constructs.
As an example. The play below
- hosts: localhost
vars:
replace: ""
service_location: "/opt/{{ item.0|string_replace(item.1, replace)
}}-scheduler-service"
tasks:
- debug:
var: service_location
with_together:
- - '`TEST`'
- '"TEST"'
- "'TEST'"
- - '`'
- '"'
- "'"
gives
ok: [localhost] => (item=[u'`TEST`', u'`']) =>
item:
- '`TEST`'
- '`'
service_location: /opt/TEST-scheduler-service
ok: [localhost] => (item=[u'"TEST"', u'"']) =>
item:
- '"TEST"'
- '"'
service_location: /opt/TEST-scheduler-service
ok: [localhost] => (item=[u"'TEST'", u"'"]) =>
item:
- '''TEST'''
- ''''
service_location: /opt/TEST-scheduler-service
FWIW, see other examples of filter_plugins.
Slightly different, but maybe related: I was getting unwanted quotes when using a lookup to initialize a variable which were then used in a "blockinfile" task. But it turns out the quotes were caused by the "blockinfile" and not the lookup, ie:
- name: set some variables
set_fact:
my_var: "{{ lookup('ini', 'my_var section=variables file=~/myconf.ini') }}"
- name: update myconf.cfg
blockinfile:
dest: myconf.cfg
state: present
create: true
owner: "{{user}}"
group: "{{user}}"
block: |
[Credentials]
access_key_id = blabla
secret_access_key = "{{my_var}}"
become: true
removing the quotes from this line worked:
secret_access_key = "{{my_var}}"
I am trying to look up the un used disks using ansible. So far I am able to save the unused disk paths in a variable. But I am getting a lot of other useless information such as plus signs and brackets. I was wondering how can I filter the list down and get the strings that include /dev.
This is the code I have so far:
- set_fact:
list_of_disks: []
- name: getting list of disks
set_fact:
list_of_disks: "[{{list_of_disks}} + '/dev/{{item.key}}']"
when:
- not item.value.partitions
- not item.value.holders
- not item.value.links.ids
- item.key | search ("sd")
with_dict: "{{ ansible_devices }}"
- name: display unused disks
debug:
var: list_of_disks.split("'")
And this is the output I am getting
"list_of_disks.split(\"'\")": [
"[[[] + ",
"/dev/sde",
"] + ",
"/dev/sdc",
"]"
]
}
I would like the output to look like
"/dev/sde",
"/dev/sdc",
Note: the disk names would change from one server to another. So I don't want to look for sde and sdc only. It should be able to filer down all disks that come up in the list.
Well, might not be exactly the format you want, but:
- debug:
msg: "{{ item.device }}"
with_items: "{{ ansible_facts.mounts }}"
when: "'/dev/' in item.device"
Should do the trick.
On my local, I got:
TASK [debug] **********************************************************************************************************************************
ok: [localhost] => (item=None) => {
"msg": "/dev/mapper/fedora-root"
}
ok: [localhost] => (item=None) => {
"msg": "/dev/nvme0n1p1"
}
ok: [localhost] => (item=None) => {
"msg": "/dev/mapper/fedora-home"
}
If you need them all on one line, then create a variable and append to it:
- set_fact:
list_of_disks=""
- set_fact:
list_of_disks: "{{ list_of_disks | regex_replace('^, ','') }}, {{ item.device }}"
with_items: "{{ ansible_facts.mounts }}"
when: "'/dev/' in item.device"
- debug:
msg: "{{ list_of_disks }}"
For this, I got:
TASK [debug] **********************************************************************************************************************************
ok: [localhost] => {
"msg": "/dev/mapper/fedora-root, /dev/nvme0n1p1, /dev/mapper/fedora-home"
}