Ansible - Looping on a list of lists one after another - list

Even if this question looks rather simple (and I've been searching and testing a lot), I cannot figure out how to loop on two ore more lists in ansible in a single task.
For instance, a very simple example :
My vars.yml :
list1:
- /tmp/dir1
- /tmp/dir2
- /tmp/dir3
list2:
- /tmp/dir4
- /tmp/dir5
- /tmp/dir6
My task:
task:
- name: Create basic directories if they do not exist
file:
path: "{{ DIRECTORY }}"
state: directory
loop:
- "{{ list1 }}"
- "{{ list2 }}"
loop_control:
loop_var: DIRECTORY
Obviously I could create all of them in a single task in a single list, but I have ALOT to create, hence my question.
Thanks in advance,

For example, given the lists below
list1:
- /tmp/dir1
- /tmp/dir2
- /tmp/dir3
list2:
- /tmp/dir4
- /tmp/dir5
- /tmp/dir6
list3:
- /tmp/dir1
- /tmp/dir4
- /tmp/dir9
Create a list of these lists, then flatten them and select unique items only, e.g.
list3: "{{ [list1, list2, list3]|flatten|unique }}"
gives probably what you are looking for
list3:
- /tmp/dir1
- /tmp/dir2
- /tmp/dir3
- /tmp/dir4
- /tmp/dir5
- /tmp/dir6
- /tmp/dir9
You can use it in the loop, of course
task:
- name: Create basic directories if they do not exist
file:
path: "{{ DIRECTORY }}"
state: directory
loop: "{{ [list1, list2, list3]|flatten|unique }}"
loop_control:
loop_var: DIRECTORY
The situation is even simpler if the lists are stored in a file, e.g.
shell> cat vars.yml
list1:
- /tmp/dir1
- /tmp/dir2
- /tmp/dir3
list2:
- /tmp/dir4
- /tmp/dir5
- /tmp/dir6
list3:
- /tmp/dir1
- /tmp/dir4
- /tmp/dir9
Include the variables(lists) from the file into a dictionary
- include_vars:
file: vars.yml
name: lists
This will create the dictionary lists
lists:
list1:
- /tmp/dir1
- /tmp/dir2
- /tmp/dir3
list2:
- /tmp/dir4
- /tmp/dir5
- /tmp/dir6
list3:
- /tmp/dir1
- /tmp/dir4
- /tmp/dir9
Then, flatten the values and select unique items. The expression below creates the same list as before without explicitly knowing the lists, i.e. you can keep the code untouched and change the data in the file vars.yml only
list3: "{{ lists.values()|flatten|unique }}"

Related

Need to generate a list based on one item name and number of items

I have two variables:
name: "abc232323defg10"
cycle: "4"
I need to generate a list:
list:
- abc232323defg10
- abc232323defg9
- abc232323defg8
- abc232323defg7
where:
abc232323defg9 = abc232323defg(10-(cycle-3)),
abc232323defg8 = abc232323defg(10-(cycle-2)),
abc232323defg7 = abc232323defg(10-(cycle-1))
The variable "cycle" is the same as the number of items in the list, and I already have item where last 2 characters are the "largest number" (that is number 10 in the example). So other items should have last two characters subtracted from this "largest number" (with increments for each cycle). Cycle is never bigger then "largest number", but can be equal.
Order in the list is not relevant.
PS last two characters can be any number or even combination of one letter (a-z,A-Z) and one number. So it can be t1, or e9, or 88... that is why I think I need regex.
Any idea?
Given the variables
_name: "abc232323defg10"
cycle: "4"
Declare the variables prefix and index by splitting _name
prefix: "{{ _name|regex_replace('^(.+?)(\\d+)$', '\\1') }}"
index: "{{ _name|regex_replace('^(.+?)(\\d+)$', '\\2') }}"
give
prefix: abc232323defg
index: 10
Declare the list indexes
indexes: "{{ query('sequence', params) }}"
params: "start={{ index|int }} end={{ index|int - cycle|int + 1 }} stride=-1"
give
indexes: ['10', '9', '8', '7']
Finally, create product of the prefix and indexes, and join them
names: "{{ [prefix]|product(indexes)|map('join')|list }}"
gives the expected result
names:
- abc232323defg10
- abc232323defg9
- abc232323defg8
- abc232323defg7
Example of a complete playbook for testing
- hosts: localhost
vars:
_name: "abc232323defg10"
cycle: "4"
prefix: "{{ _name|regex_replace('^(.+?)(\\d+)$', '\\1') }}"
index: "{{ _name|regex_replace('^(.+?)(\\d+)$', '\\2') }}"
indexes: "{{ query('sequence', params) }}"
params: "start={{ index|int }} end={{ index|int - cycle|int + 1 }} stride=-1"
names: "{{ [prefix]|product(indexes)|map('join')|list }}"
tasks:
- debug:
msg: |
prefix: {{ prefix }}
index: {{ index }}
indexes: {{ indexes }}
names:
{{ names|to_nice_yaml|indent(2) }}

QAbstractProxyModel: How to implement methods?

I have a TreeModel with a database like structure:
+Table1
-"Table1_key" -"name"
- 1 -"John"
- 2 -"Peter"
-...
+Table2
-"Table2_key" -"Table1_key" -"value1" -"value2"
- 1 - 1 - 1000 - 20000
- 2 - 1 - 3000 - 4000
- 3 - 2 - 1000 -2000
-...
+...
So the Data comes from an .xml file and is displayed in a TreeView which works just fine.
However i want to display some of the Tables in different Views and resolve the keys in that view.
For the example model the required View could look like this:
+"John"
-"value1" -"value2"
- 1000 - 20000
- 3000 - 4000
+"Peter"
- 1000 -2000
I guess using a QAbstractProxyModel would be the proper way.
So my question is: How do i implement this?
I can't find any examples and i have no idea how to map between the indexes of the source and the proxymodel in the mapToSource/mapFromSource methods.

How do I extract only the IPs from the data?

Trying to pull some logs and break it down. The following regex match gives me a correct match for all 4 IPs: ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}+) but not sure how to either delete the rest of the data or "extract" the IPs. Only need the IPs as shown.
June 3rd 2020, 21:18:02.193 [2020-06-03T21:18:02.781503+00:00,192.168.5.134,0,172.16.139.61,514,rslog1,imtcp,]<183>Jun 3 21:18:02 005-1attt01 atas_ssl: 1591219073.296175 CAspjq31LV8F0b 146.233.244.131 38530 104.16.148.244 443 - - - www.yahoo.com F - - F - - - - - - -
June 3rd 2020, 21:18:02.193 [2020-06-03T21:18:02.781503+00:00,192.168.5.134,0,172.16.139.61,514,rslog1,imtcp,]<183>Jun 3 21:18:02 005-1attt01 atas_ssl: 1591219073.296175 CAspjq31LV8F0b 146.233.244.131 38530 104.16.148.244 443 - - - www.yahoo.com F - - F - - - - - - -
Need this:
192.168.5.134 172.16.139.61 146.233.244.131 104.16.148.244
192.168.5.134 172.16.139.61 146.233.244.131 104.16.148.244

Ansible regex_search/regex_findall

I'm trying to parse the output of a command that returned a line like this (there's more output, but this is the line that I'm after):
Remaining Time: 3 Minutes and 12 Seconds
And when there is no time left it returns a line like this:
Remaining Time: 0 Seconds
I'd like to extract the amount of minutes and seconds, so I can feed it to GNU date -d. First I tried this:
- name: determine how much time we have left
set_fact:
time_left: "{{ cmd_output.stdout | regex_search(time_left_regex, '\\1', '\\2') | join(' ') }}"
vars:
time_left_regex: 'Remaining Time: ([0-9]+ Minutes) and ([0-9]+ Seconds)'
But this does not handle the case when there is no time left. So I then tried something like this:
- name: determine how much time we have left
set_fact:
time_left: "{{ cmd_output.stdout | regex_findall(time_left_regex, '\\1') }}"
vars:
time_left_regex: 'Next Execution:.*([0-9]{1,2} (Minutes|Seconds))'
But this only returns something like:
ok: [localhost] => {
"msg": "time left: [[u'2 Seconds', u'Seconds']]" }
I think I'm on the right track but I need a better regex, so maybe somebody can help me out here?
Thank you so much in advance.
You can make the minutes part optional. The minutes will be in group 1 and the seconds will be in group 2.
Remaining Time: (?:([0-9]+ Minutes) and )?([0-9]+ Seconds)
Regex demo
It's possible to split the string (line) and combine a dictionary. For example
- set_fact:
time_left: "{{ time_left|default({})|
combine({myline[item]: myline[item+1]}) }}"
loop: "{{ range(0, myline|length + 1, 3)|list }}"
vars:
myline: "{{ cmd_output.stdout.split(':').1.split()|reverse|list }}"
- debug:
var: time_left
for various command outputs
cmd_output.stdout: 'Remaining Time: 3 Minutes and 12 Seconds'
cmd_output.stdout: 'Remaining Time: 0 Seconds'
cmd_output.stdout: 'Remaining Time: 2 Days and 7 Hours and 3 Minutes and 12 Seconds'
gives (respectively)
"time_left": {
"Minutes": "3",
"Seconds": "12"
}
"time_left": {
"Seconds": "0"
}
"time_left": {
"Days": "2",
"Hours": "7",
"Minutes": "3",
"Seconds": "12"
}

Limit ansible playbook task concurrency

I'm updating several hosts with ansible at the same time, however I have a limitation...
I have to download artifacts from a common repository with no more than 3 simultaneous downloads!
The current solution I have is to limit the whole playbook to max three concurrent tasks
strategy: linear
serial: 3
Is it possible to limit concurrency only for particular task step rather than the whole playbook?
There's no direct way. Only workarounds like run_once loop with delegate_to or multiplying the task with loop and executing only one item per host.
See issue #12170, which is closed with "won't fix" status for details.
delegate_to loop:
- mytask: ..
delegate_to: "{{item}}"
run_once: true
# many diff ways to make the loop
with_inventory_hostnames: all
multiplied task:
- name: target task
debug: msg="Performing task on {{ inventory_hostname }}, item is {{ item }}"
with_items: "{{ play_hosts }}"
when: "hostvars[item].inventory_hostname == inventory_hostname"
Yes, it's possible to only limit the concurrency of a certain task.
You just need to add the throttle keyword to your download task.
Example:
- name: Download payload.tar.gz
get_url:
url: https://example.com/path/payload.tar.gz
dest: /mnt/scratch/payload.tar.gz
mode: '0640'
throttle: 3
Note that throttle was introduced in Ansible 2.9.
Thanks to previous comments and #12170 I have come with my proposal, which still is not working as desired for the download case.
Notice that 2 is the maximum number of concurrent tasks executions desired.
- name: Download at ratio three at most
win_get_url:
url: http://ipv4.download.thinkbroadband.com/100MB.zip
dest: c:/ansible/100MB.zip
force: yes
with_sequence: start=0 end={{ (( play_hosts | length ) / 2 ) | round (0, 'floor') | int }}
when: "(( ansible_play_batch.index(inventory_hostname) % (( play_hosts | length ) / 2 )) | round) == (item | int)"
While this will match the when on each iteration only if for certain hosts I still can see all the server performing the download at the same time.
Another way of testing it is with debug a message and a add a delay between iterations. This way is clear that only two are executed at each iterations.
- debug:
msg: "Item {{ item }} with modulus {{ (( ansible_play_batch.index(inventory_hostname) % (( play_hosts | length ) / 2 )) | round) }}"
with_sequence: start=0 end={{ (( play_hosts | length ) / 2 ) | round (0, 'floor') | int }}
loop_control:
pause: 2
when: "(( ansible_play_batch.index(inventory_hostname) % (( play_hosts | length ) / 2 )) | round) == (item | int)"
This seems like a case of the XY problem.
Why not download the files once to your controller, and then use the copy task to fan out from the controller to each host?
(I guess bandwidth concerns between your controller and hosts may cause issues for large files, but it's probably not going to be much different to limiting the download to 3 hosts anyway.)
You can override forks variable in ansible.cf. default value is 5.
ansible.cf
[defaults]
forks = 3
More info