Limit ansible playbook task concurrency - concurrency

Question

I'm updating several hosts with ansible at the same time, however I have a limitation...
I have to download artifacts from a common repository with no more than 3 simultaneous downloads!
The current solution I have is to limit the whole playbook to max three concurrent tasks
strategy: linear
serial: 3
Is it possible to limit concurrency only for particular task step rather than the whole playbook?

Answer 1

There's no direct way. Only workarounds like run_once loop with delegate_to or multiplying the task with loop and executing only one item per host.
See issue #12170, which is closed with "won't fix" status for details.
delegate_to loop:
- mytask: ..
delegate_to: "{{item}}"
run_once: true
# many diff ways to make the loop
with_inventory_hostnames: all
multiplied task:
- name: target task
debug: msg="Performing task on {{ inventory_hostname }}, item is {{ item }}"
with_items: "{{ play_hosts }}"
when: "hostvars[item].inventory_hostname == inventory_hostname"

Answer 2

Yes, it's possible to only limit the concurrency of a certain task.
You just need to add the throttle keyword to your download task.
Example:
- name: Download payload.tar.gz
get_url:
url: https://example.com/path/payload.tar.gz
dest: /mnt/scratch/payload.tar.gz
mode: '0640'
throttle: 3
Note that throttle was introduced in Ansible 2.9.

Answer 3

Thanks to previous comments and #12170 I have come with my proposal, which still is not working as desired for the download case.
Notice that 2 is the maximum number of concurrent tasks executions desired.
- name: Download at ratio three at most
win_get_url:
url: http://ipv4.download.thinkbroadband.com/100MB.zip
dest: c:/ansible/100MB.zip
force: yes
with_sequence: start=0 end={{ (( play_hosts | length ) / 2 ) | round (0, 'floor') | int }}
when: "(( ansible_play_batch.index(inventory_hostname) % (( play_hosts | length ) / 2 )) | round) == (item | int)"
While this will match the when on each iteration only if for certain hosts I still can see all the server performing the download at the same time.
Another way of testing it is with debug a message and a add a delay between iterations. This way is clear that only two are executed at each iterations.
- debug:
msg: "Item {{ item }} with modulus {{ (( ansible_play_batch.index(inventory_hostname) % (( play_hosts | length ) / 2 )) | round) }}"
with_sequence: start=0 end={{ (( play_hosts | length ) / 2 ) | round (0, 'floor') | int }}
loop_control:
pause: 2
when: "(( ansible_play_batch.index(inventory_hostname) % (( play_hosts | length ) / 2 )) | round) == (item | int)"

Answer 4

This seems like a case of the XY problem.
Why not download the files once to your controller, and then use the copy task to fan out from the controller to each host?
(I guess bandwidth concerns between your controller and hosts may cause issues for large files, but it's probably not going to be much different to limiting the download to 3 hosts anyway.)

Answer 5

You can override forks variable in ansible.cf. default value is 5.
ansible.cf
[defaults]
forks = 3
More info