SaltStack: Use directory as source only if it exists - directory-structure

I'd like to know if there's a way of running a SaltStack state only if a directory source is defined in the master.
Basically, I want to allow fo users to put certain configuration files in their HOME directory. Most of the users won't have anything special, but some of them might (a custom .vimrc, for instance)
What I'd like to do is execute a file.recurse for the user's HOMEdirectory only if that directory exists in master.
As of now, I have the following:
{% for username in pillar['users'] %}
{{ username }}:
user:
- present
- home: /home/{{ username }}
[ . . . ] # (yadda, yadda, yadda)
# ToDo: Is there a way of not running this AT ALL if the users's directory is
# not present?
/home/{{ username }}:
file.recurse:
- user: {{ username }}
- group: ubuntu
- source:
- salt://users/{{ username }}
- require:
- user: {{ username }}
{% endfor %}
What I want to do is what in the code appears as the ToDo: If there's a directory salt://users/{{ username }} in the salt tree (or in the salt master), then execute the file.recurse with that directory as the source for the file.recurse. Otherwise, just skip that state and use with whatever default content of $HOME is has when a user is created (just do nothing, I mean).
I thought adding an onlyif clause to the file.recurse configuration (something like - onlyif: test -e salt://users/{{ username }} would help... but nopes... I also tried to create an empty directory in salt://users/empty_dir/ and pass it as a default (as described here for a file.managed), but that trick doesn't work with file.recurse (at least not yet):
Thank you in advance.

You could try something similar to
{% for username in pillar['users'] %}
{{ username }}:
user:
- present
- home: /home/{{ username }}
[ . . . ] # (yadda, yadda, yadda)
# NB!!!! POTENTIAL PERF BOTTLENECK
{% if 'users/'+username not in salt['cp.list_master_dirs'](prefix='users') %}
/home/{{ username }}:
file.recurse:
- user: {{ username }}
- group: ubuntu
- source:
- salt://users/{{ username }}
- require:
- user: {{ username }}
{% endif %}
{% endfor %}

Related

Include files in Jinja, applying template then filtering

Summary
I have a Jinja2 template which I'm running with Ansible.
I would like my template to load another file, as a template (i.e. evaluating {{ var }}), then I'll filter that, and then paste the result in to the top level template.
I think I'm almost there, I just need to find a Jinja2 filter which takes in a string and parses it as a template.
MWE
In this example lets assume the filter I want to apply is just to make the file uppercase.
(Obviously this case is so simple I could do it in one template file. But my real use case is more complex.)
Top level template main.yaml.j2:
---
something:
blah:
x: {{ y }}
{%- set names = [ 'John', 'Amy' ] %}
z: >
{{ lookup('file', './other-file.j2') | upper | indent(4*2) }}
other-file.j2:
{%- for name in names %}
Hello {{ name }}
{%- endfor %}
Running it with this Ansible playbook:
---
- hosts: localhost
connection: local
tasks:
- name: generate template
template:
src: "main.yaml.j2"
dest: "output.yaml.j2"
trim_blocks: False
register: templating
vars:
y: 5
Desired output
---
something:
blah:
x: 5
z: >
HELLO JOHN
HELLO AMY
Actual Output
---
something:
blah:
x: 5
z: >
{%- FOR NAME IN NAMES %}
HELLO {{ NAME }}
{%- ENDFOR %}
Best Guess
I think I'm almost there.
I just need a filter which applies a Jinja2 template to text.
i.e. something like:
{{ lookup('file', './other-file.j2') | template | upper | indent(4*2) }}
(But template is not a real filter. Maybe there's another name?)
What else I've tried
{{ include './other-file.j2' | upper | indent(4*2) }}
doesn't work.
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "AnsibleError: template error while templating string: expected token 'end of print statement', got 'string'. String: ---\nsomething:\n blah:\n x: {{ y }}\n {%- set names = [ 'John', 'Amy' ] %}\n z: >\n {{ include './other-file.j2' | upper | indent(4*2) }}"}
{% include './other-file.j2' | upper | indent(4*2) %}
"TemplateNotFound: ./OTHER-FILE.J2"
doesn't work.
Use Case
For context, my use case is that I have a Jinja2 template generating AWS CloudFormation templates.
I'm trying to do it all in YAML, not JSON.
(Because YAML can have comments, and you don't have to worry about whether the last item in a list has a trailing comma, and it's generally easier to read and write and debug.)
Some CloudFormation resources need literal JSON pasted into the YAML file. (e.g. CloudWatch Dashboard bodies).
So I want to have another file in YAML, which Jinja2 converts to json, and pastes into my overall YAML template.
I want this dashboard to be generated with a for loop, and to pass in variables.
I would like to have a separate
Instead of file plugin
lookup('file', './other-file.j2')
use template plugin
lookup('template', './other-file.j2')
Note that the scope of the variable {% set names = ['John', 'Amy'] %} is the template main.yaml.j2. If this variable is used in the template other-file.j2 the command lookup('template', './other-file.j2') will crash with the error:
"AnsibleUndefinedVariable: 'names' is undefined"
Solution
Declare the variable in the scope of the playbook. For example
- template:
src: "main.j2"
dest: "output.txt"
vars:
names: ['John', 'Amy']
main.j2
{{ lookup('template', './other-file.j2') }}
other-file.j2
{% for name in names %}
Hello {{ name }}
{% endfor %}
give
shell> cat output.txt
Hello John
Hello Amy

Unable to set true/false as an environment variable's value for Cloud Function

I am writing a Deployment Manager script which creates a Cloud Function and sets some environment variables.
Everything works well apart from the fact that one of my properties/variables is not recognized by the Deployment Manager correctly. I keep on getting an error.
I have a property is-local that I supply from CMD line.
Its value needs to be false/true or I can also live with yes/no.
In the schema file if I specify the property as boolean and supply the value as false/true then the deployment starts and only the Cloud Function component fails with an error. I have specified the error as Error#1 below.
if I specify the property as string and supply the value as false/true then the deployment starts but fails immediately with an error. I have specified the error as Error#2 below.
main.jinja
{% set PROJECT_NAME = env['project'] %}
{% set CODE_BUCKET = properties['code-bucket'] %}
{% set IS_LOCAL = properties['is-local'] %}
resources:
- name: create-cf
type: create_cloud_function.jinja
properties:
name: test-cf
project: {{ PROJECT_NAME }}
region: europe-west1
bucket: {{ CODE_BUCKET }}
runtime: nodejs10
entryPoint: test
topic: test
environmentVariables: { 'CODE_BUCKET': {{ CODE_BUCKET }}, 'IS_LOCAL': {{IS_LOCAL}} }
main.jinja.schema
imports:
- path: create_cloud_function.jinja
required:
- code-bucket
- is-local
properties:
code-bucket:
type: string
description: Name of the code bucket to host the code for Cloud Function.
is-local:
type: boolean
description: Will Cloud Function run locally or in cloud.
create_cloud_function.jinja
{% set codeFolder = properties['name'] %}
{% set environmentVariables = properties['environmentVariables'] %}
resources:
#- type: cloudfunctions.v1.function
- type: gcp-types/cloudfunctions-v1:projects.locations.functions
name: {{ properties['name'] }}
properties:
parent: projects/{{ properties['project'] }}/locations/{{ properties['region'] }}
location: {{ properties['region'] }}
function: {{ properties['name'] }}
sourceArchiveUrl: gs://$(ref.{{ properties['bucket'] }}.name)/{{ codeFolder }}.zip
entryPoint: {{ properties['entryPoint'] }}
runtime: {{properties['runtime']}}
eventTrigger:
resource: $(ref.{{ properties['topic'] }}.name)
eventType: providers/cloud.pubsub/eventTypes/topic.publish
environmentVariables:
{% for key, value in environmentVariables.items() %}
{{ key }} : {{ value }}
{% endfor %}
Deployment Manager CMD
gcloud deployment-manager deployments create setup --template main.jinja --properties code-bucket:something-random-test-code-bucket,is-local:false
Error#1: - when the property type is boolean in schema file
{"ResourceType":"gcp-types/cloudfunctions-v1:projects.locations.functions","ResourceErrorCode":"400","ResourceErrorMessage":{"code":400,"message":"Invalid value at 'function.environment_variables[1].value' (TYPE_STRING), false","status":"INVALID_ARGUMENT","details":[{"#type":"type.googleapis.com/google.rpc.BadRequest","fieldViolations":[{"field":"function.environment_variables[1].value","description":"Invalid value at 'function.environment_variables[1].value' (TYPE_STRING), false"}]}],"statusMessage":"Bad Request","requestPath":"https://cloudfunctions.googleapis.com/v1/projects/someproject/locations/europe-west1/functions","httpMethod":"POST"}}
Error#2: - when the property type is string in schema file
errors:
- code: MANIFEST_EXPANSION_USER_ERROR
location: /deployments/setup/manifests/manifest-1571821997285
message: |-
Manifest expansion encountered the following errors: Invalid properties for 'main.jinja':
True is not of type 'string' at ['is-local']
Resource: main-jinja Resource: config
Any idea whats the issue here...
I'm unfamiliar with jinja but from my understanding, environment variables cannot be anything else but strings.
Said this, reading Error#1 I'd conclude that, effectively, the var type has to be string.
Then, at the second error we can clearly see that you are trying to put a boolean into a string.
So yeah, you have to play with true / false as strings.
You can define set the value as a string within the jinja file itself. See this post for some details and this page that provides different methods you can use.
In your case, you can edit the create_cloud_function.jinja file and change:
environmentVariables:
{% for key, value in environmentVariables.items() %}
{{ key }} : {{ value }}
to:
environmentVariables:
{% for key, value in environmentVariables.items() %}
{{ key }} : {{ value|string }}
Once the manifest is fully expanded, the value should be considered a string for the purpose of the API call to the Cloud Functions API
Eventually what I had 2 do was pass IS_LOCAL: '''false'''from the command line and {{ key }} : {{ value }} in my jinja file.
According to this documentation about Using environment variables in Jinja, you should use the following syntax to add an environment var to your templates:
{{ env["deployment"] }} # Jinja
And they show the following example:
- type: compute.v1.instance
name: vm-{{ env["deployment"] }}
properties:
machineType: zones/us-central1-a/machineTypes/f1-micro
serviceAccounts:
- email: {{ env['project_number'] }}-compute#developer.gserviceaccount.com
scopes:
- ...
Given that you are providing the value of is-local from CMD line, and according to this documentation:
Boolean values are case insensitive, so TRUE, true, and True are treated the same.
AND
To specify multiple properties, provide comma-separated key:value pairs. It does not matter in what order you specify the pairs. For example:
`gcloud deployment-manager deployments create my-igm
--template vm_template.jinja
--properties zone:us-central1-a,machineType:n1-standard-1,image:debian-9`
You should use TRUE, true, or True for is-local param.

how to append to a list in jinja2 for ansible

Below is the jinja2 template that i wrote to use in ansible.
{% set port = 1234 %}
{% set server_ip = [] %}
{% for ip in host_ip %}
{% do server_ip.append({{ ip }}:{{ port }}) %}
{% endfor %}
{% server_ip|join(', ') %}
Below is the my desired output:
devices = 192.168.56.14:1234,192.168.56.13:1234,192.168.56.10:1234
But when i am running the ansible playbook, it is throwing the error as below:
"AnsibleError: teme templating string: Encountered unknown tag 'do'. Jinja was looking for th: 'endfor' or 'else'
Any help would be appreciated..
Try below code:
{% set port = '1234' %}
{% set server_ip = [] %}
{% for ip in host_ip %}
{{ server_ip.append( ip+":"+port ) }}
{% endfor %}
{{ server_ip|join(',') }}
You ll get:
192.168.56.14:1234,192.168.56.13:1234,192.168.56.10:1234
I didn't like any of the answers, they feel too hacky (having to worry about outputting None, or spurious whitespace using other techniques), but I think I've found a solution that works well. I took inspiration from this answer on a related question and realized that you can call set multiple times for the same variable and seemingly not incur any penalty.
It's still a tad hacky, because I don't think it's intended to work like this (then again, several design decisions in Jinja make me scratch my head, so who knows).
{% set server_ip = server_ip.append({{ ip }}:{{ port }}) %}
Interestingly, while the value is indeed appended to server_ip, the return value of that append (which we now know very well is None) isn't assigned back to server_ip on the LHS. Which led me to discover that the LHS side of the statement seems to be a no-op.
So you can also do this and the append works:
{% set tmp = server_ip.append({{ ip }}:{{ port }}) %}
Yet, if you print tmp, nothing shows up. Go figure.
That worked for me:
- set_fact:
devices: >-
{% for ip in host_ip %}{{ ip }}:1234{% if not loop.last %},{% endif %}{% endfor %}
If you still want to use do then add
jinja2_extensions = jinja2.ext.do
to your ansible config file and change
{% do server_ip.append({{ ip }}:{{ port }}) %}` to `{% do server_ip.append({ip:port}) %}`
The most voted answer will cause a lot of whitespaces in the rendered result. Beside using do extension from jinja, the alternative solution is using whitespace control from jinja. Add minus sign - inside the block
{%- for ip in host_ip -%}...{%- endfor %}
will remove the whitespace.
In order to avoid having None printed all over using {{ server_ip.append( ip+":"+port ) }} (just spent 20 min debugging this) and if you don't want to use the misleading {% set _ = server_ip.append( ip+":"+port ) %}, you can go back to Python basics and do the following:
# Remember that [1, 2] + [3] = [1, 2, 3]
{% set server_ip = server_ip + [ip+":"+port] %}
In 99.9% situations this will do the job. However in special cases where you work with very large lists there may be a small performance downside here in terms of memory usage: in the above example, [1, 2] + [3] = [1, 2, 3], both [1, 2] and [1, 2, 3] (initial and modified list) will coexist in memory for a brief moment, contrary to the append method which doesn't create additional objects in memory.
One-line solution with map() and regex:
{{ ["1.1.1.1","2.2.2.2"]|map('regex_replace', '(.+)', "\\1:1234")|join(', ') }}
map('regex_replace', '(.+)', "\\1:1234") adds :1234 to any non-empty string (.+) in the passed array ["1.1.1.1","2.2.2.2"]. Result:
1.1.1.1:1234, 2.2.2.2:1234

How to compared a nested pillar key value in an if statement in jinja2 for saltstack

I am working on a saltstack state with some salt wrapped in jinja2.
When I attempt to compare a value from a pillar using jinja2 it appears argument evaluates to nothing.
If I query the value using salt cli, it returns the expected value.
I expect I am referencing the value incorrectly in the if statement with jinja2.
Here is all the needed info to understand and look at this problem:
Salt Master id is salt-dev
Salt Minion id is on same instance and is salt-dev
Here is the pillar top file:
base:
'salt-dev':
- docker-daemon.docker-daemon
Here is the nested pillar file locate at /srv/pillar/docker-daemon/docker-daemon.sls
docker-daemon:
- action: start
- runlevel: enabled
Here is the output of the salt cli command returning the content of the pillar for the minion salt-dev:
# salt 'salt-dev' pillar.items
salt-dev:
----------
docker-daemon:
|_
----------
action:
start
|_
----------
runlevel:
enabled
Here is the output of the value I am using in the if statement where the value returns nothing with jinja2, but returns as expected here with cli:
# salt 'salt-dev' pillar.get docker-daemon:action
salt-dev:
start
The line of jinja2 that is incorrect is:
{% if salt['pillar.get']('docker-daemon:action') == 'start' %}
It appears: salt['pillar.get']('docker-daemon:action') returns nothing, but from cli as shown above it does return something.
Also if I add a default value, which is used in the event this arg returned nothing it also works.
An example of adding a default value is:
{% if salt['pillar.get']('docker-daemon:action', 'def_value') == 'start' %}
I have shown it in context below:
Here is the state file where the if statements are having the same issue:
{% if ( (grains['osfinger'] == 'Oracle Linux Server-6') and (grains['osarch'] == 'x86_64')) %}
sync_docker-init:
file.managed:
- name: /etc/init.d/docker
- source: salt://docker-daemon/templates/docker-init
- user: root
- group: root
- mode: 755
action_docker-init:
{% if salt['pillar.get']('docker-daemon:action') == 'start' %}
service.running:
{% endif %}
{% if salt['pillar.get']('docker-daemon:action') == 'stop' %}
service.dead:
{% endif %}
- name: docker
- require:
- pkg: install_docker-engine
- watch:
- file: sync_docker-init
{% if salt['pillar.get']('docker-daemon:runlevel') == 'enabled' %}
-- enable: True
{% endif %}
{% if salt['pillar.get']('docker-daemon:runlevel') == 'disabled' %}
-- enable: False
{% endif %}
{% else %}
event.send:
- tag: 'salt/custom/docker-init/failure'
- data: "Management of docker init failed, OS not permitted."
{% endif %}
I am quite new at the moment to salt and jinja2, so this is 101 stuff, but I would appreciate some help, I have found nothing for some hours yet.
I attempted to echo this out and it seemed I just get a blank line
I found the solution.
The pillar file /srv/pillar/docker-daemon/docker-daemon.sls was formed as a list instead of a map.
I changed it to this:
docker-daemon:
action: restart
runlevel: disabled

Saltstack load pillar in a for loop

I am developing a automatic proftd installation whit Salt, i wont to get the ftp users from a template but I cant get work the pillar, i initialized the pillar whit the users data and call it into a for loop, but you don't get the pillar user data in the loop.
When i make salt-call pillar.get ftpusers in the minion, the response is:
local:
This is my pillar ftpusers.sls:
ftp-server.ftpusers:
user:
- user: user
- passhash: j2k3hk134123l1234ljh!"ยท$ser
- uuid: 1001
- guid: 1001
- home: /srv/ftp/user
- shel: /bin/false
And this is the for loop:
{% for users in pillar.get('ftpusers', {}).items() %}
/srv/herma-ftp/.ftpusers:
file.managed:
- user: root
- group: root
- mode: 444
- contents:'{{ user }}:{{ args['passhash'] }}:{{args['uuid'] }}:{{ args['guid'] }}::{{ args['home'] }}:{{ args['shel'] }}'
- require:
- file: /srv/herma-ftp
/srv/herma-ftp/{{user}}:
file.directory:
- user: nobody
- group: nobody
- dir_mode: 775
- makedirs: True
- require:
- file: /srv/herma-ftp
- watch:
- file: /srv/herma-ftp
module.run:
- name: file.set_selinux_context
- path: {{ args['home']}}
- type: public_content_t
- unless:
- stat -c %C {{ args['home'] }} |grep -q public_content_t
{% endfor %}
When I make in the minion
salt-call -l debug state.sls herma-ftp-server saltenv=My-enviroment test=True
Don't expect this for because don't can get the pillar data.
Your loop should also look like:
{% for user, args in pillar.get('ftpusers', {}).items() %}
Also, contents argument for a file.managed doesn't support templating. What you need to do is move /srv/herma-ftp/.ftpusers state outside of the loop, and make the loop inside the file template. The final layout of your state should look like:
/srv/herma-ftp/.ftpusers
file.managed:
source: salt://ftpserver/dot.ftpusers
template: jinja
...
...
{% for user, args in pillar.get('ftpusers', {}).items() %}
/srv/herma-ftp/{{user}}:
file.managed:
...
{% endfor %}
And your ftpserver/dot.ftpusers would look like:
{% for user, args in pillar.get('ftpusers', {}).items() %}
{{ user }}:{{ args['passhash'] }}:{{args['uuid'] }}:{{ args['guid'] }}::{{ args['home'] }}:{{ args['shel'] }}
{% endfor %}