How to compared a nested pillar key value in an if statement in jinja2 for saltstack - python-2.7

I am working on a saltstack state with some salt wrapped in jinja2.
When I attempt to compare a value from a pillar using jinja2 it appears argument evaluates to nothing.
If I query the value using salt cli, it returns the expected value.
I expect I am referencing the value incorrectly in the if statement with jinja2.
Here is all the needed info to understand and look at this problem:
Salt Master id is salt-dev
Salt Minion id is on same instance and is salt-dev
Here is the pillar top file:
base:
'salt-dev':
- docker-daemon.docker-daemon
Here is the nested pillar file locate at /srv/pillar/docker-daemon/docker-daemon.sls
docker-daemon:
- action: start
- runlevel: enabled
Here is the output of the salt cli command returning the content of the pillar for the minion salt-dev:
# salt 'salt-dev' pillar.items
salt-dev:
----------
docker-daemon:
|_
----------
action:
start
|_
----------
runlevel:
enabled
Here is the output of the value I am using in the if statement where the value returns nothing with jinja2, but returns as expected here with cli:
# salt 'salt-dev' pillar.get docker-daemon:action
salt-dev:
start
The line of jinja2 that is incorrect is:
{% if salt['pillar.get']('docker-daemon:action') == 'start' %}
It appears: salt['pillar.get']('docker-daemon:action') returns nothing, but from cli as shown above it does return something.
Also if I add a default value, which is used in the event this arg returned nothing it also works.
An example of adding a default value is:
{% if salt['pillar.get']('docker-daemon:action', 'def_value') == 'start' %}
I have shown it in context below:
Here is the state file where the if statements are having the same issue:
{% if ( (grains['osfinger'] == 'Oracle Linux Server-6') and (grains['osarch'] == 'x86_64')) %}
sync_docker-init:
file.managed:
- name: /etc/init.d/docker
- source: salt://docker-daemon/templates/docker-init
- user: root
- group: root
- mode: 755
action_docker-init:
{% if salt['pillar.get']('docker-daemon:action') == 'start' %}
service.running:
{% endif %}
{% if salt['pillar.get']('docker-daemon:action') == 'stop' %}
service.dead:
{% endif %}
- name: docker
- require:
- pkg: install_docker-engine
- watch:
- file: sync_docker-init
{% if salt['pillar.get']('docker-daemon:runlevel') == 'enabled' %}
-- enable: True
{% endif %}
{% if salt['pillar.get']('docker-daemon:runlevel') == 'disabled' %}
-- enable: False
{% endif %}
{% else %}
event.send:
- tag: 'salt/custom/docker-init/failure'
- data: "Management of docker init failed, OS not permitted."
{% endif %}
I am quite new at the moment to salt and jinja2, so this is 101 stuff, but I would appreciate some help, I have found nothing for some hours yet.
I attempted to echo this out and it seemed I just get a blank line

I found the solution.
The pillar file /srv/pillar/docker-daemon/docker-daemon.sls was formed as a list instead of a map.
I changed it to this:
docker-daemon:
action: restart
runlevel: disabled

Related

Jinja commented out commands breaks templating

I have been trying to automate a template for deploy by Ansible:
Inventory contents:
[splunk_license]
10.10.113.209
[splunk_master]
[splunk_search]
10.10.113.209
[splunk_indexer]
10.10.113.234
My template has logic based on whether the splunk_master group has a host defined or not.
Original code:
{% if inventory_hostname in groups['splunk_indexer'] and
groups['splunk_master']|length > 0 %}
#{% if blah blah blah blah...%}
# CUSTOMER INDEXES go to $SPLUNK_HOME/etc/master-apps/_cluster/local/indexes.conf
# on Master node
{% elif inventory_hostname in groups['splunk_master'] %}
#{% if some other blah blah blah blah...%}
# CUSTOMER INDEXES go to $SPLUNK_HOME/etc/master-apps/_cluster/local/indexes.conf
{% else %}
# CUSTOMER INDEXES
[nothing]
coldToFrozenDir = $SPLUNK_DB/frozen/nothing/frozendb
thawedPath = $SPLUNK_DB/hotwarm/nothing/thaweddb
coldPath = volume:secondary/nothing/colddb
homePath = volume:primary/nothing/db
{% endif %}
No matter what I did I could not get the bottom part after {% else %} to work. Turns out the commenting out '#' does not actually cause that line to be ignored, which I had for testing purposes as I was tired of typing stuffs out over and over.
I tried to modify my (uncommented) if statements every which way from Sunday and I would either get only the top part of template, an Ansible error complaining about unexpected 'elif' or groups not found errors.
Jinja comments are as follows {# comment #} if using single # jinja will still evaluate those lines causing errors or a bad formatted destination file, see Jinja Templating docs
I was going to ask for help here but last minute tried to remove all commented lines out and now my template finally works.
Working code (commented lines removed):
{% if inventory_hostname in groups['splunk_indexer'] and
groups['splunk_master']|length > 0 %}
# CUSTOMER INDEXES go to $SPLUNK_HOME/etc/master-apps/_cluster/local/indexes.conf
# on Master node
{% elif inventory_hostname in groups['splunk_master'] %}
# CUSTOMER INDEXES go to $SPLUNK_HOME/etc/master-apps/_cluster/local/indexes.conf
{% else %}
# CUSTOMER INDEXES
[nothing]
coldToFrozenDir = $SPLUNK_DB/frozen/nothing/frozendb
thawedPath = $SPLUNK_DB/hotwarm/nothing/thaweddb
coldPath = volume:secondary/nothing/colddb
homePath = volume:primary/nothing/db
{% endif %}
Apologies if this is Jinja's obvious behaviour.

Unable to set true/false as an environment variable's value for Cloud Function

I am writing a Deployment Manager script which creates a Cloud Function and sets some environment variables.
Everything works well apart from the fact that one of my properties/variables is not recognized by the Deployment Manager correctly. I keep on getting an error.
I have a property is-local that I supply from CMD line.
Its value needs to be false/true or I can also live with yes/no.
In the schema file if I specify the property as boolean and supply the value as false/true then the deployment starts and only the Cloud Function component fails with an error. I have specified the error as Error#1 below.
if I specify the property as string and supply the value as false/true then the deployment starts but fails immediately with an error. I have specified the error as Error#2 below.
main.jinja
{% set PROJECT_NAME = env['project'] %}
{% set CODE_BUCKET = properties['code-bucket'] %}
{% set IS_LOCAL = properties['is-local'] %}
resources:
- name: create-cf
type: create_cloud_function.jinja
properties:
name: test-cf
project: {{ PROJECT_NAME }}
region: europe-west1
bucket: {{ CODE_BUCKET }}
runtime: nodejs10
entryPoint: test
topic: test
environmentVariables: { 'CODE_BUCKET': {{ CODE_BUCKET }}, 'IS_LOCAL': {{IS_LOCAL}} }
main.jinja.schema
imports:
- path: create_cloud_function.jinja
required:
- code-bucket
- is-local
properties:
code-bucket:
type: string
description: Name of the code bucket to host the code for Cloud Function.
is-local:
type: boolean
description: Will Cloud Function run locally or in cloud.
create_cloud_function.jinja
{% set codeFolder = properties['name'] %}
{% set environmentVariables = properties['environmentVariables'] %}
resources:
#- type: cloudfunctions.v1.function
- type: gcp-types/cloudfunctions-v1:projects.locations.functions
name: {{ properties['name'] }}
properties:
parent: projects/{{ properties['project'] }}/locations/{{ properties['region'] }}
location: {{ properties['region'] }}
function: {{ properties['name'] }}
sourceArchiveUrl: gs://$(ref.{{ properties['bucket'] }}.name)/{{ codeFolder }}.zip
entryPoint: {{ properties['entryPoint'] }}
runtime: {{properties['runtime']}}
eventTrigger:
resource: $(ref.{{ properties['topic'] }}.name)
eventType: providers/cloud.pubsub/eventTypes/topic.publish
environmentVariables:
{% for key, value in environmentVariables.items() %}
{{ key }} : {{ value }}
{% endfor %}
Deployment Manager CMD
gcloud deployment-manager deployments create setup --template main.jinja --properties code-bucket:something-random-test-code-bucket,is-local:false
Error#1: - when the property type is boolean in schema file
{"ResourceType":"gcp-types/cloudfunctions-v1:projects.locations.functions","ResourceErrorCode":"400","ResourceErrorMessage":{"code":400,"message":"Invalid value at 'function.environment_variables[1].value' (TYPE_STRING), false","status":"INVALID_ARGUMENT","details":[{"#type":"type.googleapis.com/google.rpc.BadRequest","fieldViolations":[{"field":"function.environment_variables[1].value","description":"Invalid value at 'function.environment_variables[1].value' (TYPE_STRING), false"}]}],"statusMessage":"Bad Request","requestPath":"https://cloudfunctions.googleapis.com/v1/projects/someproject/locations/europe-west1/functions","httpMethod":"POST"}}
Error#2: - when the property type is string in schema file
errors:
- code: MANIFEST_EXPANSION_USER_ERROR
location: /deployments/setup/manifests/manifest-1571821997285
message: |-
Manifest expansion encountered the following errors: Invalid properties for 'main.jinja':
True is not of type 'string' at ['is-local']
Resource: main-jinja Resource: config
Any idea whats the issue here...
I'm unfamiliar with jinja but from my understanding, environment variables cannot be anything else but strings.
Said this, reading Error#1 I'd conclude that, effectively, the var type has to be string.
Then, at the second error we can clearly see that you are trying to put a boolean into a string.
So yeah, you have to play with true / false as strings.
You can define set the value as a string within the jinja file itself. See this post for some details and this page that provides different methods you can use.
In your case, you can edit the create_cloud_function.jinja file and change:
environmentVariables:
{% for key, value in environmentVariables.items() %}
{{ key }} : {{ value }}
to:
environmentVariables:
{% for key, value in environmentVariables.items() %}
{{ key }} : {{ value|string }}
Once the manifest is fully expanded, the value should be considered a string for the purpose of the API call to the Cloud Functions API
Eventually what I had 2 do was pass IS_LOCAL: '''false'''from the command line and {{ key }} : {{ value }} in my jinja file.
According to this documentation about Using environment variables in Jinja, you should use the following syntax to add an environment var to your templates:
{{ env["deployment"] }} # Jinja
And they show the following example:
- type: compute.v1.instance
name: vm-{{ env["deployment"] }}
properties:
machineType: zones/us-central1-a/machineTypes/f1-micro
serviceAccounts:
- email: {{ env['project_number'] }}-compute#developer.gserviceaccount.com
scopes:
- ...
Given that you are providing the value of is-local from CMD line, and according to this documentation:
Boolean values are case insensitive, so TRUE, true, and True are treated the same.
AND
To specify multiple properties, provide comma-separated key:value pairs. It does not matter in what order you specify the pairs. For example:
`gcloud deployment-manager deployments create my-igm
--template vm_template.jinja
--properties zone:us-central1-a,machineType:n1-standard-1,image:debian-9`
You should use TRUE, true, or True for is-local param.

Django template cache - Dynamic key based on template variable

I need to store template cache using a parameter as prefix.
Es.
With:
{% cache 5 :name:variable1: variable_y variable_z %}
<p> {{variable1}} </p>
{% endcache %}
where variable1 is a string
I need to store:
:1:template.cache:name:"variable1 value":.0ecadd093a225ba502d3f6490e19b4
Actually I obtain:
:1:template.cache:name:variable1:.0ecadd093a225ba502d3f6490e19b4
where variable1 is the variable name.
There's a way?
Thanks
It's possible using Django Advanced Cache Templatetag.
For python 3: pip install django-adv-cache-tag
For python 2 support: pip install 'django-adv-cache-tag<1.0'
Add 'adv_cache_tag' to INSTALLED_APPS=[...]
Add ADV_CACHE_RESOLVE_NAME = True in your settings.py
Change {% load cache %} with {% load adv_cache %}
From documentation:
With ADV_CACHE_RESOLVE_NAME set to True, you can do this if you have
a variable named fragment_name in your context:
{% cache 0 fragment_name obj.pk obj.date_last_updated %}
And if you want to pass a name, you have to surround it by quotes:
{% cache 0 "myobj_main_template" obj.pk obj.date_last_updated %}
With ADV_CACHE_RESOLVE_NAME set to False, the default, the name is
always seen as a string, but if surrounded by quotes, they are removed.

Saltstack load pillar in a for loop

I am developing a automatic proftd installation whit Salt, i wont to get the ftp users from a template but I cant get work the pillar, i initialized the pillar whit the users data and call it into a for loop, but you don't get the pillar user data in the loop.
When i make salt-call pillar.get ftpusers in the minion, the response is:
local:
This is my pillar ftpusers.sls:
ftp-server.ftpusers:
user:
- user: user
- passhash: j2k3hk134123l1234ljh!"ยท$ser
- uuid: 1001
- guid: 1001
- home: /srv/ftp/user
- shel: /bin/false
And this is the for loop:
{% for users in pillar.get('ftpusers', {}).items() %}
/srv/herma-ftp/.ftpusers:
file.managed:
- user: root
- group: root
- mode: 444
- contents:'{{ user }}:{{ args['passhash'] }}:{{args['uuid'] }}:{{ args['guid'] }}::{{ args['home'] }}:{{ args['shel'] }}'
- require:
- file: /srv/herma-ftp
/srv/herma-ftp/{{user}}:
file.directory:
- user: nobody
- group: nobody
- dir_mode: 775
- makedirs: True
- require:
- file: /srv/herma-ftp
- watch:
- file: /srv/herma-ftp
module.run:
- name: file.set_selinux_context
- path: {{ args['home']}}
- type: public_content_t
- unless:
- stat -c %C {{ args['home'] }} |grep -q public_content_t
{% endfor %}
When I make in the minion
salt-call -l debug state.sls herma-ftp-server saltenv=My-enviroment test=True
Don't expect this for because don't can get the pillar data.
Your loop should also look like:
{% for user, args in pillar.get('ftpusers', {}).items() %}
Also, contents argument for a file.managed doesn't support templating. What you need to do is move /srv/herma-ftp/.ftpusers state outside of the loop, and make the loop inside the file template. The final layout of your state should look like:
/srv/herma-ftp/.ftpusers
file.managed:
source: salt://ftpserver/dot.ftpusers
template: jinja
...
...
{% for user, args in pillar.get('ftpusers', {}).items() %}
/srv/herma-ftp/{{user}}:
file.managed:
...
{% endfor %}
And your ftpserver/dot.ftpusers would look like:
{% for user, args in pillar.get('ftpusers', {}).items() %}
{{ user }}:{{ args['passhash'] }}:{{args['uuid'] }}:{{ args['guid'] }}::{{ args['home'] }}:{{ args['shel'] }}
{% endfor %}

SaltStack: Use directory as source only if it exists

I'd like to know if there's a way of running a SaltStack state only if a directory source is defined in the master.
Basically, I want to allow fo users to put certain configuration files in their HOME directory. Most of the users won't have anything special, but some of them might (a custom .vimrc, for instance)
What I'd like to do is execute a file.recurse for the user's HOMEdirectory only if that directory exists in master.
As of now, I have the following:
{% for username in pillar['users'] %}
{{ username }}:
user:
- present
- home: /home/{{ username }}
[ . . . ] # (yadda, yadda, yadda)
# ToDo: Is there a way of not running this AT ALL if the users's directory is
# not present?
/home/{{ username }}:
file.recurse:
- user: {{ username }}
- group: ubuntu
- source:
- salt://users/{{ username }}
- require:
- user: {{ username }}
{% endfor %}
What I want to do is what in the code appears as the ToDo: If there's a directory salt://users/{{ username }} in the salt tree (or in the salt master), then execute the file.recurse with that directory as the source for the file.recurse. Otherwise, just skip that state and use with whatever default content of $HOME is has when a user is created (just do nothing, I mean).
I thought adding an onlyif clause to the file.recurse configuration (something like - onlyif: test -e salt://users/{{ username }} would help... but nopes... I also tried to create an empty directory in salt://users/empty_dir/ and pass it as a default (as described here for a file.managed), but that trick doesn't work with file.recurse (at least not yet):
Thank you in advance.
You could try something similar to
{% for username in pillar['users'] %}
{{ username }}:
user:
- present
- home: /home/{{ username }}
[ . . . ] # (yadda, yadda, yadda)
# NB!!!! POTENTIAL PERF BOTTLENECK
{% if 'users/'+username not in salt['cp.list_master_dirs'](prefix='users') %}
/home/{{ username }}:
file.recurse:
- user: {{ username }}
- group: ubuntu
- source:
- salt://users/{{ username }}
- require:
- user: {{ username }}
{% endif %}
{% endfor %}