How to get ansible.module_utils to resolve my custom directory - python-2.7

Certain variables in ansible.cfg seem to not be taking affect.
Ansible 2.2.1.0
Python 2.7.10
Mac OS Version 10.12.5
We have created a custom class that will be used in our custom Ansible module. The class lives here:
/sites/utils/local/ansible/agt_module_utils/ldapData.py
/sites/utils/local/ansible/agt_module_utils/init.py
The class inside ldapData.py is named:
class ldapDataClass(object):
In all cases, init.py is a zero byte file.
Our Ansible module is located here:
/sites/utils/local/ansible/agt_modules/init.py
/sites/utils/local/ansible/agt_modules/agtWeblogic.py
The import statement in atgWeblogic looks as follows:
from ansible.module_utils.ldapData import ldapDataClass
I have also tried:
from ldapData import ldapDataClass
The config file has the following lines:
library = /sites/utils/local/ansible/att_modules
module_utils = /sites/utils/local/ansible/att_module_utils
When running our module, the modules directory IS resolved but the module_utils directory is not resolved. When the include is "from ansible.module_utils.ldapData import ldapDataClass" the failure is before ansible even connects to the remote machine. When the include is "from ldapData import ldapDataClass" the failure is on the remote machine. Below I am showing the failure "on the remote machine" (scroll right to see full error):
nverkland#local>ansible ecomtest37 -m agtWeblogic -a "action=stop instances=tst37-shop-main" -vvv
ecomtest37 | FAILED! => {
"changed": false,
"failed": true,
"invocation": {
"module_name": "agtWeblogic"
},
"module_stderr": "",
"module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_FwHCmh/ansible_module_attWeblogic.py\", line 63, in <module>\r\n from ldapData import ldapDataClass\r\nImportError: No module named ldapData\r\n",
"msg": "MODULE FAILURE"
}
If I move the ldapData.py file into the "ansible installed" module_utils directory (/Library/Python/2.7/site-packages/ansible/module_utils/ on my Mac) the module runs fine. What have I done incorrectly in my config file that has prevented the use of my "custom" module_utils directory?
Thanks.

module_utils path is configurable since Ansible 2.3: changelog, PR.
I guess you have to upgrade or refactor the module.
Update: working example
Project tree:
.
├── ansible
│   └── ansible.cfg
├── module_utils
│   └── mycommon.py
└── modules
└── test_module.py
ansible.cfg:
[defaults]
library = ../modules
module_utils = ../module_utils
mycommon.py:
class MyCommonClass(object):
#staticmethod
def hello():
return 'world'
test_module.py:
#!/usr/bin/python
from ansible.module_utils.basic import *
from ansible.module_utils.mycommon import MyCommonClass
module = AnsibleModule(
argument_spec = dict()
)
module.exit_json(changed=True, hello=MyCommonClass.hello())
Execution:
$ ansible localhost -m test_module
[WARNING]: Host file not found: /etc/ansible/hosts
[WARNING]: provided hosts list is empty, only localhost is available
localhost | SUCCESS => {
"changed": true,
"hello": "world"
}
Ansible version:
$ ansible --version
ansible 2.3.1.0
config file = /<masked>/ansible/ansible.cfg
configured module search path = [u'../modules']
python version = 2.7.10 (default, Feb 7 2017, 00:08:15) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)]

Konstantin's notes helped. I managed to solve the riddle.
In my config file I had:
module_utils = /sites/utils/local/ansible/agt_module_utils
This causes Ansible to think that my class is found at:
import ansible.agt_module_utils.ldapData (working)
until now I was attempting to find the class at:
import ansible.module_utils.ldapData (broken)

Related

How to use DBT with AWS Managed Airflow?

hope you are doing well.
I wanted to check if anyone has get up and running with dbt in aws mwaa airflow.
I have tried without success this one and this python packages but fails for some reason or another (can't find the dbt path, etc).
Did anyone has managed to use MWAA (Airflow 2) and DBT without having to build a docker image and placing it somewhere?
Thank you!
I've managed to solve this by doing the following steps:
Add dbt-core==0.19.1 to your requirements.txt
Add DBT cli executable into plugins.zip
#!/usr/bin/env python3
# EASY-INSTALL-ENTRY-SCRIPT: 'dbt-core==0.19.1','console_scripts','dbt'
__requires__ = 'dbt-core==0.19.1'
import re
import sys
from pkg_resources import load_entry_point
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(
load_entry_point('dbt-core==0.19.1', 'console_scripts', 'dbt')()
)
And from here you have two options:
Setting dbt_bin operator argument to /usr/local/airflow/plugins/dbt
Add /usr/local/airflow/plugins/ to the $PATH by following the docs
Environment variable setter example:
from airflow.plugins_manager import AirflowPlugin
import os
os.environ["PATH"] = os.getenv(
"PATH") + ":/usr/local/airflow/.local/lib/python3.7/site-packages:/usr/local/airflow/plugins/"
class EnvVarPlugin(AirflowPlugin):
name = 'env_var_plugin'
The plugins zip content:
plugins.zip
├── dbt (DBT cli executable)
└── env_var_plugin.py (environment variable setter)
Using the pypi airflow-dbt-python package has simplified the setup of dbt_ to MWAA for us, as it avoids needing to amend PATH environment variables in the plugins file. However, I've yet to have a successful dbt_ run via either airflow-dbt or airflow-dbt-python packages, as MWAA worker seems to be a read only filesystem, so as soon as dbt_ tries to compile to the target directory, the following error occurs:
File "/usr/lib64/python3.7/os.py", line 223, in makedirs
mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/usr/local/airflow/dags/dbt/target'
This is how I managed to do it:
#dag(**default_args)
def dbt_dag():
#task()
def run_dbt():
from dbt.main import handle_and_check
os.environ["DBT_TARGET_DIR"] = "/usr/local/airflow/tmp/target"
os.environ["DBT_LOG_DIR"] = "/usr/local/airflow/tmp/logs"
os.environ["DBT_PACKAGE_DIR"] = "/usr/local/airflow/tmp/packages"
succeeded = True
try:
args = ['run', '--whatever', 'bla']
results, succeeded = handle_and_check(args)
print(results, succeeded)
except SystemExit as e:
if e.code != 0:
raise e
if not succeeded:
raise Exception("DBT failed")
note that my dbt_project.yml has the following paths, this is to avoid os exception when trying to write to read only paths:
target-path: "{{ env_var('DBT_TARGET_DIR', 'target') }}" # directory which will store compiled SQL files
log-path: "{{ env_var('DBT_LOG_DIR', 'logs') }}" # directory which will store dbt logs
packages-install-path: "{{ env_var('DBT_PACKAGE_DIR', 'packages') }}" # directory which will store dbt packages
Combining the answer from #Yonatan Kiron & #Ofer Helman works for me.
I just need to fix these 3 files:
requiremnt.txt
plugins.zip
dbt_project.yml
My requirements.txt I specify the version I want, and looks like this:
airflow-dbt==0.4.0
dbt-core==1.0.1
dbt-redshift==1.0.0
Note that, as of v1.0.0, pip install dbt is no longer supported and will raise an explicit error. Since v0.13, the PyPi package named dbt was a simple "pass-through" of dbt-core. (refer https://docs.getdbt.com/dbt-cli/install/pip#install-dbt-core-only)
For my plugins.zip I add a file env_var_plugin.py that looks like this
from airflow.plugins_manager import AirflowPlugin
import os
os.environ["DBT_LOG_DIR"] = "/usr/local/airflow/tmp/logs"
os.environ["DBT_PACKAGE_DIR"] = "/usr/local/airflow/tmp/dbt_packages"
os.environ["DBT_TARGET_DIR"] = "/usr/local/airflow/tmp/target"
class EnvVarPlugin(AirflowPlugin):
name = 'env_var_plugin'
And finally I add this in my dbt_project.yml
log-path: "{{ env_var('DBT_LOG_DIR', 'logs') }}" # directory which will store dbt logs
packages-install-path: "{{ env_var('DBT_PACKAGE_DIR', 'dbt_packages') }}" # directory which will store dbt packages
target-path: "{{ env_var('DBT_TARGET_DIR', 'target') }}" # directory which will store compiled SQL files
And as stated in the airflow-dbt github, (https://github.com/gocardless/airflow-dbt#amazon-managed-workflows-for-apache-airflow-mwaa) configure the dbt task like below:
dbt_bin='/usr/local/airflow/.local/bin/dbt',
profiles_dir='/usr/local/airflow/dags/{DBT_FOLDER}/',
dir='/usr/local/airflow/dags/{DBT_FOLDER}/'

How do I use "discover" to run tests in my "tests" directory?

Using DJango/Python 3.7. I read here -- How do I run all Python unit tests in a directory? that I could use a "discover" command to find tests in a specified directory. I want to have a "tests" folder, so I cretaed one and then ran
(venv) localhost:myproject davea$ python -m unittest discover tests
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/__main__.py", line 18, in <module>
main(module=None)
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/main.py", line 100, in __init__
self.parseArgs(argv)
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/main.py", line 124, in parseArgs
self._do_discovery(argv[2:])
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/main.py", line 244, in _do_discovery
self.createTests(from_discovery=True, Loader=Loader)
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/main.py", line 154, in createTests
self.test = loader.discover(self.start, self.pattern, self.top)
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/loader.py", line 344, in discover
raise ImportError('Start directory is not importable: %r' % start_dir)
ImportError: Start directory is not importable: 'tests'
This is odd to me because I have an (empty) init file ...
(venv) localhost:myproject davea$ ls web/tests/
__init__.py model_tests.py
What else do I need to do to get my test directory recognized?
Edit: Below are the contents of model_tests.py ...
from django.conf import settings
from django.test import TestCase
from django.core import management
def setup():
print("setup")
management.call_command('loaddata', 'test_data.yaml', verbosity=0)
def teardown():
management.call_command('flush', verbosity=0, interactive=False)
class ModelTest(TestCase):
# Verify we can correctly calculate the amount of taxes when we are working
# with a state whose tax rates are defined in our test data
def test_calculate_tax_rate_for_defined_state(self):
state = "MN"
income = 30000
taxes = IndividualTaxBracket.objects.get_taxes_owed(state, income)
print(taxes)
self.assertTrue(taxes > 0, "Failed to calucate taxes owed properly.")
I think you are having some confusion about discover command. According to docs.
Unittest supports simple test discovery. In order to be compatible
with test discovery, all of the test files must be modules or packages
(including namespace packages) importable from the top-level directory
of the project (this means that their filenames must be valid
identifiers).
It means all the test files must be importable from the directory from which you are running the command (directory that holds your web directory). It make this sure, all test files must be in valid python packages (directories containing __init__.py).
Secondly you are running the command python -m unittest discover tests which is wrong. You don't have to add tests at the end. unittests with discover command support 4 options. You can read more about it here.
I have following directory structure.
web
├── __init__.py
└── tests
├── __init__.py
└── test_models.py
And I am running following command.
python3 -m unittest discover
With following results.
...
----------------------------------------------------------------------
Ran 3 tests in 0.000s
OK
First things first: Having an __init__.py is not unusual, because the __init__.py tells python that the directory is a module; Its usual to have an empty __init__.py file. I had the same error, and fixed it by renaming my directory ..
Should a file named tests.py exist as a sibling of tests module, that would probably cause the mentioned ImportError, and removing test.py should fix it.
If still unit tests are not discovered, a couple of question are in order:
1) does the test module contain at least a class derived from django.test.TestCase ?
2) and in that case, does that class contain at least one method whose name starts with "test_"
Please note that the name of any file containing a unit test should start with "test".
So model_test.py will not work; is is generally used to setup some fake Models, but unit tests should reside elsewhere.
You can discover and run tests with this management command:
python manage.py test
or
python manage.py test appname
Is there any particular reason for using python -m unittest discover instead ? I think that could work either, but then you'll have to manually bootstrap the django environment
For completion ...
You already know that form here:
The names of your tests and files have to match a specific pattern in order to be discoverable by discover().
But then you got this error:
"django.core.exceptions.ImproperlyConfigured: Requested settings, but settings are not configured"
That means Django wasn't able to find its settings while running your tests. You can tell where to find settings using an environment variable:
DJANGO_SETTINGS_MODULE='myproyect.settings' python3 -m unittest discover
Reference: https://docs.djangoproject.com/en/2.2/topics/settings/#designating-the-settings
On the other hand ...
You should be running your Django tests with
./manage.py tests
this will search tests automatically using the same mechanism than discover(), and since you would be running a Django command, you will have some benefits against running the Django tests directly.
#Nafees Anwar asked: How does setting environment variable configure settings?
At the very beginning of the model_tests.py file there is the line from django.conf import settings, while creating the settings LazyObject instance, Django will search for that environment variable. Read the code for more detail.
I'll post here a snippet from that code for illustration.
# django.conf module.
ENVIRONMENT_VARIABLE = "DJANGO_SETTINGS_MODULE"
class LazySettings(LazyObject):
"""
A lazy proxy for either global Django settings or a custom settings object.
The user can manually configure settings prior to using them. Otherwise,
Django uses the settings module pointed to by DJANGO_SETTINGS_MODULE.
"""
def _setup(self, name=None):
"""
Load the settings module pointed to by the environment variable. This
is used the first time we need any settings at all, if the user has not
previously configured the settings manually.
"""
settings_module = os.environ.get(ENVIRONMENT_VARIABLE)
if not settings_module:
desc = ("setting %s" % name) if name else "settings"
raise ImproperlyConfigured(
"Requested %s, but settings are not configured. "
"You must either define the environment variable %s "
"or call settings.configure() before accessing settings."
% (desc, ENVIRONMENT_VARIABLE))
self._wrapped = Settings(settings_module)
So if you do:
from django.conf import settings
having that environment variable settled, the statement
settings.configure()
will fail with RuntimeError('Settings already configured.')

Using Sphinx autodoc to document Flask app

I am having problems in using Sphinx to generate documentation for a Flask app. Without going into specific details of the app its basic structure looks as follows.
__all__ = ['APP']
<python 2 imports>
<flask imports>
<custom imports>
APP = None # module-level variable to store the Flask app
<other module level variables>
# App initialisation
def init():
global APP
APP = Flask(__name__)
<other initialisation code>
try:
init()
except Exception as e:
logger.exception(str(e))
#APP.route(os.path.join(<service base url>, <request action>), methods=["POST"])
<request action handler>
if __name__ == '__main__':
init()
APP.run(debug=True, host='0.0.0.0', port=5000)
I've installed Sphinx in a venv along with other packages needed for the web service, and the build folder is within a docs subfolder which looks like this
docs
├── Makefile
├── _build
├── _static
├── _templates
├── conf.py
├── index.rst
├── introduction.rst
└── make.bat
The conf.py was generated by running sphinx-quickstart and it contains the line
autodoc_mock_imports = [<external imports to ignore>]
to ensure that Sphinx will ignore the listed external imports. The index.rst is standard
.. myapp documentation master file, created by
sphinx-quickstart on Fri Jun 16 12:35:40 2017.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to myapp's documentation!
=============================================
.. toctree::
:maxdepth: 2
:caption: Contents:
introduction
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
and I've added an introduction.rst page to document the app members
===================
`myapp`
===================
Oasis Flask app that handles keys requests.
myapp.app
---------------------
.. automodule:: myapp.app
:members:
:undoc-members:
When I run make html in docs I am getting HTML output in the _build subfolder but I get the following warning
WARNING: /path/to/myapp/docs/introduction.rst:10: (WARNING/2) autodoc: failed to import module u'myapp.app'; the following exception was raised:
Traceback (most recent call last):
File "/path/to/myapp/venv/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 657, in import_object
__import__(self.modname)
File "/path/to/myapp/__init__.py", line 4, in <module>
from .app import APP
File "/path/to/myapp/app.py", line 139, in <module>
#APP.route(os.path.join(<service base url>, <request action>), methods=['GET'])
File "/path/to/myapp/venv/lib/python2.7/posixpath.py", line 70, in join
elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'
and I am not seeing the documentation I am expecting to see for the app members like the request handler and the app init method.
I don't know what the problem is, any help would be appreciated.
Try using sphinx-apidoc to automatically generate Sphinx sources that, using the autodoc extension, document a whole package in the style of other automatic API documentation tools. You will need to add 'sphinx.ext.autodoc' to your list of Sphinx extensions in your conf.py, too.

python ConfigParser.NoSectionError: - not working on server

Python 2.7
Django 1.10
settings.ini file(located at "/opts/myproject/settings.ini"):
[settings]
DEBUG: True
SECRET_KEY: '5a88V*GuaQgAZa8W2XgvD%dDogQU9Gcc5juq%ax64kyqmzv2rG'
On my django settings file I have:
import os
from ConfigParser import RawConfigParser
config = RawConfigParser()
config.read('/opts/myproject/settings.ini')
SECRET_KEY = config.get('settings', 'SECRET_KEY')
DEBUG = config.get('settings', 'DEBUG')
The setup works fine locally, but when I deploy to my server I get the following error if I try run any django management commands:
ConfigParser.NoSectionError: No section: 'settings'
If I go into Python shell locally I type in the above imports and read the file I get back:
['/opts/myproject/settings.ini']
On server I get back:
[]
I have tried changing "confif.read()" to "config.readfp()" as suggested on here but it didn't work.
Any help or advice is appreciated.

uwsgi: no app loaded. going in full dynamic mode

In my uwsgi config, I have these options:
[uwsgi]
chmod-socket = 777
socket = 127.0.0.1:9031
plugins = python
pythonpath = /adminserver/
callable = app
master = True
processes = 4
reload-mercy = 8
cpu-affinity = 1
max-requests = 2000
limit-as = 512
reload-on-as = 256
reload-on-rss = 192
no-orphans
vacuum
My app structure looks like this:
/adminserver
app.py
...
My app.py has these bits of code:
app = Flask(__name__)
...
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5003, debug=True)
The result is that when I try to curl my server, I get this error:
Wed Sep 11 23:28:56 2013 - added /adminserver/ to pythonpath.
Wed Sep 11 23:28:56 2013 - *** no app loaded. going in full dynamic mode ***
Wed Sep 11 23:28:56 2013 - *** uWSGI is running in multiple interpreter mode ***
What do the module and callable options do? The docs say:
module, wsgi Argument: string
Load a WSGI module as the application. The module (sans .py) must be
importable, ie. be in PYTHONPATH.
This option may be set with -w from the command line.
callable Argument: string Default: application
Set default WSGI callable name.
Module
A module in Python maps to a file on disk - when you have a directory like this:
/some-dir
module1.py
module2.py
If you start up a python interpreter while the current working directory is /some-dir you will be able to import each of the modules:
some-dir$ python
>>> import module1, module2
# Module1 and Module2 are now imported
Python searches sys.path (and a few other things, see the docs on import for more information) for a file that matches the name you are trying to import. uwsgi uses Python's import process under the covers to load the module that contains your WSGI application.
Callable
The WSGI PEPs (333 and 3333) specify that a WSGI application is a callable that takes two arguments and returns an iterable that yields bytestrings:
# simple_wsgi.py
# The simplest WSGI application
HELLO_WORLD = b"Hello world!\n"
def simple_app(environ, start_response):
"""Simplest possible application object"""
status = '200 OK'
response_headers = [('Content-type', 'text/plain')]
start_response(status, response_headers)
return [HELLO_WORLD]
uwsgi needs to know the name of a symbol inside of your module that maps to the WSGI application callable, so it can pass in the environment and the start_response callable - essentially, it needs to be able to do the following:
wsgi_app = getattr(simple_wsgi, 'simple_app')
TL;PC (Too Long; Prefer Code)
A simple parallel of what uwsgi is doing:
# Use `module` to know *what* to import
import simple_wsgi
# construct request environment from user input
# create a callable to pass for start_response
# and then ...
# use `callable` to know what to call
wsgi_app = getattr(simple_wsgi, 'simple_app')
# and then call it to respond to the user
response = wsgi_app(environ, start_response)
For anyone else having this problem, if you are sure your configuration is correct, you should check your uWSGI version.
Ubuntu 12.04 LTS provides 1.0.3. Removing that and using pip to install 2.0.4 resolved my issues.
First, check your configuration whether is correct.
my uwsgi.ini configuration:
[uwsgi]
chdir=/home/air/repo/Qiy
uid=nobody
gid=nobody
module=Qiy.wsgi:application
socket=/home/air/repo/Qiy/uwsgi.sock
master=true
workers=5
pidfile=/home/air/repo/Qiy/uwsgi.pid
vacuum=true
thunder-lock=true
enable-threads=true
harakiri=30
post-buffering=4096
daemonize=/home/air/repo/Qiy/uwsgi.log
then use uwsgi --ini uwsgi.ini to run uwsgi.
if not work, you rm -rf the venv directory, and re-initial the venv, and re-try my step.
I re-initial the venv solved my issue, seems the problem is when I pip3 install some packages of requirements.txt, and upgrade the pip, then install uwsgi package. so, I delete the venv, and re-initial my virtual environment.