Using Sphinx autodoc to document Flask app - python-2.7

I am having problems in using Sphinx to generate documentation for a Flask app. Without going into specific details of the app its basic structure looks as follows.
__all__ = ['APP']
<python 2 imports>
<flask imports>
<custom imports>
APP = None # module-level variable to store the Flask app
<other module level variables>
# App initialisation
def init():
global APP
APP = Flask(__name__)
<other initialisation code>
try:
init()
except Exception as e:
logger.exception(str(e))
#APP.route(os.path.join(<service base url>, <request action>), methods=["POST"])
<request action handler>
if __name__ == '__main__':
init()
APP.run(debug=True, host='0.0.0.0', port=5000)
I've installed Sphinx in a venv along with other packages needed for the web service, and the build folder is within a docs subfolder which looks like this
docs
├── Makefile
├── _build
├── _static
├── _templates
├── conf.py
├── index.rst
├── introduction.rst
└── make.bat
The conf.py was generated by running sphinx-quickstart and it contains the line
autodoc_mock_imports = [<external imports to ignore>]
to ensure that Sphinx will ignore the listed external imports. The index.rst is standard
.. myapp documentation master file, created by
sphinx-quickstart on Fri Jun 16 12:35:40 2017.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to myapp's documentation!
=============================================
.. toctree::
:maxdepth: 2
:caption: Contents:
introduction
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
and I've added an introduction.rst page to document the app members
===================
`myapp`
===================
Oasis Flask app that handles keys requests.
myapp.app
---------------------
.. automodule:: myapp.app
:members:
:undoc-members:
When I run make html in docs I am getting HTML output in the _build subfolder but I get the following warning
WARNING: /path/to/myapp/docs/introduction.rst:10: (WARNING/2) autodoc: failed to import module u'myapp.app'; the following exception was raised:
Traceback (most recent call last):
File "/path/to/myapp/venv/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 657, in import_object
__import__(self.modname)
File "/path/to/myapp/__init__.py", line 4, in <module>
from .app import APP
File "/path/to/myapp/app.py", line 139, in <module>
#APP.route(os.path.join(<service base url>, <request action>), methods=['GET'])
File "/path/to/myapp/venv/lib/python2.7/posixpath.py", line 70, in join
elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'
and I am not seeing the documentation I am expecting to see for the app members like the request handler and the app init method.
I don't know what the problem is, any help would be appreciated.

Try using sphinx-apidoc to automatically generate Sphinx sources that, using the autodoc extension, document a whole package in the style of other automatic API documentation tools. You will need to add 'sphinx.ext.autodoc' to your list of Sphinx extensions in your conf.py, too.

Related

How do I use "discover" to run tests in my "tests" directory?

Using DJango/Python 3.7. I read here -- How do I run all Python unit tests in a directory? that I could use a "discover" command to find tests in a specified directory. I want to have a "tests" folder, so I cretaed one and then ran
(venv) localhost:myproject davea$ python -m unittest discover tests
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/__main__.py", line 18, in <module>
main(module=None)
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/main.py", line 100, in __init__
self.parseArgs(argv)
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/main.py", line 124, in parseArgs
self._do_discovery(argv[2:])
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/main.py", line 244, in _do_discovery
self.createTests(from_discovery=True, Loader=Loader)
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/main.py", line 154, in createTests
self.test = loader.discover(self.start, self.pattern, self.top)
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/loader.py", line 344, in discover
raise ImportError('Start directory is not importable: %r' % start_dir)
ImportError: Start directory is not importable: 'tests'
This is odd to me because I have an (empty) init file ...
(venv) localhost:myproject davea$ ls web/tests/
__init__.py model_tests.py
What else do I need to do to get my test directory recognized?
Edit: Below are the contents of model_tests.py ...
from django.conf import settings
from django.test import TestCase
from django.core import management
def setup():
print("setup")
management.call_command('loaddata', 'test_data.yaml', verbosity=0)
def teardown():
management.call_command('flush', verbosity=0, interactive=False)
class ModelTest(TestCase):
# Verify we can correctly calculate the amount of taxes when we are working
# with a state whose tax rates are defined in our test data
def test_calculate_tax_rate_for_defined_state(self):
state = "MN"
income = 30000
taxes = IndividualTaxBracket.objects.get_taxes_owed(state, income)
print(taxes)
self.assertTrue(taxes > 0, "Failed to calucate taxes owed properly.")
I think you are having some confusion about discover command. According to docs.
Unittest supports simple test discovery. In order to be compatible
with test discovery, all of the test files must be modules or packages
(including namespace packages) importable from the top-level directory
of the project (this means that their filenames must be valid
identifiers).
It means all the test files must be importable from the directory from which you are running the command (directory that holds your web directory). It make this sure, all test files must be in valid python packages (directories containing __init__.py).
Secondly you are running the command python -m unittest discover tests which is wrong. You don't have to add tests at the end. unittests with discover command support 4 options. You can read more about it here.
I have following directory structure.
web
├── __init__.py
└── tests
├── __init__.py
└── test_models.py
And I am running following command.
python3 -m unittest discover
With following results.
...
----------------------------------------------------------------------
Ran 3 tests in 0.000s
OK
First things first: Having an __init__.py is not unusual, because the __init__.py tells python that the directory is a module; Its usual to have an empty __init__.py file. I had the same error, and fixed it by renaming my directory ..
Should a file named tests.py exist as a sibling of tests module, that would probably cause the mentioned ImportError, and removing test.py should fix it.
If still unit tests are not discovered, a couple of question are in order:
1) does the test module contain at least a class derived from django.test.TestCase ?
2) and in that case, does that class contain at least one method whose name starts with "test_"
Please note that the name of any file containing a unit test should start with "test".
So model_test.py will not work; is is generally used to setup some fake Models, but unit tests should reside elsewhere.
You can discover and run tests with this management command:
python manage.py test
or
python manage.py test appname
Is there any particular reason for using python -m unittest discover instead ? I think that could work either, but then you'll have to manually bootstrap the django environment
For completion ...
You already know that form here:
The names of your tests and files have to match a specific pattern in order to be discoverable by discover().
But then you got this error:
"django.core.exceptions.ImproperlyConfigured: Requested settings, but settings are not configured"
That means Django wasn't able to find its settings while running your tests. You can tell where to find settings using an environment variable:
DJANGO_SETTINGS_MODULE='myproyect.settings' python3 -m unittest discover
Reference: https://docs.djangoproject.com/en/2.2/topics/settings/#designating-the-settings
On the other hand ...
You should be running your Django tests with
./manage.py tests
this will search tests automatically using the same mechanism than discover(), and since you would be running a Django command, you will have some benefits against running the Django tests directly.
#Nafees Anwar asked: How does setting environment variable configure settings?
At the very beginning of the model_tests.py file there is the line from django.conf import settings, while creating the settings LazyObject instance, Django will search for that environment variable. Read the code for more detail.
I'll post here a snippet from that code for illustration.
# django.conf module.
ENVIRONMENT_VARIABLE = "DJANGO_SETTINGS_MODULE"
class LazySettings(LazyObject):
"""
A lazy proxy for either global Django settings or a custom settings object.
The user can manually configure settings prior to using them. Otherwise,
Django uses the settings module pointed to by DJANGO_SETTINGS_MODULE.
"""
def _setup(self, name=None):
"""
Load the settings module pointed to by the environment variable. This
is used the first time we need any settings at all, if the user has not
previously configured the settings manually.
"""
settings_module = os.environ.get(ENVIRONMENT_VARIABLE)
if not settings_module:
desc = ("setting %s" % name) if name else "settings"
raise ImproperlyConfigured(
"Requested %s, but settings are not configured. "
"You must either define the environment variable %s "
"or call settings.configure() before accessing settings."
% (desc, ENVIRONMENT_VARIABLE))
self._wrapped = Settings(settings_module)
So if you do:
from django.conf import settings
having that environment variable settled, the statement
settings.configure()
will fail with RuntimeError('Settings already configured.')

flask routes not working on gae application

I have the following directory structure for my GAE project:
endpoints/
insights/
insights.py
init.py
lib/
__init__.py
insights.yaml
so in order to get access to the third libraries within lib folder I add the next code in __init__.py file.
import os
import sys
def add_lib_path():
lib_directory = os.path.dirname(os.path.abspath(__file__))
if lib_directory not in sys.path:
sys.path.insert(0, lib_directory)
and I added this code in the insights.py file before the import statements:
from lib import add_lib_path
add_lib_path()
the problem is that now I can import third libraries correctly but my #app.route('/something', methods=['POST']) are not working.
I send a post request and it returns status 200 but it doesn't go inside my #app.route code, I can actually send any route and it just pass out returning 200 but not data and not error.
My imports look like this:
from lib import add_lib_path
add_lib_path()
from flask import Flask, request
And my code inside #app.route('/something', methods=['POST']) looks like this:
def someDef():
some code ...
return response
my yaml file looks like this:
runtime: python27
api_version: 1
threadsafe: false
service: insights
handlers:
- url: /.*
script: endpoints/insights/insights.py
libraries:
- name: ssl
version: latest
Any suggestions about this? Thanks in advance!
The wildcard URL handler in your app.yaml is intercepting the /something post:
handlers:
- url: /.*
script: application.app
You will need to either map out individual urls in your app, or make unique url set for your insights, like /insights/.* in app.yaml. Either way, you can't have a catch-all url handler in your app.yaml if there are other urls you want to give special treatment to, like sending to a separate service.
ok I just changed in the app.yaml file the threadsafe to true, then I added the script like this endpoints.insights.insights.app, using Python module path (with dots not slash) and in my insights.py file I changed #app.route('/something', methods=['POST']) to #app.route('/insights/something', methods=['POST']) .. I added the complete URL that I defined in the app.yaml file and now it's working.
Thanks to #GAEfan for the help, I'll accept the GAEfan answer because it helped me a lot

python 3 - subtle import error

I am having tricky ImportError when packaging some modules, to try and support both python 2.7 and 3.6. There is nothing inside the code that's fancy to exlude one of these versions, so I thought I would try. The repo is at https://github.com/raamana/neuropredict
It worked fine before on 2.7 alone. To support both 2.7 and 3.6,
After much googling and head-scracting, I thought I figured out how to do that with the following import code at the top of each of the modules in my package:
from sys import version_info
if version_info.major==2 and version_info.minor==7:
import rhst, visualize
from freesurfer import aseg_stats_subcortical, aseg_stats_whole_brain
import config_neuropredict as cfg
elif version_info.major > 2:
from neuropredict import rhst, visualize
from neuropredict.freesurfer import aseg_stats_subcortical, aseg_stats_whole_brain
from neuropredict import config_neuropredict as cfg
else:
raise NotImplementedError('neuropredict supports only 2.7 or Python 3+. Upgrade to Python 3+ is recommended.')
The directory structure is :
$ 22:19:54 miner neuropredict >> tree
.
├── config_neuropredict.py
├── freesurfer.py
├── __init__.py
├── __main__.py
├── model_comparison.py
├── neuropredict.py
├── rhst.py
├── test_rhst.py
└── visualize.py
The init.py looks like this:
__all__ = ['neuropredict', 'rhst', 'visualize', 'freesurfer',
'config_neuropredict', 'model_comparison']
from sys import version_info
if version_info.major==2 and version_info.minor==7:
import neuropredict, config_neuropredict, rhst, visualize, freesurfer, model_comparison
elif version_info.major > 2:
from neuropredict import neuropredict, config_neuropredict, rhst, visualize, freesurfer, model_comparison
else:
raise NotImplementedError('neuropredict supports only 2.7 or Python 3+. Upgrade to Python 3+ is recommended.')
So when I run the unit tests locally or on CI, it works fine with above import mechanism.
$ 22:19:25 miner neuropredict >> pytest test_rhst.py
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.6.1, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: /data1/strother_lab/praamana/neuropredict, inifile:
plugins: hypothesis-3.23.2
collected 1 item
test_rhst.py .
============================================================================================ 1 passed in 8.73 seconds =============================================================================================
$ 22:19:42 miner neuropredict >>
However, when I run neuropredict.py directly, it thrown an error
$ 22:19:57 miner neuropredict >> python ./neuropredict.py
Traceback (most recent call last):
File "neuropredict.py", line 23, in <module>
from neuropredict import rhst, visualize
File "/data1/strother_lab/praamana/neuropredict/neuropredict/neuropredict.py", line 23, in <module>
from neuropredict import rhst, visualize
ImportError: cannot import name 'rhst'
$ 22:29:39 miner neuropredict >> pwd
/data1/strother_lab/praamana/neuropredict/neuropredict
$ 22:29:43 miner neuropredict >>
This is killing me - I need figure out where I am making a mistake, or if I am doing something sacrilegious.
Question 1: why am I not able to
python ./neuropredict.py
and get some parser help, when the test scripts can import it successfully? This used to work in many other scenarios. Driving me crazy as I can't understand what the hell is going on.
This is what I need to be able to achieve:
import config_neuropredict.py into all other modules
import config_neuropredict, rhst, visualize, freesurfer, model_comparison into the neuropredict.py module
make this reliably work in Python 3+ (2.7 support not important)
understand python's behaviour during Question 1 above.
If you need more details or into the code, I would appreciate if you can quickly head over to this public repo: https://github.com/raamana/neuropredict

How to get ansible.module_utils to resolve my custom directory

Certain variables in ansible.cfg seem to not be taking affect.
Ansible 2.2.1.0
Python 2.7.10
Mac OS Version 10.12.5
We have created a custom class that will be used in our custom Ansible module. The class lives here:
/sites/utils/local/ansible/agt_module_utils/ldapData.py
/sites/utils/local/ansible/agt_module_utils/init.py
The class inside ldapData.py is named:
class ldapDataClass(object):
In all cases, init.py is a zero byte file.
Our Ansible module is located here:
/sites/utils/local/ansible/agt_modules/init.py
/sites/utils/local/ansible/agt_modules/agtWeblogic.py
The import statement in atgWeblogic looks as follows:
from ansible.module_utils.ldapData import ldapDataClass
I have also tried:
from ldapData import ldapDataClass
The config file has the following lines:
library = /sites/utils/local/ansible/att_modules
module_utils = /sites/utils/local/ansible/att_module_utils
When running our module, the modules directory IS resolved but the module_utils directory is not resolved. When the include is "from ansible.module_utils.ldapData import ldapDataClass" the failure is before ansible even connects to the remote machine. When the include is "from ldapData import ldapDataClass" the failure is on the remote machine. Below I am showing the failure "on the remote machine" (scroll right to see full error):
nverkland#local>ansible ecomtest37 -m agtWeblogic -a "action=stop instances=tst37-shop-main" -vvv
ecomtest37 | FAILED! => {
"changed": false,
"failed": true,
"invocation": {
"module_name": "agtWeblogic"
},
"module_stderr": "",
"module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_FwHCmh/ansible_module_attWeblogic.py\", line 63, in <module>\r\n from ldapData import ldapDataClass\r\nImportError: No module named ldapData\r\n",
"msg": "MODULE FAILURE"
}
If I move the ldapData.py file into the "ansible installed" module_utils directory (/Library/Python/2.7/site-packages/ansible/module_utils/ on my Mac) the module runs fine. What have I done incorrectly in my config file that has prevented the use of my "custom" module_utils directory?
Thanks.
module_utils path is configurable since Ansible 2.3: changelog, PR.
I guess you have to upgrade or refactor the module.
Update: working example
Project tree:
.
├── ansible
│   └── ansible.cfg
├── module_utils
│   └── mycommon.py
└── modules
└── test_module.py
ansible.cfg:
[defaults]
library = ../modules
module_utils = ../module_utils
mycommon.py:
class MyCommonClass(object):
#staticmethod
def hello():
return 'world'
test_module.py:
#!/usr/bin/python
from ansible.module_utils.basic import *
from ansible.module_utils.mycommon import MyCommonClass
module = AnsibleModule(
argument_spec = dict()
)
module.exit_json(changed=True, hello=MyCommonClass.hello())
Execution:
$ ansible localhost -m test_module
[WARNING]: Host file not found: /etc/ansible/hosts
[WARNING]: provided hosts list is empty, only localhost is available
localhost | SUCCESS => {
"changed": true,
"hello": "world"
}
Ansible version:
$ ansible --version
ansible 2.3.1.0
config file = /<masked>/ansible/ansible.cfg
configured module search path = [u'../modules']
python version = 2.7.10 (default, Feb 7 2017, 00:08:15) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)]
Konstantin's notes helped. I managed to solve the riddle.
In my config file I had:
module_utils = /sites/utils/local/ansible/agt_module_utils
This causes Ansible to think that my class is found at:
import ansible.agt_module_utils.ldapData (working)
until now I was attempting to find the class at:
import ansible.module_utils.ldapData (broken)

Python packaging can't import class handler

I have two packages (diretories ) in my Python project
src
/textmining
mining.py...def mining():#...
__init.py....__all__ = ["mining"]
/crawler
crawler.py
in crawler.py I use the mining class
mining=mining()
main.py
__init__.py
my main.py is as follow:
scrapy_command = 'scrapy runspider {spider_name} -a crawling_level="{param_1}"'.format(spider_name='crawler/crawler.py',
param_1=crawling_level)
process = subprocess.Popen(scrapy_command, shell=True)
when I run crawler, it prompts
runspider: error: Unable to load 'Crawler.py': cannot import name mining
You need an __init__.py in every folder that's a part of the same package module.
src
__init__.py
/textmining
__init__.py
mining.py
/crawler
__init__.py
crawler.py
For simplicity, you should add a main.py in the src folder and call the function you want to start your program with from there as it's fairly difficult to import modules from sibling directories if you start your script in a non-root directory.
main.py
from crawler import crawler
crawler.start_function()
crawler.py
from src.textmining import mining
miner = mining()
Without turning everything into a python module you'd have to import a folder into the current script or __init__.py module by adding to path:
# In crawler.py
import sys
import os
sys.path.append(os.path.abspath('../textmining'))
import mining
However, messing around with the path requires you to keep in mind what you've done and may not be a thing you desire.