Introduction

When we start new Python projects, we usually make a git repository with README, setup.py, project_module, docs, and tests in the top-level. The project_module can be a simple python script or package. The docs folder is typically a Sphinx site and tests might contain unittest suites:

project/
  README
  setup.py
  docs/
    Makefile
    index.rst
    config.rst
    ...
  tests/
    __init__.py
    ...
  project_name/
    __init__.py
    ...

Running a few doctests is pretty easy out-of-the-box with Python. If we keep setup.py up to date, we can run all of our tests at once.

%find project_name -name '*.rst' | xargs python -m doctest

%(cd docs && make doctest)

%python setup.py test

After hacking for a while, we end up with some unittest TestCase definitions in tests/, doctests in project_name docstrings, more unittests scattered throughout project_name, and literal python embedded in docs/. Finding all of the tests and running them takes some thought. Once the project gets big enough, running all the tests is thorough, but it takes a while. Quickly running only a specific subset of the tests can be challenging.

Using pytest

To quickly run tests, we made a script called pytest. It's part of the md package. Install it and use it like this:

%git clone git://github.com/thisismedium/md.git
...

%cd md
%python setup.py develop  # or use "install" to for a permanent installation...
...

%pytest --help
Usage: pytest [options] module ...

Options:
  -h, --help            show this help message and exit
  -v                    Verbosity.
  --doc-ext=DOCEXT      Doctest filename extensions
  --doc-folders=DOCFOLDERS
                        Folders relative to the top-level package in which to
                        search for doctest files
  --unit-ext=UNITEXT    Doctest filename extensions

%pytest -vv md
...

pytest can be run against any module, sub-module, or doctest file. We made it very greedy. In the example above, running pytest md causes it to search through the filesystem starting at dirname(md.__file__). It loads all of the modules it finds and turns them into unittest or doctest test suites. It also looks for text files ending in .rst or .txt and turns them into docfile suites.

Next, it tries to find a docs/ folder in the parent directory of the top-level package (md in this example). While developing we might use setup.py develop or made a symlink from site-packages to our project in version control; pytest uses realpath() to find the parent directory. If a docs/ folder is found, it looks for .rst or .txt files prefixed with the module name being tested. It turns directory separators into dots, so naming.files.like.this.rst or naming/files/like/this.rst works equally well. In this example it will try to test docs/md*.rst, docs/md*.txt, docs/md/*.rst and docs/md/*.txt.

Once it finds all of the modules and doctest files it can, they're all added to a TestSuite and run.

Conclusion

When unit tests are easy to run, we like writing them more. We made pytest to stay out of our way and do what we mean. Running tests is easy. pytest uses md.test to find and run tests. See the md documentation for more information and examples.

After we made pytest, we found py.test. It's a tool in the same spirit and has a good plugin system and ways to run tests on multiple cores and even remotely. The py.test module is more featureful, but our pytest script finds tests more aggressively. A future release of pytest might be a plugin for py.test.