Is it possible to extend Sphinx automodule to domains other than Python?
Question: I'm looking to use Sphinx to document VHDL source code. Ideally I'd like to be
able to take a VHDL type like this:

    
    
    type T_SDRAM_REQ is record
        req     : STD_LOGIC;
        wr      : STD_LOGIC;
        address : STD_LOGIC_VECTOR;
        wr_data : STD_LOGIC_VECTOR;
        wr_ben  : STD_LOGIC_VECTOR;
    end record T_SDRAM_REQ;
    

And use a RST directive something like this:

    
    
    .. vhdl:type:: sdram_pack.T_SDRAM_REQ is record
        :members:
    

To extract all of the fields from the source code and RST-ify them for me.

I've created a Sphinx domain, but it's dawning on me that this alone is not
going to be enough - that's just a bunch of custom directives really. What I
actually want is something akin to autoclass or automodule, which scans Python
source files to generate directives.

However as far as I can tell the Sphinx automodule functionality is just for
Python. Is it possible to extend Sphinx to include similar functionality for
other languages? In VHDL that would probably be called autopackage or
autoentity, in C++ I guess autonamespace or a different autoclass? Could I
somehow add a `vhdl:autopackage::` directive to my domain? From what I can
tell from the Sphinx source I don't think the automodule directive is part of
the Python domain.


Answer: The answer to my own question is: yes. I've managed to do it, but it wasn't
easy and the results are far from perfect.

While the Sphinx Domain API has been set up with some generic base classes and
Python-specific subclasses, the same can't be said of autodoc. Some of the
Python autodoc classes can be used as base classes, but need a lot of
overrides.

The components of my autodoc system are:

  * A new directive `VHDLAutoDirective`, A subclass of `AutoDirective`, it maintains separate registries for documenters and special attrgetters, and trims "vhdl:auto" from the start of the directive name instead of just "auto". Like to original this calls object-specific documenter.
  * New documenters. One generic VHDL documenter base class `VHDLDocumenter`, then subclasses for each VHDL object. These documenters do all of the heavy lifting, taking options and content from the directives, and parsing VHDL to generate content. The key issue here is that Python autodoc relies on the modules it's documenting being installed. As Sphinx is written in Python it's very easy to `import` those installed modules and extract all required info that way, for example `__doc__` can be used to extract doctrings. For any other language you're going to have to first find the file containing the object you want to document, then parse it. To solve the first problem I added a `current-file` directive to specify a file for all subsequent auto directives, and a `file` option to my documenter base class to allow files to be specified per-directive. This is a bit clunky, as the file's paths are relative to the base of my repositiory and assumes that Sphinx is run there - it won't work if Sphinx is run in a sub-directory. For the second I wrote a rudimentary tokeniser and parser, before thinking it might be better to just copy the raw source code into a `code-block` directive - now I give the option of doing either.
  * An `add_autodocumenter` function in my domain, which registers a directive in the domain, then imports my autodoc module and calls an `add_documenter` function to register the documenter. The autodoc modules's setup function then calls `add_autodocumenter` on each object documenter with the auto directive. This is similar to what Python autodoc does, but the Python version registers its directives with the app rather than a domain.

There's still a lot of room for improvement, but at least it serves as a proof
of concept that it is possible to do this.

