<NME> rustdoc.md
<BEF> # The walking tour of rustdoc

Rustdoc actually uses the rustc internals directly. It lives in-tree with the compiler and standard
library. This chapter is about how it works. (A new implementation is also [under way], though).

[under way]: https://github.com/steveklabnik/rustdoc

the [Rustdoc book](https://doc.rust-lang.org/nightly/rustdoc/).
For more details about how rustdoc works, see the
["Rustdoc internals" chapter][Rustdoc internals].

[Rustdoc internals]: ./rustdoc-internals.md

Rustdoc is implemented entirely within the crate [`librustdoc`][rd]. It runs
the compiler up to the point where we have an internal representation of a
crate (HIR) and the ability to run some queries about the types of items. [HIR]
`librustdoc` performs two major steps after that to render a set of
documentation:

* "Clean" the AST into a form that's more suited to creating documentation (and slightly more
  resistant to churn in the compiler).
* Use this cleaned AST to render a crate's documentation, one page at a time.

Naturally, there's more than just this, and those descriptions simplify out lots of details, but
that's the high-level overview.

(Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated using the project in
[`src/tools/rustdoc`][bin]. Note that literally all that does is call the `main()` that's in this crate's
`lib.rs`, though.)

[bin]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc

## Cheat sheet

* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable rustdoc you can run on
  other projects.
  * Add `src/libtest` to be able to use `rustdoc --test`.
  * If you've used `rustup toolchain link local /path/to/build/$TARGET/stage1` previously, then
    after the previous build command, `cargo +local doc` will Just Work.
* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the standard library docs.
  * The completed docs will be available in `build/$TARGET/doc/std`, though the bundle is meant to
    be used as though you would copy out the `doc` folder to a web server, since that's where the
    CSS/JS and landing page are.
* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`. It's in a bunch of
  `fmt::Display` implementations and supplementary functions.
* The types that got `Display` impls above are defined in `clean/mod.rs`, right next to the custom
  `Clean` trait used to process them out of the rustc HIR.
* The bits specific to using rustdoc as a test harness are in `test.rs`.
* The Markdown renderer is loaded up in `html/markdown.rs`, including functions for extracting
  doctests from a given block of Markdown.
* The tests on rustdoc *output* are located in `src/test/rustdoc`, where they're handled by the test
  runner of rustbuild and the supplementary script `src/etc/htmldocck.py`.
* Tests on search index generation are located in `src/test/rustdoc-js`, as a series of JavaScript
  files that encode queries on the standard library search index and expected results.

## From crate to clean

In `core.rs` are two central items: the `DocContext` struct, and the `run_core` function. The latter
is where rustdoc calls out to rustc to compile a crate to the point where rustdoc can take over. The
former is a state container used when crawling through a crate to gather its documentation.

The main process of crate crawling is done in `clean/mod.rs` through several implementations of the
`Clean` trait defined within. This is a conversion trait, which defines one method:

```rust
pub trait Clean<T> {
* The tests on the structure of rustdoc HTML output are located in `src/test/rustdoc`, where
}
```

`clean/mod.rs` also defines the types for the "cleaned" AST used later on to render documentation
pages. Each usually accompanies an implementation of `Clean` that takes some AST or HIR type from
rustc and converts it into the appropriate "cleaned" type. "Big" items like modules or associated
items may have some extra processing in its `Clean` implementation, but for the most part these
impls are straightforward conversions. The "entry point" to this module is the `impl Clean<Crate>
for visit_ast::RustdocVisitor`, which is called by `run_core` above.

You see, I actually lied a little earlier: There's another AST transformation that happens before
the events in `clean/mod.rs`.  In `visit_ast.rs` is the type `RustdocVisitor`, which *actually*
crawls a `hir::Crate` to get the first intermediate representation, defined in `doctree.rs`. This
pass is mainly to get a few intermediate wrappers around the HIR types and to process visibility
and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and `#[doc(hidden)]` are
processed, as well as the logic for whether a `pub use` should get the full page or a "Reexport"
line in the module page.

The other major thing that happens in `clean/mod.rs` is the collection of doc comments and
`#[doc=""]` attributes into a separate field of the Attributes struct, present on anything that gets
hand-written documentation. This makes it easier to collect this documentation later in the process.

The primary output of this process is a `clean::Crate` with a tree of Items which describe the
publicly-documentable items in the target crate.

### Hot potato

Before moving on to the next major step, a few important "passes" occur over the documentation.
These do things like combine the separate "attributes" into a single string and strip leading
whitespace to make the document easier on the markdown parser, or drop items that are not public or
deliberately hidden with `#[doc(hidden)]`. These are all implemented in the `passes/` directory, one
file per pass. By default, all of these passes are run on a crate, but the ones regarding dropping
private/hidden items can be bypassed by passing `--document-private-items` to rustdoc. Note that
unlike the previous set of AST transformations, the passes happen on the _cleaned_ crate.

(Strictly speaking, you can fine-tune the passes run and even add your own, but [we're trying to
deprecate that][44136]. If you need finer-grain control over these passes, please let us know!)

[44136]: https://github.com/rust-lang/rust/issues/44136

There are a few major use cases for rustdoc that you should keep in mind when
working on it:

### Standard library docs

These are published at <https://doc.rust-lang.org/std> as part of the Rust release
process. Stable releases are also uploaded to specific versioned URLs like
<https://doc.rust-lang.org/1.57.0/std/>. Beta and nightly docs are published to
  markdown to like it. This is necessary because the convention for writing
  documentation is to provide a space between the `///` or `//!` marker and the
  text, and stripping that leading space will make the text easier to parse by
  the Markdown parser. (In the past, the markdown parser used was not Commonmark-
  compliant, which caused annoyances with extra whitespace but this seems to be 
  less of an issue today.)
- `strip-priv-imports` strips all private import statements (`use`, `extern
  crate`) from a crate. This is necessary because rustdoc will handle *public*
  imports by either inlining the item's documentation to the module or creating

When crates are published to crates.io, docs.rs automatically builds
and publishes their documentation, for instance at
<https://docs.rs/serde/latest/serde/>. It always builds with the current nightly
rustdoc, so any changes you land in rustdoc are "insta-stable" in that they will

## From clean to crate

This is where the "second phase" in rustdoc begins. This phase primarily lives in the `html/`
folder, and it all starts with `run()` in `html/render.rs`. This code is responsible for setting up
the `Context`, `SharedContext`, and `Cache` which are used during rendering, copying out the static
files which live in every rendered set of documentation (things like the fonts, CSS, and JavaScript
that live in `html/static/`), creating the search index, and printing out the source code rendering,
before beginning the process of rendering all the documentation for the crate.

Several functions implemented directly on `Context` take the `clean::Crate` and set up some state
between rendering items or recursing on a module's child items. From here the "page rendering"
begins, via an enormous `write!()` call in `html/layout.rs`. The parts that actually generate HTML
from the items and documentation occurs within a series of `std::fmt::Display` implementations and
functions that pass around a `&mut std::fmt::Formatter`. The top-level implementation that writes
out the page body is the `impl<'a> fmt::Display for Item<'a>` in `html/render.rs`, which switches
out to one of several `item_*` functions based on the kind of `Item` being rendered.

Depending on what kind of rendering code you're looking for, you'll probably find it either in
`html/render.rs` for major items like "what sections should I print for a struct page" or
`html/format.rs` for smaller component pieces like "how should I print a where clause as part of
some other item".

Whenever rustdoc comes across an item that should print hand-written documentation alongside, it
calls out to `html/markdown.rs` which interfaces with the Markdown parser. This is exposed as a
series of types that wrap a string of Markdown, and implement `fmt::Display` to emit HTML text. It
takes special care to enable certain features like footnotes and tables and add syntax highlighting
to Rust code blocks (via `html/highlight.rs`) before running the Markdown parser. There's also a
function in here (`find_testable_code`) that specifically scans for Rust code blocks so the
test-runner code can find all the doctests in the crate.

### From soup to nuts

(alternate title: ["An unbroken thread that stretches from those first `Cell`s to us"][video])

[video]: https://www.youtube.com/watch?v=hOLAGYmUQV0

It's important to note that the AST cleaning can ask the compiler for information (crucially,
`DocContext` contains a `TyCtxt`), but page rendering cannot. The `clean::Crate` created within
`run_core` is passed outside the compiler context before being handed to `html::render::run`. This
means that a lot of the "supplementary data" that isn't immediately available inside an item's
definition, like which trait is the `Deref` trait used by the language, needs to be collected during
cleaning, stored in the `DocContext`, and passed along to the `SharedContext` during HTML rendering.
This manifests as a bunch of shared state, context variables, and `RefCell`s.

Also of note is that some items that come from "asking the compiler" don't go directly into the
`DocContext` - for example, when loading items from a foreign crate, rustdoc will ask about trait
implementations and generate new `Item`s for the impls based on that information. This goes directly
into the returned `Crate` rather than roundabout through the `DocContext`. This way, these
implementations can be collected alongside the others, right before rendering the HTML.

## Other tricks up its sleeve

All this describes the process for generating HTML documentation from a Rust crate, but there are
couple other major modes that rustdoc runs in. It can also be run on a standalone Markdown file, or
it can run doctests on Rust code or standalone Markdown files. For the former, it shortcuts straight
to `html/markdown.rs`, optionally including a mode which inserts a Table of Contents to the output
HTML.

For the latter, rustdoc runs a similar partial-compilation to get relevant documentation in
`test.rs`, but instead of going through the full clean and render process, it runs a much simpler
crate walk to grab *just* the hand-written documentation. Combined with the aforementioned
"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of tests to run before
handing them off to the libtest test runner. One notable location in `test.rs` is the function
`make_test`, which is where hand-written doctests get transformed into something that can be
executed.

Some extra reading about `make_test` can be found
[here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/).

## Dotting i's and crossing t's

So that's rustdoc's code in a nutshell, but there's more things in the repo that deal with it. Since
we have the full `compiletest` suite at hand, there's a set of tests in `src/test/rustdoc` that make
sure the final HTML is what we expect in various situations. These tests also use a supplementary
script, `src/etc/htmldocck.py`, that allows it to look through the final HTML using XPath notation
to get a precise look at the output. The full description of all the commands available to rustdoc
tests is in `htmldocck.py`.

In addition, there are separate tests for the search index and rustdoc's ability to query it. The
files in `src/test/rustdoc-js` each contain a different search query and the expected results,
broken out by search tab. These files are processed by a script in `src/tools/rustdoc-js` and the
Node.js runtime. These tests don't have as thorough of a writeup, but a broad example that features
results in all tabs can be found in `basic.js`. The basic idea is that you match a given `QUERY`
with a set of `EXPECTED` results, complete with the full item path of each item.

<MSG> reformat basically the whole rustdoc.md

<DFF> @@ -1,7 +1,8 @@
 # The walking tour of rustdoc
 
-Rustdoc actually uses the rustc internals directly. It lives in-tree with the compiler and standard
-library. This chapter is about how it works. (A new implementation is also [under way], though).
+Rustdoc actually uses the rustc internals directly. It lives in-tree with the
+compiler and standard library. This chapter is about how it works. (A new
+implementation is also [under way], though).
 
 [under way]: https://github.com/steveklabnik/rustdoc
 
@@ -17,50 +18,57 @@ and [queries] are discussed in the linked chapters.
 `librustdoc` performs two major steps after that to render a set of
 documentation:
 
-* "Clean" the AST into a form that's more suited to creating documentation (and slightly more
-  resistant to churn in the compiler).
+* "Clean" the AST into a form that's more suited to creating documentation (and
+  slightly more resistant to churn in the compiler).
 * Use this cleaned AST to render a crate's documentation, one page at a time.
 
-Naturally, there's more than just this, and those descriptions simplify out lots of details, but
-that's the high-level overview.
+Naturally, there's more than just this, and those descriptions simplify out
+lots of details, but that's the high-level overview.
 
-(Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated using the project in
-[`src/tools/rustdoc`][bin]. Note that literally all that does is call the `main()` that's in this crate's
-`lib.rs`, though.)
+(Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated
+using the project in [`src/tools/rustdoc`][bin]. Note that literally all that
+does is call the `main()` that's in this crate's `lib.rs`, though.)
 
 [bin]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc
 
 ## Cheat sheet
 
-* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable rustdoc you can run on
-  other projects.
+* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable
+  rustdoc you can run on other projects.
   * Add `src/libtest` to be able to use `rustdoc --test`.
-  * If you've used `rustup toolchain link local /path/to/build/$TARGET/stage1` previously, then
-    after the previous build command, `cargo +local doc` will Just Work.
-* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the standard library docs.
-  * The completed docs will be available in `build/$TARGET/doc/std`, though the bundle is meant to
-    be used as though you would copy out the `doc` folder to a web server, since that's where the
-    CSS/JS and landing page are.
-* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`. It's in a bunch of
-  `fmt::Display` implementations and supplementary functions.
-* The types that got `Display` impls above are defined in `clean/mod.rs`, right next to the custom
-  `Clean` trait used to process them out of the rustc HIR.
+  * If you've used `rustup toolchain link local /path/to/build/$TARGET/stage1`
+    previously, then after the previous build command, `cargo +local doc` will
+    Just Work.
+* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the
+  standard library docs.
+  * The completed docs will be available in `build/$TARGET/doc/std`, though the
+    bundle is meant to be used as though you would copy out the `doc` folder to
+    a web server, since that's where the CSS/JS and landing page are.
+* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`.
+  It's in a bunch of `fmt::Display` implementations and supplementary
+  functions.
+* The types that got `Display` impls above are defined in `clean/mod.rs`, right
+  next to the custom `Clean` trait used to process them out of the rustc HIR.
 * The bits specific to using rustdoc as a test harness are in `test.rs`.
-* The Markdown renderer is loaded up in `html/markdown.rs`, including functions for extracting
-  doctests from a given block of Markdown.
-* The tests on rustdoc *output* are located in `src/test/rustdoc`, where they're handled by the test
-  runner of rustbuild and the supplementary script `src/etc/htmldocck.py`.
-* Tests on search index generation are located in `src/test/rustdoc-js`, as a series of JavaScript
-  files that encode queries on the standard library search index and expected results.
+* The Markdown renderer is loaded up in `html/markdown.rs`, including functions
+  for extracting doctests from a given block of Markdown.
+* The tests on rustdoc *output* are located in `src/test/rustdoc`, where
+  they're handled by the test runner of rustbuild and the supplementary script
+  `src/etc/htmldocck.py`.
+* Tests on search index generation are located in `src/test/rustdoc-js`, as a
+  series of JavaScript files that encode queries on the standard library search
+  index and expected results.
 
 ## From crate to clean
 
-In `core.rs` are two central items: the `DocContext` struct, and the `run_core` function. The latter
-is where rustdoc calls out to rustc to compile a crate to the point where rustdoc can take over. The
-former is a state container used when crawling through a crate to gather its documentation.
+In `core.rs` are two central items: the `DocContext` struct, and the `run_core`
+function. The latter is where rustdoc calls out to rustc to compile a crate to
+the point where rustdoc can take over. The former is a state container used
+when crawling through a crate to gather its documentation.
 
-The main process of crate crawling is done in `clean/mod.rs` through several implementations of the
-`Clean` trait defined within. This is a conversion trait, which defines one method:
+The main process of crate crawling is done in `clean/mod.rs` through several
+implementations of the `Clean` trait defined within. This is a conversion
+trait, which defines one method:
 
 ```rust
 pub trait Clean<T> {
@@ -68,40 +76,47 @@ pub trait Clean<T> {
 }
 ```
 
-`clean/mod.rs` also defines the types for the "cleaned" AST used later on to render documentation
-pages. Each usually accompanies an implementation of `Clean` that takes some AST or HIR type from
-rustc and converts it into the appropriate "cleaned" type. "Big" items like modules or associated
-items may have some extra processing in its `Clean` implementation, but for the most part these
-impls are straightforward conversions. The "entry point" to this module is the `impl Clean<Crate>
-for visit_ast::RustdocVisitor`, which is called by `run_core` above.
-
-You see, I actually lied a little earlier: There's another AST transformation that happens before
-the events in `clean/mod.rs`.  In `visit_ast.rs` is the type `RustdocVisitor`, which *actually*
-crawls a `hir::Crate` to get the first intermediate representation, defined in `doctree.rs`. This
-pass is mainly to get a few intermediate wrappers around the HIR types and to process visibility
-and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and `#[doc(hidden)]` are
-processed, as well as the logic for whether a `pub use` should get the full page or a "Reexport"
-line in the module page.
-
-The other major thing that happens in `clean/mod.rs` is the collection of doc comments and
-`#[doc=""]` attributes into a separate field of the Attributes struct, present on anything that gets
-hand-written documentation. This makes it easier to collect this documentation later in the process.
-
-The primary output of this process is a `clean::Crate` with a tree of Items which describe the
-publicly-documentable items in the target crate.
+`clean/mod.rs` also defines the types for the "cleaned" AST used later on to
+render documentation pages. Each usually accompanies an implementation of
+`Clean` that takes some AST or HIR type from rustc and converts it into the
+appropriate "cleaned" type. "Big" items like modules or associated items may
+have some extra processing in its `Clean` implementation, but for the most part
+these impls are straightforward conversions. The "entry point" to this module
+is the `impl Clean<Crate> for visit_ast::RustdocVisitor`, which is called by
+`run_core` above.
+
+You see, I actually lied a little earlier: There's another AST transformation
+that happens before the events in `clean/mod.rs`.  In `visit_ast.rs` is the
+type `RustdocVisitor`, which *actually* crawls a `hir::Crate` to get the first
+intermediate representation, defined in `doctree.rs`. This pass is mainly to
+get a few intermediate wrappers around the HIR types and to process visibility
+and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and
+`#[doc(hidden)]` are processed, as well as the logic for whether a `pub use`
+should get the full page or a "Reexport" line in the module page.
+
+The other major thing that happens in `clean/mod.rs` is the collection of doc
+comments and `#[doc=""]` attributes into a separate field of the Attributes
+struct, present on anything that gets hand-written documentation. This makes it
+easier to collect this documentation later in the process.
+
+The primary output of this process is a `clean::Crate` with a tree of Items
+which describe the publicly-documentable items in the target crate.
 
 ### Hot potato
 
-Before moving on to the next major step, a few important "passes" occur over the documentation.
-These do things like combine the separate "attributes" into a single string and strip leading
-whitespace to make the document easier on the markdown parser, or drop items that are not public or
-deliberately hidden with `#[doc(hidden)]`. These are all implemented in the `passes/` directory, one
-file per pass. By default, all of these passes are run on a crate, but the ones regarding dropping
-private/hidden items can be bypassed by passing `--document-private-items` to rustdoc. Note that
-unlike the previous set of AST transformations, the passes happen on the _cleaned_ crate.
+Before moving on to the next major step, a few important "passes" occur over
+the documentation.  These do things like combine the separate "attributes" into
+a single string and strip leading whitespace to make the document easier on the
+markdown parser, or drop items that are not public or deliberately hidden with
+`#[doc(hidden)]`. These are all implemented in the `passes/` directory, one
+file per pass. By default, all of these passes are run on a crate, but the ones
+regarding dropping private/hidden items can be bypassed by passing
+`--document-private-items` to rustdoc. Note that unlike the previous set of AST
+transformations, the passes happen on the _cleaned_ crate.
 
-(Strictly speaking, you can fine-tune the passes run and even add your own, but [we're trying to
-deprecate that][44136]. If you need finer-grain control over these passes, please let us know!)
+(Strictly speaking, you can fine-tune the passes run and even add your own, but
+[we're trying to deprecate that][44136]. If you need finer-grain control over
+these passes, please let us know!)
 
 [44136]: https://github.com/rust-lang/rust/issues/44136
 
@@ -116,9 +131,9 @@ Here is current (as of this writing) list of passes:
   markdown to like it. This is necessary because the convention for writing
   documentation is to provide a space between the `///` or `//!` marker and the
   text, and stripping that leading space will make the text easier to parse by
-  the Markdown parser. (In the past, the markdown parser used was not Commonmark-
-  compliant, which caused annoyances with extra whitespace but this seems to be 
-  less of an issue today.)
+  the Markdown parser. (In the past, the markdown parser used was not
+  Commonmark- compliant, which caused annoyances with extra whitespace but this
+  seems to be less of an issue today.)
 - `strip-priv-imports` strips all private import statements (`use`, `extern
   crate`) from a crate. This is necessary because rustdoc will handle *public*
   imports by either inlining the item's documentation to the module or creating
@@ -130,85 +145,101 @@ Here is current (as of this writing) list of passes:
 
 ## From clean to crate
 
-This is where the "second phase" in rustdoc begins. This phase primarily lives in the `html/`
-folder, and it all starts with `run()` in `html/render.rs`. This code is responsible for setting up
-the `Context`, `SharedContext`, and `Cache` which are used during rendering, copying out the static
-files which live in every rendered set of documentation (things like the fonts, CSS, and JavaScript
-that live in `html/static/`), creating the search index, and printing out the source code rendering,
-before beginning the process of rendering all the documentation for the crate.
-
-Several functions implemented directly on `Context` take the `clean::Crate` and set up some state
-between rendering items or recursing on a module's child items. From here the "page rendering"
-begins, via an enormous `write!()` call in `html/layout.rs`. The parts that actually generate HTML
-from the items and documentation occurs within a series of `std::fmt::Display` implementations and
-functions that pass around a `&mut std::fmt::Formatter`. The top-level implementation that writes
-out the page body is the `impl<'a> fmt::Display for Item<'a>` in `html/render.rs`, which switches
-out to one of several `item_*` functions based on the kind of `Item` being rendered.
-
-Depending on what kind of rendering code you're looking for, you'll probably find it either in
-`html/render.rs` for major items like "what sections should I print for a struct page" or
-`html/format.rs` for smaller component pieces like "how should I print a where clause as part of
-some other item".
-
-Whenever rustdoc comes across an item that should print hand-written documentation alongside, it
-calls out to `html/markdown.rs` which interfaces with the Markdown parser. This is exposed as a
-series of types that wrap a string of Markdown, and implement `fmt::Display` to emit HTML text. It
-takes special care to enable certain features like footnotes and tables and add syntax highlighting
-to Rust code blocks (via `html/highlight.rs`) before running the Markdown parser. There's also a
-function in here (`find_testable_code`) that specifically scans for Rust code blocks so the
+This is where the "second phase" in rustdoc begins. This phase primarily lives
+in the `html/` folder, and it all starts with `run()` in `html/render.rs`. This
+code is responsible for setting up the `Context`, `SharedContext`, and `Cache`
+which are used during rendering, copying out the static files which live in
+every rendered set of documentation (things like the fonts, CSS, and JavaScript
+that live in `html/static/`), creating the search index, and printing out the
+source code rendering, before beginning the process of rendering all the
+documentation for the crate.
+
+Several functions implemented directly on `Context` take the `clean::Crate` and
+set up some state between rendering items or recursing on a module's child
+items. From here the "page rendering" begins, via an enormous `write!()` call
+in `html/layout.rs`. The parts that actually generate HTML from the items and
+documentation occurs within a series of `std::fmt::Display` implementations and
+functions that pass around a `&mut std::fmt::Formatter`. The top-level
+implementation that writes out the page body is the `impl<'a> fmt::Display for
+Item<'a>` in `html/render.rs`, which switches out to one of several `item_*`
+functions based on the kind of `Item` being rendered.
+
+Depending on what kind of rendering code you're looking for, you'll probably
+find it either in `html/render.rs` for major items like "what sections should I
+print for a struct page" or `html/format.rs` for smaller component pieces like
+"how should I print a where clause as part of some other item".
+
+Whenever rustdoc comes across an item that should print hand-written
+documentation alongside, it calls out to `html/markdown.rs` which interfaces
+with the Markdown parser. This is exposed as a series of types that wrap a
+string of Markdown, and implement `fmt::Display` to emit HTML text. It takes
+special care to enable certain features like footnotes and tables and add
+syntax highlighting to Rust code blocks (via `html/highlight.rs`) before
+running the Markdown parser. There's also a function in here
+(`find_testable_code`) that specifically scans for Rust code blocks so the
 test-runner code can find all the doctests in the crate.
 
 ### From soup to nuts
 
-(alternate title: ["An unbroken thread that stretches from those first `Cell`s to us"][video])
+(alternate title: ["An unbroken thread that stretches from those first `Cell`s
+to us"][video])
 
 [video]: https://www.youtube.com/watch?v=hOLAGYmUQV0
 
-It's important to note that the AST cleaning can ask the compiler for information (crucially,
-`DocContext` contains a `TyCtxt`), but page rendering cannot. The `clean::Crate` created within
-`run_core` is passed outside the compiler context before being handed to `html::render::run`. This
-means that a lot of the "supplementary data" that isn't immediately available inside an item's
-definition, like which trait is the `Deref` trait used by the language, needs to be collected during
-cleaning, stored in the `DocContext`, and passed along to the `SharedContext` during HTML rendering.
-This manifests as a bunch of shared state, context variables, and `RefCell`s.
-
-Also of note is that some items that come from "asking the compiler" don't go directly into the
-`DocContext` - for example, when loading items from a foreign crate, rustdoc will ask about trait
-implementations and generate new `Item`s for the impls based on that information. This goes directly
-into the returned `Crate` rather than roundabout through the `DocContext`. This way, these
-implementations can be collected alongside the others, right before rendering the HTML.
+It's important to note that the AST cleaning can ask the compiler for
+information (crucially, `DocContext` contains a `TyCtxt`), but page rendering
+cannot. The `clean::Crate` created within `run_core` is passed outside the
+compiler context before being handed to `html::render::run`. This means that a
+lot of the "supplementary data" that isn't immediately available inside an
+item's definition, like which trait is the `Deref` trait used by the language,
+needs to be collected during cleaning, stored in the `DocContext`, and passed
+along to the `SharedContext` during HTML rendering.  This manifests as a bunch
+of shared state, context variables, and `RefCell`s.
+
+Also of note is that some items that come from "asking the compiler" don't go
+directly into the `DocContext` - for example, when loading items from a foreign
+crate, rustdoc will ask about trait implementations and generate new `Item`s
+for the impls based on that information. This goes directly into the returned
+`Crate` rather than roundabout through the `DocContext`. This way, these
+implementations can be collected alongside the others, right before rendering
+the HTML.
 
 ## Other tricks up its sleeve
 
-All this describes the process for generating HTML documentation from a Rust crate, but there are
-couple other major modes that rustdoc runs in. It can also be run on a standalone Markdown file, or
-it can run doctests on Rust code or standalone Markdown files. For the former, it shortcuts straight
-to `html/markdown.rs`, optionally including a mode which inserts a Table of Contents to the output
-HTML.
-
-For the latter, rustdoc runs a similar partial-compilation to get relevant documentation in
-`test.rs`, but instead of going through the full clean and render process, it runs a much simpler
-crate walk to grab *just* the hand-written documentation. Combined with the aforementioned
-"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of tests to run before
-handing them off to the libtest test runner. One notable location in `test.rs` is the function
-`make_test`, which is where hand-written doctests get transformed into something that can be
-executed.
+All this describes the process for generating HTML documentation from a Rust
+crate, but there are couple other major modes that rustdoc runs in. It can also
+be run on a standalone Markdown file, or it can run doctests on Rust code or
+standalone Markdown files. For the former, it shortcuts straight to
+`html/markdown.rs`, optionally including a mode which inserts a Table of
+Contents to the output HTML.
+
+For the latter, rustdoc runs a similar partial-compilation to get relevant
+documentation in `test.rs`, but instead of going through the full clean and
+render process, it runs a much simpler crate walk to grab *just* the
+hand-written documentation. Combined with the aforementioned
+"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of
+tests to run before handing them off to the libtest test runner. One notable
+location in `test.rs` is the function `make_test`, which is where hand-written
+doctests get transformed into something that can be executed.
 
 Some extra reading about `make_test` can be found
 [here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/).
 
 ## Dotting i's and crossing t's
 
-So that's rustdoc's code in a nutshell, but there's more things in the repo that deal with it. Since
-we have the full `compiletest` suite at hand, there's a set of tests in `src/test/rustdoc` that make
-sure the final HTML is what we expect in various situations. These tests also use a supplementary
-script, `src/etc/htmldocck.py`, that allows it to look through the final HTML using XPath notation
-to get a precise look at the output. The full description of all the commands available to rustdoc
-tests is in `htmldocck.py`.
-
-In addition, there are separate tests for the search index and rustdoc's ability to query it. The
-files in `src/test/rustdoc-js` each contain a different search query and the expected results,
-broken out by search tab. These files are processed by a script in `src/tools/rustdoc-js` and the
-Node.js runtime. These tests don't have as thorough of a writeup, but a broad example that features
-results in all tabs can be found in `basic.js`. The basic idea is that you match a given `QUERY`
-with a set of `EXPECTED` results, complete with the full item path of each item.
+So that's rustdoc's code in a nutshell, but there's more things in the repo
+that deal with it. Since we have the full `compiletest` suite at hand, there's
+a set of tests in `src/test/rustdoc` that make sure the final HTML is what we
+expect in various situations. These tests also use a supplementary script,
+`src/etc/htmldocck.py`, that allows it to look through the final HTML using
+XPath notation to get a precise look at the output. The full description of all
+the commands available to rustdoc tests is in `htmldocck.py`.
+
+In addition, there are separate tests for the search index and rustdoc's
+ability to query it. The files in `src/test/rustdoc-js` each contain a
+different search query and the expected results, broken out by search tab.
+These files are processed by a script in `src/tools/rustdoc-js` and the Node.js
+runtime. These tests don't have as thorough of a writeup, but a broad example
+that features results in all tabs can be found in `basic.js`. The basic idea is
+that you match a given `QUERY` with a set of `EXPECTED` results, complete with
+the full item path of each item.

{'commit': 'diff --git a/src/rustdoc.md b/src/rustdoc.md\nindex ae640d1..7a9905d 100644\n--- a/src/rustdoc.md\n+++ b/src/rustdoc.md\n@@ -1,7 +1,8 @@\n # The walking tour of rustdoc\n \n-Rustdoc actually uses the rustc internals directly. It lives in-tree with the compiler and standard\n-library. This chapter is about how it works. (A new implementation is also [under way], though).\n+Rustdoc actually uses the rustc internals directly. It lives in-tree with the\n+compiler and standard library. This chapter is about how it works. (A new\n+implementation is also [under way], though).\n \n [under way]: https://github.com/steveklabnik/rustdoc\n \n@@ -17,50 +18,57 @@ and [queries] are discussed in the linked chapters.\n `librustdoc` performs two major steps after that to render a set of\n documentation:\n \n-* "Clean" the AST into a form that\'s more suited to creating documentation (and slightly more\n-  resistant to churn in the compiler).\n+* "Clean" the AST into a form that\'s more suited to creating documentation (and\n+  slightly more resistant to churn in the compiler).\n * Use this cleaned AST to render a crate\'s documentation, one page at a time.\n \n-Naturally, there\'s more than just this, and those descriptions simplify out lots of details, but\n-that\'s the high-level overview.\n+Naturally, there\'s more than just this, and those descriptions simplify out\n+lots of details, but that\'s the high-level overview.\n \n-(Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated using the project in\n-[`src/tools/rustdoc`][bin]. Note that literally all that does is call the `main()` that\'s in this crate\'s\n-`lib.rs`, though.)\n+(Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated\n+using the project in [`src/tools/rustdoc`][bin]. Note that literally all that\n+does is call the `main()` that\'s in this crate\'s `lib.rs`, though.)\n \n [bin]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc\n \n ## Cheat sheet\n \n-* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable rustdoc you can run on\n-  other projects.\n+* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable\n+  rustdoc you can run on other projects.\n   * Add `src/libtest` to be able to use `rustdoc --test`.\n-  * If you\'ve used `rustup toolchain link local /path/to/build/$TARGET/stage1` previously, then\n-    after the previous build command, `cargo +local doc` will Just Work.\n-* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the standard library docs.\n-  * The completed docs will be available in `build/$TARGET/doc/std`, though the bundle is meant to\n-    be used as though you would copy out the `doc` folder to a web server, since that\'s where the\n-    CSS/JS and landing page are.\n-* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`. It\'s in a bunch of\n-  `fmt::Display` implementations and supplementary functions.\n-* The types that got `Display` impls above are defined in `clean/mod.rs`, right next to the custom\n-  `Clean` trait used to process them out of the rustc HIR.\n+  * If you\'ve used `rustup toolchain link local /path/to/build/$TARGET/stage1`\n+    previously, then after the previous build command, `cargo +local doc` will\n+    Just Work.\n+* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the\n+  standard library docs.\n+  * The completed docs will be available in `build/$TARGET/doc/std`, though the\n+    bundle is meant to be used as though you would copy out the `doc` folder to\n+    a web server, since that\'s where the CSS/JS and landing page are.\n+* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`.\n+  It\'s in a bunch of `fmt::Display` implementations and supplementary\n+  functions.\n+* The types that got `Display` impls above are defined in `clean/mod.rs`, right\n+  next to the custom `Clean` trait used to process them out of the rustc HIR.\n * The bits specific to using rustdoc as a test harness are in `test.rs`.\n-* The Markdown renderer is loaded up in `html/markdown.rs`, including functions for extracting\n-  doctests from a given block of Markdown.\n-* The tests on rustdoc *output* are located in `src/test/rustdoc`, where they\'re handled by the test\n-  runner of rustbuild and the supplementary script `src/etc/htmldocck.py`.\n-* Tests on search index generation are located in `src/test/rustdoc-js`, as a series of JavaScript\n-  files that encode queries on the standard library search index and expected results.\n+* The Markdown renderer is loaded up in `html/markdown.rs`, including functions\n+  for extracting doctests from a given block of Markdown.\n+* The tests on rustdoc *output* are located in `src/test/rustdoc`, where\n+  they\'re handled by the test runner of rustbuild and the supplementary script\n+  `src/etc/htmldocck.py`.\n+* Tests on search index generation are located in `src/test/rustdoc-js`, as a\n+  series of JavaScript files that encode queries on the standard library search\n+  index and expected results.\n \n ## From crate to clean\n \n-In `core.rs` are two central items: the `DocContext` struct, and the `run_core` function. The latter\n-is where rustdoc calls out to rustc to compile a crate to the point where rustdoc can take over. The\n-former is a state container used when crawling through a crate to gather its documentation.\n+In `core.rs` are two central items: the `DocContext` struct, and the `run_core`\n+function. The latter is where rustdoc calls out to rustc to compile a crate to\n+the point where rustdoc can take over. The former is a state container used\n+when crawling through a crate to gather its documentation.\n \n-The main process of crate crawling is done in `clean/mod.rs` through several implementations of the\n-`Clean` trait defined within. This is a conversion trait, which defines one method:\n+The main process of crate crawling is done in `clean/mod.rs` through several\n+implementations of the `Clean` trait defined within. This is a conversion\n+trait, which defines one method:\n \n ```rust\n pub trait Clean<T> {\n@@ -68,40 +76,47 @@ pub trait Clean<T> {\n }\n ```\n \n-`clean/mod.rs` also defines the types for the "cleaned" AST used later on to render documentation\n-pages. Each usually accompanies an implementation of `Clean` that takes some AST or HIR type from\n-rustc and converts it into the appropriate "cleaned" type. "Big" items like modules or associated\n-items may have some extra processing in its `Clean` implementation, but for the most part these\n-impls are straightforward conversions. The "entry point" to this module is the `impl Clean<Crate>\n-for visit_ast::RustdocVisitor`, which is called by `run_core` above.\n-\n-You see, I actually lied a little earlier: There\'s another AST transformation that happens before\n-the events in `clean/mod.rs`.  In `visit_ast.rs` is the type `RustdocVisitor`, which *actually*\n-crawls a `hir::Crate` to get the first intermediate representation, defined in `doctree.rs`. This\n-pass is mainly to get a few intermediate wrappers around the HIR types and to process visibility\n-and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and `#[doc(hidden)]` are\n-processed, as well as the logic for whether a `pub use` should get the full page or a "Reexport"\n-line in the module page.\n-\n-The other major thing that happens in `clean/mod.rs` is the collection of doc comments and\n-`#[doc=""]` attributes into a separate field of the Attributes struct, present on anything that gets\n-hand-written documentation. This makes it easier to collect this documentation later in the process.\n-\n-The primary output of this process is a `clean::Crate` with a tree of Items which describe the\n-publicly-documentable items in the target crate.\n+`clean/mod.rs` also defines the types for the "cleaned" AST used later on to\n+render documentation pages. Each usually accompanies an implementation of\n+`Clean` that takes some AST or HIR type from rustc and converts it into the\n+appropriate "cleaned" type. "Big" items like modules or associated items may\n+have some extra processing in its `Clean` implementation, but for the most part\n+these impls are straightforward conversions. The "entry point" to this module\n+is the `impl Clean<Crate> for visit_ast::RustdocVisitor`, which is called by\n+`run_core` above.\n+\n+You see, I actually lied a little earlier: There\'s another AST transformation\n+that happens before the events in `clean/mod.rs`.  In `visit_ast.rs` is the\n+type `RustdocVisitor`, which *actually* crawls a `hir::Crate` to get the first\n+intermediate representation, defined in `doctree.rs`. This pass is mainly to\n+get a few intermediate wrappers around the HIR types and to process visibility\n+and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and\n+`#[doc(hidden)]` are processed, as well as the logic for whether a `pub use`\n+should get the full page or a "Reexport" line in the module page.\n+\n+The other major thing that happens in `clean/mod.rs` is the collection of doc\n+comments and `#[doc=""]` attributes into a separate field of the Attributes\n+struct, present on anything that gets hand-written documentation. This makes it\n+easier to collect this documentation later in the process.\n+\n+The primary output of this process is a `clean::Crate` with a tree of Items\n+which describe the publicly-documentable items in the target crate.\n \n ### Hot potato\n \n-Before moving on to the next major step, a few important "passes" occur over the documentation.\n-These do things like combine the separate "attributes" into a single string and strip leading\n-whitespace to make the document easier on the markdown parser, or drop items that are not public or\n-deliberately hidden with `#[doc(hidden)]`. These are all implemented in the `passes/` directory, one\n-file per pass. By default, all of these passes are run on a crate, but the ones regarding dropping\n-private/hidden items can be bypassed by passing `--document-private-items` to rustdoc. Note that\n-unlike the previous set of AST transformations, the passes happen on the _cleaned_ crate.\n+Before moving on to the next major step, a few important "passes" occur over\n+the documentation.  These do things like combine the separate "attributes" into\n+a single string and strip leading whitespace to make the document easier on the\n+markdown parser, or drop items that are not public or deliberately hidden with\n+`#[doc(hidden)]`. These are all implemented in the `passes/` directory, one\n+file per pass. By default, all of these passes are run on a crate, but the ones\n+regarding dropping private/hidden items can be bypassed by passing\n+`--document-private-items` to rustdoc. Note that unlike the previous set of AST\n+transformations, the passes happen on the _cleaned_ crate.\n \n-(Strictly speaking, you can fine-tune the passes run and even add your own, but [we\'re trying to\n-deprecate that][44136]. If you need finer-grain control over these passes, please let us know!)\n+(Strictly speaking, you can fine-tune the passes run and even add your own, but\n+[we\'re trying to deprecate that][44136]. If you need finer-grain control over\n+these passes, please let us know!)\n \n [44136]: https://github.com/rust-lang/rust/issues/44136\n \n@@ -116,9 +131,9 @@ Here is current (as of this writing) list of passes:\n   markdown to like it. This is necessary because the convention for writing\n   documentation is to provide a space between the `///` or `//!` marker and the\n   text, and stripping that leading space will make the text easier to parse by\n-  the Markdown parser. (In the past, the markdown parser used was not Commonmark-\n-  compliant, which caused annoyances with extra whitespace but this seems to be \n-  less of an issue today.)\n+  the Markdown parser. (In the past, the markdown parser used was not\n+  Commonmark- compliant, which caused annoyances with extra whitespace but this\n+  seems to be less of an issue today.)\n - `strip-priv-imports` strips all private import statements (`use`, `extern\n   crate`) from a crate. This is necessary because rustdoc will handle *public*\n   imports by either inlining the item\'s documentation to the module or creating\n@@ -130,85 +145,101 @@ Here is current (as of this writing) list of passes:\n \n ## From clean to crate\n \n-This is where the "second phase" in rustdoc begins. This phase primarily lives in the `html/`\n-folder, and it all starts with `run()` in `html/render.rs`. This code is responsible for setting up\n-the `Context`, `SharedContext`, and `Cache` which are used during rendering, copying out the static\n-files which live in every rendered set of documentation (things like the fonts, CSS, and JavaScript\n-that live in `html/static/`), creating the search index, and printing out the source code rendering,\n-before beginning the process of rendering all the documentation for the crate.\n-\n-Several functions implemented directly on `Context` take the `clean::Crate` and set up some state\n-between rendering items or recursing on a module\'s child items. From here the "page rendering"\n-begins, via an enormous `write!()` call in `html/layout.rs`. The parts that actually generate HTML\n-from the items and documentation occurs within a series of `std::fmt::Display` implementations and\n-functions that pass around a `&mut std::fmt::Formatter`. The top-level implementation that writes\n-out the page body is the `impl<\'a> fmt::Display for Item<\'a>` in `html/render.rs`, which switches\n-out to one of several `item_*` functions based on the kind of `Item` being rendered.\n-\n-Depending on what kind of rendering code you\'re looking for, you\'ll probably find it either in\n-`html/render.rs` for major items like "what sections should I print for a struct page" or\n-`html/format.rs` for smaller component pieces like "how should I print a where clause as part of\n-some other item".\n-\n-Whenever rustdoc comes across an item that should print hand-written documentation alongside, it\n-calls out to `html/markdown.rs` which interfaces with the Markdown parser. This is exposed as a\n-series of types that wrap a string of Markdown, and implement `fmt::Display` to emit HTML text. It\n-takes special care to enable certain features like footnotes and tables and add syntax highlighting\n-to Rust code blocks (via `html/highlight.rs`) before running the Markdown parser. There\'s also a\n-function in here (`find_testable_code`) that specifically scans for Rust code blocks so the\n+This is where the "second phase" in rustdoc begins. This phase primarily lives\n+in the `html/` folder, and it all starts with `run()` in `html/render.rs`. This\n+code is responsible for setting up the `Context`, `SharedContext`, and `Cache`\n+which are used during rendering, copying out the static files which live in\n+every rendered set of documentation (things like the fonts, CSS, and JavaScript\n+that live in `html/static/`), creating the search index, and printing out the\n+source code rendering, before beginning the process of rendering all the\n+documentation for the crate.\n+\n+Several functions implemented directly on `Context` take the `clean::Crate` and\n+set up some state between rendering items or recursing on a module\'s child\n+items. From here the "page rendering" begins, via an enormous `write!()` call\n+in `html/layout.rs`. The parts that actually generate HTML from the items and\n+documentation occurs within a series of `std::fmt::Display` implementations and\n+functions that pass around a `&mut std::fmt::Formatter`. The top-level\n+implementation that writes out the page body is the `impl<\'a> fmt::Display for\n+Item<\'a>` in `html/render.rs`, which switches out to one of several `item_*`\n+functions based on the kind of `Item` being rendered.\n+\n+Depending on what kind of rendering code you\'re looking for, you\'ll probably\n+find it either in `html/render.rs` for major items like "what sections should I\n+print for a struct page" or `html/format.rs` for smaller component pieces like\n+"how should I print a where clause as part of some other item".\n+\n+Whenever rustdoc comes across an item that should print hand-written\n+documentation alongside, it calls out to `html/markdown.rs` which interfaces\n+with the Markdown parser. This is exposed as a series of types that wrap a\n+string of Markdown, and implement `fmt::Display` to emit HTML text. It takes\n+special care to enable certain features like footnotes and tables and add\n+syntax highlighting to Rust code blocks (via `html/highlight.rs`) before\n+running the Markdown parser. There\'s also a function in here\n+(`find_testable_code`) that specifically scans for Rust code blocks so the\n test-runner code can find all the doctests in the crate.\n \n ### From soup to nuts\n \n-(alternate title: ["An unbroken thread that stretches from those first `Cell`s to us"][video])\n+(alternate title: ["An unbroken thread that stretches from those first `Cell`s\n+to us"][video])\n \n [video]: https://www.youtube.com/watch?v=hOLAGYmUQV0\n \n-It\'s important to note that the AST cleaning can ask the compiler for information (crucially,\n-`DocContext` contains a `TyCtxt`), but page rendering cannot. The `clean::Crate` created within\n-`run_core` is passed outside the compiler context before being handed to `html::render::run`. This\n-means that a lot of the "supplementary data" that isn\'t immediately available inside an item\'s\n-definition, like which trait is the `Deref` trait used by the language, needs to be collected during\n-cleaning, stored in the `DocContext`, and passed along to the `SharedContext` during HTML rendering.\n-This manifests as a bunch of shared state, context variables, and `RefCell`s.\n-\n-Also of note is that some items that come from "asking the compiler" don\'t go directly into the\n-`DocContext` - for example, when loading items from a foreign crate, rustdoc will ask about trait\n-implementations and generate new `Item`s for the impls based on that information. This goes directly\n-into the returned `Crate` rather than roundabout through the `DocContext`. This way, these\n-implementations can be collected alongside the others, right before rendering the HTML.\n+It\'s important to note that the AST cleaning can ask the compiler for\n+information (crucially, `DocContext` contains a `TyCtxt`), but page rendering\n+cannot. The `clean::Crate` created within `run_core` is passed outside the\n+compiler context before being handed to `html::render::run`. This means that a\n+lot of the "supplementary data" that isn\'t immediately available inside an\n+item\'s definition, like which trait is the `Deref` trait used by the language,\n+needs to be collected during cleaning, stored in the `DocContext`, and passed\n+along to the `SharedContext` during HTML rendering.  This manifests as a bunch\n+of shared state, context variables, and `RefCell`s.\n+\n+Also of note is that some items that come from "asking the compiler" don\'t go\n+directly into the `DocContext` - for example, when loading items from a foreign\n+crate, rustdoc will ask about trait implementations and generate new `Item`s\n+for the impls based on that information. This goes directly into the returned\n+`Crate` rather than roundabout through the `DocContext`. This way, these\n+implementations can be collected alongside the others, right before rendering\n+the HTML.\n \n ## Other tricks up its sleeve\n \n-All this describes the process for generating HTML documentation from a Rust crate, but there are\n-couple other major modes that rustdoc runs in. It can also be run on a standalone Markdown file, or\n-it can run doctests on Rust code or standalone Markdown files. For the former, it shortcuts straight\n-to `html/markdown.rs`, optionally including a mode which inserts a Table of Contents to the output\n-HTML.\n-\n-For the latter, rustdoc runs a similar partial-compilation to get relevant documentation in\n-`test.rs`, but instead of going through the full clean and render process, it runs a much simpler\n-crate walk to grab *just* the hand-written documentation. Combined with the aforementioned\n-"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of tests to run before\n-handing them off to the libtest test runner. One notable location in `test.rs` is the function\n-`make_test`, which is where hand-written doctests get transformed into something that can be\n-executed.\n+All this describes the process for generating HTML documentation from a Rust\n+crate, but there are couple other major modes that rustdoc runs in. It can also\n+be run on a standalone Markdown file, or it can run doctests on Rust code or\n+standalone Markdown files. For the former, it shortcuts straight to\n+`html/markdown.rs`, optionally including a mode which inserts a Table of\n+Contents to the output HTML.\n+\n+For the latter, rustdoc runs a similar partial-compilation to get relevant\n+documentation in `test.rs`, but instead of going through the full clean and\n+render process, it runs a much simpler crate walk to grab *just* the\n+hand-written documentation. Combined with the aforementioned\n+"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of\n+tests to run before handing them off to the libtest test runner. One notable\n+location in `test.rs` is the function `make_test`, which is where hand-written\n+doctests get transformed into something that can be executed.\n \n Some extra reading about `make_test` can be found\n [here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/).\n \n ## Dotting i\'s and crossing t\'s\n \n-So that\'s rustdoc\'s code in a nutshell, but there\'s more things in the repo that deal with it. Since\n-we have the full `compiletest` suite at hand, there\'s a set of tests in `src/test/rustdoc` that make\n-sure the final HTML is what we expect in various situations. These tests also use a supplementary\n-script, `src/etc/htmldocck.py`, that allows it to look through the final HTML using XPath notation\n-to get a precise look at the output. The full description of all the commands available to rustdoc\n-tests is in `htmldocck.py`.\n-\n-In addition, there are separate tests for the search index and rustdoc\'s ability to query it. The\n-files in `src/test/rustdoc-js` each contain a different search query and the expected results,\n-broken out by search tab. These files are processed by a script in `src/tools/rustdoc-js` and the\n-Node.js runtime. These tests don\'t have as thorough of a writeup, but a broad example that features\n-results in all tabs can be found in `basic.js`. The basic idea is that you match a given `QUERY`\n-with a set of `EXPECTED` results, complete with the full item path of each item.\n+So that\'s rustdoc\'s code in a nutshell, but there\'s more things in the repo\n+that deal with it. Since we have the full `compiletest` suite at hand, there\'s\n+a set of tests in `src/test/rustdoc` that make sure the final HTML is what we\n+expect in various situations. These tests also use a supplementary script,\n+`src/etc/htmldocck.py`, that allows it to look through the final HTML using\n+XPath notation to get a precise look at the output. The full description of all\n+the commands available to rustdoc tests is in `htmldocck.py`.\n+\n+In addition, there are separate tests for the search index and rustdoc\'s\n+ability to query it. The files in `src/test/rustdoc-js` each contain a\n+different search query and the expected results, broken out by search tab.\n+These files are processed by a script in `src/tools/rustdoc-js` and the Node.js\n+runtime. These tests don\'t have as thorough of a writeup, but a broad example\n+that features results in all tabs can be found in `basic.js`. The basic idea is\n+that you match a given `QUERY` with a set of `EXPECTED` results, complete with\n+the full item path of each item.\n', 'commit_subject': 'reformat basically the whole rustdoc.md', 'repo_name': 'rust-lang/rustc-dev-guide', 'committer': {'name': 'Who? Me?!', 'email': 'c15a8b12f9f792dbc50834ec1cf7e22bcc3105c7@users.noreply.github.com', 'time_sec': '1522114080', 'tz_offset': '-300', 'date': {'seconds': '1522114080'}}, 'commit_tree': 'a16c88764854d755149faaa36ad165e30525b34e', 'commit_message': 'reformat basically the whole rustdoc.md\n', 'language_name': 'HTML', 'license': 'apache-2.0', 'hunks': '@@ -1,7 +1,8 @@\n # The walking tour of rustdoc\n \n-Rustdoc actually uses the rustc internals directly. It lives in-tree with the compiler and standard\n-library. This chapter is about how it works. (A new implementation is also [under way], though).\n+Rustdoc actually uses the rustc internals directly. It lives in-tree with the\n+compiler and standard library. This chapter is about how it works. (A new\n+implementation is also [under way], though).\n \n [under way]: https://github.com/steveklabnik/rustdoc\n \n@@ -17,50 +18,57 @@ and [queries] are discussed in the linked chapters.\n `librustdoc` performs two major steps after that to render a set of\n documentation:\n \n-* "Clean" the AST into a form that\'s more suited to creating documentation (and slightly more\n-  resistant to churn in the compiler).\n+* "Clean" the AST into a form that\'s more suited to creating documentation (and\n+  slightly more resistant to churn in the compiler).\n * Use this cleaned AST to render a crate\'s documentation, one page at a time.\n \n-Naturally, there\'s more than just this, and those descriptions simplify out lots of details, but\n-that\'s the high-level overview.\n+Naturally, there\'s more than just this, and those descriptions simplify out\n+lots of details, but that\'s the high-level overview.\n \n-(Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated using the project in\n-[`src/tools/rustdoc`][bin]. Note that literally all that does is call the `main()` that\'s in this crate\'s\n-`lib.rs`, though.)\n+(Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated\n+using the project in [`src/tools/rustdoc`][bin]. Note that literally all that\n+does is call the `main()` that\'s in this crate\'s `lib.rs`, though.)\n \n [bin]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc\n \n ## Cheat sheet\n \n-* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable rustdoc you can run on\n-  other projects.\n+* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable\n+  rustdoc you can run on other projects.\n   * Add `src/libtest` to be able to use `rustdoc --test`.\n-  * If you\'ve used `rustup toolchain link local /path/to/build/$TARGET/stage1` previously, then\n-    after the previous build command, `cargo +local doc` will Just Work.\n-* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the standard library docs.\n-  * The completed docs will be available in `build/$TARGET/doc/std`, though the bundle is meant to\n-    be used as though you would copy out the `doc` folder to a web server, since that\'s where the\n-    CSS/JS and landing page are.\n-* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`. It\'s in a bunch of\n-  `fmt::Display` implementations and supplementary functions.\n-* The types that got `Display` impls above are defined in `clean/mod.rs`, right next to the custom\n-  `Clean` trait used to process them out of the rustc HIR.\n+  * If you\'ve used `rustup toolchain link local /path/to/build/$TARGET/stage1`\n+    previously, then after the previous build command, `cargo +local doc` will\n+    Just Work.\n+* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the\n+  standard library docs.\n+  * The completed docs will be available in `build/$TARGET/doc/std`, though the\n+    bundle is meant to be used as though you would copy out the `doc` folder to\n+    a web server, since that\'s where the CSS/JS and landing page are.\n+* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`.\n+  It\'s in a bunch of `fmt::Display` implementations and supplementary\n+  functions.\n+* The types that got `Display` impls above are defined in `clean/mod.rs`, right\n+  next to the custom `Clean` trait used to process them out of the rustc HIR.\n * The bits specific to using rustdoc as a test harness are in `test.rs`.\n-* The Markdown renderer is loaded up in `html/markdown.rs`, including functions for extracting\n-  doctests from a given block of Markdown.\n-* The tests on rustdoc *output* are located in `src/test/rustdoc`, where they\'re handled by the test\n-  runner of rustbuild and the supplementary script `src/etc/htmldocck.py`.\n-* Tests on search index generation are located in `src/test/rustdoc-js`, as a series of JavaScript\n-  files that encode queries on the standard library search index and expected results.\n+* The Markdown renderer is loaded up in `html/markdown.rs`, including functions\n+  for extracting doctests from a given block of Markdown.\n+* The tests on rustdoc *output* are located in `src/test/rustdoc`, where\n+  they\'re handled by the test runner of rustbuild and the supplementary script\n+  `src/etc/htmldocck.py`.\n+* Tests on search index generation are located in `src/test/rustdoc-js`, as a\n+  series of JavaScript files that encode queries on the standard library search\n+  index and expected results.\n \n ## From crate to clean\n \n-In `core.rs` are two central items: the `DocContext` struct, and the `run_core` function. The latter\n-is where rustdoc calls out to rustc to compile a crate to the point where rustdoc can take over. The\n-former is a state container used when crawling through a crate to gather its documentation.\n+In `core.rs` are two central items: the `DocContext` struct, and the `run_core`\n+function. The latter is where rustdoc calls out to rustc to compile a crate to\n+the point where rustdoc can take over. The former is a state container used\n+when crawling through a crate to gather its documentation.\n \n-The main process of crate crawling is done in `clean/mod.rs` through several implementations of the\n-`Clean` trait defined within. This is a conversion trait, which defines one method:\n+The main process of crate crawling is done in `clean/mod.rs` through several\n+implementations of the `Clean` trait defined within. This is a conversion\n+trait, which defines one method:\n \n ```rust\n pub trait Clean<T> {\n@@ -68,40 +76,47 @@ pub trait Clean<T> {\n }\n ```\n \n-`clean/mod.rs` also defines the types for the "cleaned" AST used later on to render documentation\n-pages. Each usually accompanies an implementation of `Clean` that takes some AST or HIR type from\n-rustc and converts it into the appropriate "cleaned" type. "Big" items like modules or associated\n-items may have some extra processing in its `Clean` implementation, but for the most part these\n-impls are straightforward conversions. The "entry point" to this module is the `impl Clean<Crate>\n-for visit_ast::RustdocVisitor`, which is called by `run_core` above.\n-\n-You see, I actually lied a little earlier: There\'s another AST transformation that happens before\n-the events in `clean/mod.rs`.  In `visit_ast.rs` is the type `RustdocVisitor`, which *actually*\n-crawls a `hir::Crate` to get the first intermediate representation, defined in `doctree.rs`. This\n-pass is mainly to get a few intermediate wrappers around the HIR types and to process visibility\n-and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and `#[doc(hidden)]` are\n-processed, as well as the logic for whether a `pub use` should get the full page or a "Reexport"\n-line in the module page.\n-\n-The other major thing that happens in `clean/mod.rs` is the collection of doc comments and\n-`#[doc=""]` attributes into a separate field of the Attributes struct, present on anything that gets\n-hand-written documentation. This makes it easier to collect this documentation later in the process.\n-\n-The primary output of this process is a `clean::Crate` with a tree of Items which describe the\n-publicly-documentable items in the target crate.\n+`clean/mod.rs` also defines the types for the "cleaned" AST used later on to\n+render documentation pages. Each usually accompanies an implementation of\n+`Clean` that takes some AST or HIR type from rustc and converts it into the\n+appropriate "cleaned" type. "Big" items like modules or associated items may\n+have some extra processing in its `Clean` implementation, but for the most part\n+these impls are straightforward conversions. The "entry point" to this module\n+is the `impl Clean<Crate> for visit_ast::RustdocVisitor`, which is called by\n+`run_core` above.\n+\n+You see, I actually lied a little earlier: There\'s another AST transformation\n+that happens before the events in `clean/mod.rs`.  In `visit_ast.rs` is the\n+type `RustdocVisitor`, which *actually* crawls a `hir::Crate` to get the first\n+intermediate representation, defined in `doctree.rs`. This pass is mainly to\n+get a few intermediate wrappers around the HIR types and to process visibility\n+and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and\n+`#[doc(hidden)]` are processed, as well as the logic for whether a `pub use`\n+should get the full page or a "Reexport" line in the module page.\n+\n+The other major thing that happens in `clean/mod.rs` is the collection of doc\n+comments and `#[doc=""]` attributes into a separate field of the Attributes\n+struct, present on anything that gets hand-written documentation. This makes it\n+easier to collect this documentation later in the process.\n+\n+The primary output of this process is a `clean::Crate` with a tree of Items\n+which describe the publicly-documentable items in the target crate.\n \n ### Hot potato\n \n-Before moving on to the next major step, a few important "passes" occur over the documentation.\n-These do things like combine the separate "attributes" into a single string and strip leading\n-whitespace to make the document easier on the markdown parser, or drop items that are not public or\n-deliberately hidden with `#[doc(hidden)]`. These are all implemented in the `passes/` directory, one\n-file per pass. By default, all of these passes are run on a crate, but the ones regarding dropping\n-private/hidden items can be bypassed by passing `--document-private-items` to rustdoc. Note that\n-unlike the previous set of AST transformations, the passes happen on the _cleaned_ crate.\n+Before moving on to the next major step, a few important "passes" occur over\n+the documentation.  These do things like combine the separate "attributes" into\n+a single string and strip leading whitespace to make the document easier on the\n+markdown parser, or drop items that are not public or deliberately hidden with\n+`#[doc(hidden)]`. These are all implemented in the `passes/` directory, one\n+file per pass. By default, all of these passes are run on a crate, but the ones\n+regarding dropping private/hidden items can be bypassed by passing\n+`--document-private-items` to rustdoc. Note that unlike the previous set of AST\n+transformations, the passes happen on the _cleaned_ crate.\n \n-(Strictly speaking, you can fine-tune the passes run and even add your own, but [we\'re trying to\n-deprecate that][44136]. If you need finer-grain control over these passes, please let us know!)\n+(Strictly speaking, you can fine-tune the passes run and even add your own, but\n+[we\'re trying to deprecate that][44136]. If you need finer-grain control over\n+these passes, please let us know!)\n \n [44136]: https://github.com/rust-lang/rust/issues/44136\n \n@@ -116,9 +131,9 @@ Here is current (as of this writing) list of passes:\n   markdown to like it. This is necessary because the convention for writing\n   documentation is to provide a space between the `///` or `//!` marker and the\n   text, and stripping that leading space will make the text easier to parse by\n-  the Markdown parser. (In the past, the markdown parser used was not Commonmark-\n-  compliant, which caused annoyances with extra whitespace but this seems to be \n-  less of an issue today.)\n+  the Markdown parser. (In the past, the markdown parser used was not\n+  Commonmark- compliant, which caused annoyances with extra whitespace but this\n+  seems to be less of an issue today.)\n - `strip-priv-imports` strips all private import statements (`use`, `extern\n   crate`) from a crate. This is necessary because rustdoc will handle *public*\n   imports by either inlining the item\'s documentation to the module or creating\n@@ -130,85 +145,101 @@ Here is current (as of this writing) list of passes:\n \n ## From clean to crate\n \n-This is where the "second phase" in rustdoc begins. This phase primarily lives in the `html/`\n-folder, and it all starts with `run()` in `html/render.rs`. This code is responsible for setting up\n-the `Context`, `SharedContext`, and `Cache` which are used during rendering, copying out the static\n-files which live in every rendered set of documentation (things like the fonts, CSS, and JavaScript\n-that live in `html/static/`), creating the search index, and printing out the source code rendering,\n-before beginning the process of rendering all the documentation for the crate.\n-\n-Several functions implemented directly on `Context` take the `clean::Crate` and set up some state\n-between rendering items or recursing on a module\'s child items. From here the "page rendering"\n-begins, via an enormous `write!()` call in `html/layout.rs`. The parts that actually generate HTML\n-from the items and documentation occurs within a series of `std::fmt::Display` implementations and\n-functions that pass around a `&mut std::fmt::Formatter`. The top-level implementation that writes\n-out the page body is the `impl<\'a> fmt::Display for Item<\'a>` in `html/render.rs`, which switches\n-out to one of several `item_*` functions based on the kind of `Item` being rendered.\n-\n-Depending on what kind of rendering code you\'re looking for, you\'ll probably find it either in\n-`html/render.rs` for major items like "what sections should I print for a struct page" or\n-`html/format.rs` for smaller component pieces like "how should I print a where clause as part of\n-some other item".\n-\n-Whenever rustdoc comes across an item that should print hand-written documentation alongside, it\n-calls out to `html/markdown.rs` which interfaces with the Markdown parser. This is exposed as a\n-series of types that wrap a string of Markdown, and implement `fmt::Display` to emit HTML text. It\n-takes special care to enable certain features like footnotes and tables and add syntax highlighting\n-to Rust code blocks (via `html/highlight.rs`) before running the Markdown parser. There\'s also a\n-function in here (`find_testable_code`) that specifically scans for Rust code blocks so the\n+This is where the "second phase" in rustdoc begins. This phase primarily lives\n+in the `html/` folder, and it all starts with `run()` in `html/render.rs`. This\n+code is responsible for setting up the `Context`, `SharedContext`, and `Cache`\n+which are used during rendering, copying out the static files which live in\n+every rendered set of documentation (things like the fonts, CSS, and JavaScript\n+that live in `html/static/`), creating the search index, and printing out the\n+source code rendering, before beginning the process of rendering all the\n+documentation for the crate.\n+\n+Several functions implemented directly on `Context` take the `clean::Crate` and\n+set up some state between rendering items or recursing on a module\'s child\n+items. From here the "page rendering" begins, via an enormous `write!()` call\n+in `html/layout.rs`. The parts that actually generate HTML from the items and\n+documentation occurs within a series of `std::fmt::Display` implementations and\n+functions that pass around a `&mut std::fmt::Formatter`. The top-level\n+implementation that writes out the page body is the `impl<\'a> fmt::Display for\n+Item<\'a>` in `html/render.rs`, which switches out to one of several `item_*`\n+functions based on the kind of `Item` being rendered.\n+\n+Depending on what kind of rendering code you\'re looking for, you\'ll probably\n+find it either in `html/render.rs` for major items like "what sections should I\n+print for a struct page" or `html/format.rs` for smaller component pieces like\n+"how should I print a where clause as part of some other item".\n+\n+Whenever rustdoc comes across an item that should print hand-written\n+documentation alongside, it calls out to `html/markdown.rs` which interfaces\n+with the Markdown parser. This is exposed as a series of types that wrap a\n+string of Markdown, and implement `fmt::Display` to emit HTML text. It takes\n+special care to enable certain features like footnotes and tables and add\n+syntax highlighting to Rust code blocks (via `html/highlight.rs`) before\n+running the Markdown parser. There\'s also a function in here\n+(`find_testable_code`) that specifically scans for Rust code blocks so the\n test-runner code can find all the doctests in the crate.\n \n ### From soup to nuts\n \n-(alternate title: ["An unbroken thread that stretches from those first `Cell`s to us"][video])\n+(alternate title: ["An unbroken thread that stretches from those first `Cell`s\n+to us"][video])\n \n [video]: https://www.youtube.com/watch?v=hOLAGYmUQV0\n \n-It\'s important to note that the AST cleaning can ask the compiler for information (crucially,\n-`DocContext` contains a `TyCtxt`), but page rendering cannot. The `clean::Crate` created within\n-`run_core` is passed outside the compiler context before being handed to `html::render::run`. This\n-means that a lot of the "supplementary data" that isn\'t immediately available inside an item\'s\n-definition, like which trait is the `Deref` trait used by the language, needs to be collected during\n-cleaning, stored in the `DocContext`, and passed along to the `SharedContext` during HTML rendering.\n-This manifests as a bunch of shared state, context variables, and `RefCell`s.\n-\n-Also of note is that some items that come from "asking the compiler" don\'t go directly into the\n-`DocContext` - for example, when loading items from a foreign crate, rustdoc will ask about trait\n-implementations and generate new `Item`s for the impls based on that information. This goes directly\n-into the returned `Crate` rather than roundabout through the `DocContext`. This way, these\n-implementations can be collected alongside the others, right before rendering the HTML.\n+It\'s important to note that the AST cleaning can ask the compiler for\n+information (crucially, `DocContext` contains a `TyCtxt`), but page rendering\n+cannot. The `clean::Crate` created within `run_core` is passed outside the\n+compiler context before being handed to `html::render::run`. This means that a\n+lot of the "supplementary data" that isn\'t immediately available inside an\n+item\'s definition, like which trait is the `Deref` trait used by the language,\n+needs to be collected during cleaning, stored in the `DocContext`, and passed\n+along to the `SharedContext` during HTML rendering.  This manifests as a bunch\n+of shared state, context variables, and `RefCell`s.\n+\n+Also of note is that some items that come from "asking the compiler" don\'t go\n+directly into the `DocContext` - for example, when loading items from a foreign\n+crate, rustdoc will ask about trait implementations and generate new `Item`s\n+for the impls based on that information. This goes directly into the returned\n+`Crate` rather than roundabout through the `DocContext`. This way, these\n+implementations can be collected alongside the others, right before rendering\n+the HTML.\n \n ## Other tricks up its sleeve\n \n-All this describes the process for generating HTML documentation from a Rust crate, but there are\n-couple other major modes that rustdoc runs in. It can also be run on a standalone Markdown file, or\n-it can run doctests on Rust code or standalone Markdown files. For the former, it shortcuts straight\n-to `html/markdown.rs`, optionally including a mode which inserts a Table of Contents to the output\n-HTML.\n-\n-For the latter, rustdoc runs a similar partial-compilation to get relevant documentation in\n-`test.rs`, but instead of going through the full clean and render process, it runs a much simpler\n-crate walk to grab *just* the hand-written documentation. Combined with the aforementioned\n-"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of tests to run before\n-handing them off to the libtest test runner. One notable location in `test.rs` is the function\n-`make_test`, which is where hand-written doctests get transformed into something that can be\n-executed.\n+All this describes the process for generating HTML documentation from a Rust\n+crate, but there are couple other major modes that rustdoc runs in. It can also\n+be run on a standalone Markdown file, or it can run doctests on Rust code or\n+standalone Markdown files. For the former, it shortcuts straight to\n+`html/markdown.rs`, optionally including a mode which inserts a Table of\n+Contents to the output HTML.\n+\n+For the latter, rustdoc runs a similar partial-compilation to get relevant\n+documentation in `test.rs`, but instead of going through the full clean and\n+render process, it runs a much simpler crate walk to grab *just* the\n+hand-written documentation. Combined with the aforementioned\n+"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of\n+tests to run before handing them off to the libtest test runner. One notable\n+location in `test.rs` is the function `make_test`, which is where hand-written\n+doctests get transformed into something that can be executed.\n \n Some extra reading about `make_test` can be found\n [here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/).\n \n ## Dotting i\'s and crossing t\'s\n \n-So that\'s rustdoc\'s code in a nutshell, but there\'s more things in the repo that deal with it. Since\n-we have the full `compiletest` suite at hand, there\'s a set of tests in `src/test/rustdoc` that make\n-sure the final HTML is what we expect in various situations. These tests also use a supplementary\n-script, `src/etc/htmldocck.py`, that allows it to look through the final HTML using XPath notation\n-to get a precise look at the output. The full description of all the commands available to rustdoc\n-tests is in `htmldocck.py`.\n-\n-In addition, there are separate tests for the search index and rustdoc\'s ability to query it. The\n-files in `src/test/rustdoc-js` each contain a different search query and the expected results,\n-broken out by search tab. These files are processed by a script in `src/tools/rustdoc-js` and the\n-Node.js runtime. These tests don\'t have as thorough of a writeup, but a broad example that features\n-results in all tabs can be found in `basic.js`. The basic idea is that you match a given `QUERY`\n-with a set of `EXPECTED` results, complete with the full item path of each item.\n+So that\'s rustdoc\'s code in a nutshell, but there\'s more things in the repo\n+that deal with it. Since we have the full `compiletest` suite at hand, there\'s\n+a set of tests in `src/test/rustdoc` that make sure the final HTML is what we\n+expect in various situations. These tests also use a supplementary script,\n+`src/etc/htmldocck.py`, that allows it to look through the final HTML using\n+XPath notation to get a precise look at the output. The full description of all\n+the commands available to rustdoc tests is in `htmldocck.py`.\n+\n+In addition, there are separate tests for the search index and rustdoc\'s\n+ability to query it. The files in `src/test/rustdoc-js` each contain a\n+different search query and the expected results, broken out by search tab.\n+These files are processed by a script in `src/tools/rustdoc-js` and the Node.js\n+runtime. These tests don\'t have as thorough of a writeup, but a broad example\n+that features results in all tabs can be found in `basic.js`. The basic idea is\n+that you match a given `QUERY` with a set of `EXPECTED` results, complete with\n+the full item path of each item.\n', 'addition_count': 162, 'deletion_count': 131, 'src_file': 'a/src/rustdoc.md', 'tgt_file': 'b/src/rustdoc.md', 'file_extension': '.md', 'before_file': '# The walking tour of rustdoc\n\nRustdoc actually uses the rustc internals directly. It lives in-tree with the compiler and standard\nlibrary. This chapter is about how it works. (A new implementation is also [under way], though).\n\n[under way]: https://github.com/steveklabnik/rustdoc\n\nthe [Rustdoc book](https://doc.rust-lang.org/nightly/rustdoc/).\nFor more details about how rustdoc works, see the\n["Rustdoc internals" chapter][Rustdoc internals].\n\n[Rustdoc internals]: ./rustdoc-internals.md\n\nRustdoc is implemented entirely within the crate [`librustdoc`][rd]. It runs\nthe compiler up to the point where we have an internal representation of a\ncrate (HIR) and the ability to run some queries about the types of items. [HIR]\n`librustdoc` performs two major steps after that to render a set of\ndocumentation:\n\n* "Clean" the AST into a form that\'s more suited to creating documentation (and slightly more\n  resistant to churn in the compiler).\n* Use this cleaned AST to render a crate\'s documentation, one page at a time.\n\nNaturally, there\'s more than just this, and those descriptions simplify out lots of details, but\nthat\'s the high-level overview.\n\n(Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated using the project in\n[`src/tools/rustdoc`][bin]. Note that literally all that does is call the `main()` that\'s in this crate\'s\n`lib.rs`, though.)\n\n[bin]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc\n\n## Cheat sheet\n\n* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable rustdoc you can run on\n  other projects.\n  * Add `src/libtest` to be able to use `rustdoc --test`.\n  * If you\'ve used `rustup toolchain link local /path/to/build/$TARGET/stage1` previously, then\n    after the previous build command, `cargo +local doc` will Just Work.\n* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the standard library docs.\n  * The completed docs will be available in `build/$TARGET/doc/std`, though the bundle is meant to\n    be used as though you would copy out the `doc` folder to a web server, since that\'s where the\n    CSS/JS and landing page are.\n* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`. It\'s in a bunch of\n  `fmt::Display` implementations and supplementary functions.\n* The types that got `Display` impls above are defined in `clean/mod.rs`, right next to the custom\n  `Clean` trait used to process them out of the rustc HIR.\n* The bits specific to using rustdoc as a test harness are in `test.rs`.\n* The Markdown renderer is loaded up in `html/markdown.rs`, including functions for extracting\n  doctests from a given block of Markdown.\n* The tests on rustdoc *output* are located in `src/test/rustdoc`, where they\'re handled by the test\n  runner of rustbuild and the supplementary script `src/etc/htmldocck.py`.\n* Tests on search index generation are located in `src/test/rustdoc-js`, as a series of JavaScript\n  files that encode queries on the standard library search index and expected results.\n\n## From crate to clean\n\nIn `core.rs` are two central items: the `DocContext` struct, and the `run_core` function. The latter\nis where rustdoc calls out to rustc to compile a crate to the point where rustdoc can take over. The\nformer is a state container used when crawling through a crate to gather its documentation.\n\nThe main process of crate crawling is done in `clean/mod.rs` through several implementations of the\n`Clean` trait defined within. This is a conversion trait, which defines one method:\n\n```rust\npub trait Clean<T> {\n* The tests on the structure of rustdoc HTML output are located in `src/test/rustdoc`, where\n}\n```\n\n`clean/mod.rs` also defines the types for the "cleaned" AST used later on to render documentation\npages. Each usually accompanies an implementation of `Clean` that takes some AST or HIR type from\nrustc and converts it into the appropriate "cleaned" type. "Big" items like modules or associated\nitems may have some extra processing in its `Clean` implementation, but for the most part these\nimpls are straightforward conversions. The "entry point" to this module is the `impl Clean<Crate>\nfor visit_ast::RustdocVisitor`, which is called by `run_core` above.\n\nYou see, I actually lied a little earlier: There\'s another AST transformation that happens before\nthe events in `clean/mod.rs`.  In `visit_ast.rs` is the type `RustdocVisitor`, which *actually*\ncrawls a `hir::Crate` to get the first intermediate representation, defined in `doctree.rs`. This\npass is mainly to get a few intermediate wrappers around the HIR types and to process visibility\nand inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and `#[doc(hidden)]` are\nprocessed, as well as the logic for whether a `pub use` should get the full page or a "Reexport"\nline in the module page.\n\nThe other major thing that happens in `clean/mod.rs` is the collection of doc comments and\n`#[doc=""]` attributes into a separate field of the Attributes struct, present on anything that gets\nhand-written documentation. This makes it easier to collect this documentation later in the process.\n\nThe primary output of this process is a `clean::Crate` with a tree of Items which describe the\npublicly-documentable items in the target crate.\n\n### Hot potato\n\nBefore moving on to the next major step, a few important "passes" occur over the documentation.\nThese do things like combine the separate "attributes" into a single string and strip leading\nwhitespace to make the document easier on the markdown parser, or drop items that are not public or\ndeliberately hidden with `#[doc(hidden)]`. These are all implemented in the `passes/` directory, one\nfile per pass. By default, all of these passes are run on a crate, but the ones regarding dropping\nprivate/hidden items can be bypassed by passing `--document-private-items` to rustdoc. Note that\nunlike the previous set of AST transformations, the passes happen on the _cleaned_ crate.\n\n(Strictly speaking, you can fine-tune the passes run and even add your own, but [we\'re trying to\ndeprecate that][44136]. If you need finer-grain control over these passes, please let us know!)\n\n[44136]: https://github.com/rust-lang/rust/issues/44136\n\nThere are a few major use cases for rustdoc that you should keep in mind when\nworking on it:\n\n### Standard library docs\n\nThese are published at <https://doc.rust-lang.org/std> as part of the Rust release\nprocess. Stable releases are also uploaded to specific versioned URLs like\n<https://doc.rust-lang.org/1.57.0/std/>. Beta and nightly docs are published to\n  markdown to like it. This is necessary because the convention for writing\n  documentation is to provide a space between the `///` or `//!` marker and the\n  text, and stripping that leading space will make the text easier to parse by\n  the Markdown parser. (In the past, the markdown parser used was not Commonmark-\n  compliant, which caused annoyances with extra whitespace but this seems to be \n  less of an issue today.)\n- `strip-priv-imports` strips all private import statements (`use`, `extern\n  crate`) from a crate. This is necessary because rustdoc will handle *public*\n  imports by either inlining the item\'s documentation to the module or creating\n\nWhen crates are published to crates.io, docs.rs automatically builds\nand publishes their documentation, for instance at\n<https://docs.rs/serde/latest/serde/>. It always builds with the current nightly\nrustdoc, so any changes you land in rustdoc are "insta-stable" in that they will\n\n## From clean to crate\n\nThis is where the "second phase" in rustdoc begins. This phase primarily lives in the `html/`\nfolder, and it all starts with `run()` in `html/render.rs`. This code is responsible for setting up\nthe `Context`, `SharedContext`, and `Cache` which are used during rendering, copying out the static\nfiles which live in every rendered set of documentation (things like the fonts, CSS, and JavaScript\nthat live in `html/static/`), creating the search index, and printing out the source code rendering,\nbefore beginning the process of rendering all the documentation for the crate.\n\nSeveral functions implemented directly on `Context` take the `clean::Crate` and set up some state\nbetween rendering items or recursing on a module\'s child items. From here the "page rendering"\nbegins, via an enormous `write!()` call in `html/layout.rs`. The parts that actually generate HTML\nfrom the items and documentation occurs within a series of `std::fmt::Display` implementations and\nfunctions that pass around a `&mut std::fmt::Formatter`. The top-level implementation that writes\nout the page body is the `impl<\'a> fmt::Display for Item<\'a>` in `html/render.rs`, which switches\nout to one of several `item_*` functions based on the kind of `Item` being rendered.\n\nDepending on what kind of rendering code you\'re looking for, you\'ll probably find it either in\n`html/render.rs` for major items like "what sections should I print for a struct page" or\n`html/format.rs` for smaller component pieces like "how should I print a where clause as part of\nsome other item".\n\nWhenever rustdoc comes across an item that should print hand-written documentation alongside, it\ncalls out to `html/markdown.rs` which interfaces with the Markdown parser. This is exposed as a\nseries of types that wrap a string of Markdown, and implement `fmt::Display` to emit HTML text. It\ntakes special care to enable certain features like footnotes and tables and add syntax highlighting\nto Rust code blocks (via `html/highlight.rs`) before running the Markdown parser. There\'s also a\nfunction in here (`find_testable_code`) that specifically scans for Rust code blocks so the\ntest-runner code can find all the doctests in the crate.\n\n### From soup to nuts\n\n(alternate title: ["An unbroken thread that stretches from those first `Cell`s to us"][video])\n\n[video]: https://www.youtube.com/watch?v=hOLAGYmUQV0\n\nIt\'s important to note that the AST cleaning can ask the compiler for information (crucially,\n`DocContext` contains a `TyCtxt`), but page rendering cannot. The `clean::Crate` created within\n`run_core` is passed outside the compiler context before being handed to `html::render::run`. This\nmeans that a lot of the "supplementary data" that isn\'t immediately available inside an item\'s\ndefinition, like which trait is the `Deref` trait used by the language, needs to be collected during\ncleaning, stored in the `DocContext`, and passed along to the `SharedContext` during HTML rendering.\nThis manifests as a bunch of shared state, context variables, and `RefCell`s.\n\nAlso of note is that some items that come from "asking the compiler" don\'t go directly into the\n`DocContext` - for example, when loading items from a foreign crate, rustdoc will ask about trait\nimplementations and generate new `Item`s for the impls based on that information. This goes directly\ninto the returned `Crate` rather than roundabout through the `DocContext`. This way, these\nimplementations can be collected alongside the others, right before rendering the HTML.\n\n## Other tricks up its sleeve\n\nAll this describes the process for generating HTML documentation from a Rust crate, but there are\ncouple other major modes that rustdoc runs in. It can also be run on a standalone Markdown file, or\nit can run doctests on Rust code or standalone Markdown files. For the former, it shortcuts straight\nto `html/markdown.rs`, optionally including a mode which inserts a Table of Contents to the output\nHTML.\n\nFor the latter, rustdoc runs a similar partial-compilation to get relevant documentation in\n`test.rs`, but instead of going through the full clean and render process, it runs a much simpler\ncrate walk to grab *just* the hand-written documentation. Combined with the aforementioned\n"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of tests to run before\nhanding them off to the libtest test runner. One notable location in `test.rs` is the function\n`make_test`, which is where hand-written doctests get transformed into something that can be\nexecuted.\n\nSome extra reading about `make_test` can be found\n[here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/).\n\n## Dotting i\'s and crossing t\'s\n\nSo that\'s rustdoc\'s code in a nutshell, but there\'s more things in the repo that deal with it. Since\nwe have the full `compiletest` suite at hand, there\'s a set of tests in `src/test/rustdoc` that make\nsure the final HTML is what we expect in various situations. These tests also use a supplementary\nscript, `src/etc/htmldocck.py`, that allows it to look through the final HTML using XPath notation\nto get a precise look at the output. The full description of all the commands available to rustdoc\ntests is in `htmldocck.py`.\n\nIn addition, there are separate tests for the search index and rustdoc\'s ability to query it. The\nfiles in `src/test/rustdoc-js` each contain a different search query and the expected results,\nbroken out by search tab. These files are processed by a script in `src/tools/rustdoc-js` and the\nNode.js runtime. These tests don\'t have as thorough of a writeup, but a broad example that features\nresults in all tabs can be found in `basic.js`. The basic idea is that you match a given `QUERY`\nwith a set of `EXPECTED` results, complete with the full item path of each item.\n', 'message': 'reformat basically the whole rustdoc.md\n', 'author_name': 'Who? Me?!', 'lang': 'md'}