From 647b8663d1aa21243169f7ac7eda77daf5eeaf8d Mon Sep 17 00:00:00 2001 From: Jonathan Turner Date: Wed, 17 Jan 2018 13:13:13 +1300 Subject: [PATCH 001/648] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d41418f2f..b8929c1e3 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,3 @@ -An attempt to docuemnt how the compiler works. The intention is to +An attempt to document how the compiler works. The intention is to bootstrap this "in repo" and eventually move the text over into the main rustc repo. From 11b19a4f4b6c78f10572c7cab8b5bb527a8f699b Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Wed, 17 Jan 2018 12:30:45 +0800 Subject: [PATCH 002/648] Created a couple basic CI scripts --- .travis.yml | 17 +++++++++++++++++ ci/github_pages.sh | 10 ++++++++++ ci/install.sh | 6 ++++++ 3 files changed, 33 insertions(+) create mode 100644 .travis.yml create mode 100644 ci/github_pages.sh create mode 100644 ci/install.sh diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 000000000..e1fcfa9bf --- /dev/null +++ b/.travis.yml @@ -0,0 +1,17 @@ +language: rust + +cache: pip + +install: + - bash ci/install.sh + - source ~/.cargo/env || true + +script: + - true + +after_success: + - bash ci/github_pages.sh + +notifications: + email: + on_success: never \ No newline at end of file diff --git a/ci/github_pages.sh b/ci/github_pages.sh new file mode 100644 index 000000000..603b280f9 --- /dev/null +++ b/ci/github_pages.sh @@ -0,0 +1,10 @@ +#!/bin/bash +set -ex + +BOOK_DIR=book + +# Only upload the built book to github pages if it's a commit to master +if [ "$TRAVIS_BRANCH" = master -a "$TRAVIS_PULL_REQUEST" = false ]; then + mdbook build + ghp-import $BOOK_DIR +fi \ No newline at end of file diff --git a/ci/install.sh b/ci/install.sh new file mode 100644 index 000000000..d9cb369e5 --- /dev/null +++ b/ci/install.sh @@ -0,0 +1,6 @@ +#!/bin/bash +set -ex + +if command -v ghp-import >/dev/null 2>&1; then + pip install ghp-import +fi \ No newline at end of file From 78960d989abc2ebfd55f1fda18338bec652c47ee Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 17 Jan 2018 09:19:24 -0500 Subject: [PATCH 003/648] improve the README --- README.md | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index b8929c1e3..beb2507cd 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,25 @@ -An attempt to document how the compiler works. The intention is to -bootstrap this "in repo" and eventually move the text over into the -main rustc repo. +This is a collaborate effort to build a guide that explains how rustc +works. The aim of the guide is to help new contributors get oriented +to rustc, as well as to help more experienced folks in figuring out +some new part of the compiler that they haven't worked on before. + +The guide can be useful today, but it has a lot of work still go. +Once it gets more complete, the plan is probably to move it into the +[main Rust repository](https://github.com/rust-lang/rust/). + +### Contributing to the guide + +If you'd like to help finish the guide, we'd love to have you! The +main tracking issue for the guide +[can be found here](https://github.com/rust-lang-nursery/rustc-guide/issues/6). From +there, you can find a list of all the planned chapters and subsections +-- if you think something is missing, please open an issue about it! +Otherwise, find a chapter that sounds interesting to you and then go +to its associated issue. There should be a list of things to do. + +**In general, if you don't know how the compiler works, that is not a +problem!** In that case, what we will do is to schedule a bit of time +for you to talk with someone who **does** know the code, or who wants +to pair with you and figure it out. Then you can work on writing up +what you learned. + From 5e297beb81bab07acfd8f2b0d94335e4744a38b7 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 17 Jan 2018 09:25:59 -0500 Subject: [PATCH 004/648] remove `src/src` directory that somehow got created --- src/{src => }/SUMMARY.md | 0 src/{src => }/chap-010-how-to-build-and-run.md | 0 src/{src => }/chap-020-running-tests.md | 0 src/{src => }/chap-030-walkthrough.md | 0 src/{src => }/chap-040-compiler-conventions.md | 0 src/{src => }/chap-050-the-parser.md | 0 src/{src => }/chap-060-macro-expansion.md | 0 src/{src => }/chap-070-name-resolution.md | 0 src/{src => }/chap-080-hir-lowering.md | 0 src/{src => }/chap-090-ty.md | 0 src/{src => }/chap-100-type-inference.md | 0 src/{src => }/chap-110-trait-resolution.md | 0 src/{src => }/chap-120-type-checking.md | 0 src/{src => }/chap-130-mir-construction.md | 0 src/{src => }/chap-140-mir-borrowck.md | 0 src/{src => }/chap-150-mir-optimizations.md | 0 src/{src => }/chap-160-trans.md | 0 src/{src => }/glossary.md | 0 18 files changed, 0 insertions(+), 0 deletions(-) rename src/{src => }/SUMMARY.md (100%) rename src/{src => }/chap-010-how-to-build-and-run.md (100%) rename src/{src => }/chap-020-running-tests.md (100%) rename src/{src => }/chap-030-walkthrough.md (100%) rename src/{src => }/chap-040-compiler-conventions.md (100%) rename src/{src => }/chap-050-the-parser.md (100%) rename src/{src => }/chap-060-macro-expansion.md (100%) rename src/{src => }/chap-070-name-resolution.md (100%) rename src/{src => }/chap-080-hir-lowering.md (100%) rename src/{src => }/chap-090-ty.md (100%) rename src/{src => }/chap-100-type-inference.md (100%) rename src/{src => }/chap-110-trait-resolution.md (100%) rename src/{src => }/chap-120-type-checking.md (100%) rename src/{src => }/chap-130-mir-construction.md (100%) rename src/{src => }/chap-140-mir-borrowck.md (100%) rename src/{src => }/chap-150-mir-optimizations.md (100%) rename src/{src => }/chap-160-trans.md (100%) rename src/{src => }/glossary.md (100%) diff --git a/src/src/SUMMARY.md b/src/SUMMARY.md similarity index 100% rename from src/src/SUMMARY.md rename to src/SUMMARY.md diff --git a/src/src/chap-010-how-to-build-and-run.md b/src/chap-010-how-to-build-and-run.md similarity index 100% rename from src/src/chap-010-how-to-build-and-run.md rename to src/chap-010-how-to-build-and-run.md diff --git a/src/src/chap-020-running-tests.md b/src/chap-020-running-tests.md similarity index 100% rename from src/src/chap-020-running-tests.md rename to src/chap-020-running-tests.md diff --git a/src/src/chap-030-walkthrough.md b/src/chap-030-walkthrough.md similarity index 100% rename from src/src/chap-030-walkthrough.md rename to src/chap-030-walkthrough.md diff --git a/src/src/chap-040-compiler-conventions.md b/src/chap-040-compiler-conventions.md similarity index 100% rename from src/src/chap-040-compiler-conventions.md rename to src/chap-040-compiler-conventions.md diff --git a/src/src/chap-050-the-parser.md b/src/chap-050-the-parser.md similarity index 100% rename from src/src/chap-050-the-parser.md rename to src/chap-050-the-parser.md diff --git a/src/src/chap-060-macro-expansion.md b/src/chap-060-macro-expansion.md similarity index 100% rename from src/src/chap-060-macro-expansion.md rename to src/chap-060-macro-expansion.md diff --git a/src/src/chap-070-name-resolution.md b/src/chap-070-name-resolution.md similarity index 100% rename from src/src/chap-070-name-resolution.md rename to src/chap-070-name-resolution.md diff --git a/src/src/chap-080-hir-lowering.md b/src/chap-080-hir-lowering.md similarity index 100% rename from src/src/chap-080-hir-lowering.md rename to src/chap-080-hir-lowering.md diff --git a/src/src/chap-090-ty.md b/src/chap-090-ty.md similarity index 100% rename from src/src/chap-090-ty.md rename to src/chap-090-ty.md diff --git a/src/src/chap-100-type-inference.md b/src/chap-100-type-inference.md similarity index 100% rename from src/src/chap-100-type-inference.md rename to src/chap-100-type-inference.md diff --git a/src/src/chap-110-trait-resolution.md b/src/chap-110-trait-resolution.md similarity index 100% rename from src/src/chap-110-trait-resolution.md rename to src/chap-110-trait-resolution.md diff --git a/src/src/chap-120-type-checking.md b/src/chap-120-type-checking.md similarity index 100% rename from src/src/chap-120-type-checking.md rename to src/chap-120-type-checking.md diff --git a/src/src/chap-130-mir-construction.md b/src/chap-130-mir-construction.md similarity index 100% rename from src/src/chap-130-mir-construction.md rename to src/chap-130-mir-construction.md diff --git a/src/src/chap-140-mir-borrowck.md b/src/chap-140-mir-borrowck.md similarity index 100% rename from src/src/chap-140-mir-borrowck.md rename to src/chap-140-mir-borrowck.md diff --git a/src/src/chap-150-mir-optimizations.md b/src/chap-150-mir-optimizations.md similarity index 100% rename from src/src/chap-150-mir-optimizations.md rename to src/chap-150-mir-optimizations.md diff --git a/src/src/chap-160-trans.md b/src/chap-160-trans.md similarity index 100% rename from src/src/chap-160-trans.md rename to src/chap-160-trans.md diff --git a/src/src/glossary.md b/src/glossary.md similarity index 100% rename from src/src/glossary.md rename to src/glossary.md From 54145323f8aafad125f60b8cc44ddf7a5bf0408a Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 17 Jan 2018 09:53:45 -0500 Subject: [PATCH 005/648] add link to gh-pages --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index beb2507cd..5ff51de2b 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,8 @@ works. The aim of the guide is to help new contributors get oriented to rustc, as well as to help more experienced folks in figuring out some new part of the compiler that they haven't worked on before. +[You can read the latest version of the guide here.](https://rust-lang-nursery.github.io/rustc-guide/) + The guide can be useful today, but it has a lot of work still go. Once it gets more complete, the plan is probably to move it into the [main Rust repository](https://github.com/rust-lang/rust/). From ae4b5503b2a7178873838e737c88eb8a4889c698 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 19 Jan 2018 06:38:33 -0500 Subject: [PATCH 006/648] add an "about this guide" section --- src/SUMMARY.md | 1 + src/chap-000-about-this-guide.md | 14 ++++++++++++++ 2 files changed, 15 insertions(+) create mode 100644 src/chap-000-about-this-guide.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 21c871885..30dde6fb9 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -1,5 +1,6 @@ # Summary +- [About this guide](./chap-000-about-this-guide.md) - [How to build the compiler and run what you built](./chap-010-how-to-build-and-run.md) - [Using the compiler testing framework](./chap-020-running-tests.md) - [Walkthrough: a typical contribution](./chap-030-walkthrough.md) diff --git a/src/chap-000-about-this-guide.md b/src/chap-000-about-this-guide.md new file mode 100644 index 000000000..ea840177b --- /dev/null +++ b/src/chap-000-about-this-guide.md @@ -0,0 +1,14 @@ +# About this guide + +This guide is meant to help document how rustc -- the Rust compiler -- +works, as well as to help new contributors get involved in rustc +development. It is not meant to replace code documentation -- each +chapter gives only high-level details, the kinds of things that +(ideally) don't change frequently. + +The guide itself is of course open source as well, and the sources can +be found at [the GitHub repository]. If you find any mistakes in the +guide, please file an issue about it -- or, even better, open a PR +with a correction! + +[the GitHub repository]: https://github.com/rust-lang-nursery/rustc-guide/ From 6cafcbb06f85b95a99d8f44b554a6150c2fce5b0 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 19 Jan 2018 06:38:40 -0500 Subject: [PATCH 007/648] replace "conventions" with "high-level overview" Also bring in material from the librustc README.md --- src/SUMMARY.md | 2 +- src/chap-040-compiler-conventions.md | 1 - src/chap-040-high-level-overview.md | 141 +++++++++++++++++++++++++++ 3 files changed, 142 insertions(+), 2 deletions(-) delete mode 100644 src/chap-040-compiler-conventions.md create mode 100644 src/chap-040-high-level-overview.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 30dde6fb9..1c87e26ce 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -4,7 +4,7 @@ - [How to build the compiler and run what you built](./chap-010-how-to-build-and-run.md) - [Using the compiler testing framework](./chap-020-running-tests.md) - [Walkthrough: a typical contribution](./chap-030-walkthrough.md) -- [Conventions used in the compiler](./chap-040-compiler-conventions.md) +- [High-level overview of the compiler source](./chap-040-high-level-overview.md) - [The parser](./chap-050-the-parser.md) - [Macro expansion](./chap-060-macro-expansion.md) - [Name resolution](./chap-070-name-resolution.md) diff --git a/src/chap-040-compiler-conventions.md b/src/chap-040-compiler-conventions.md deleted file mode 100644 index 314221fa0..000000000 --- a/src/chap-040-compiler-conventions.md +++ /dev/null @@ -1 +0,0 @@ -# Conventions used in the compiler diff --git a/src/chap-040-high-level-overview.md b/src/chap-040-high-level-overview.md new file mode 100644 index 000000000..b8d75bff6 --- /dev/null +++ b/src/chap-040-high-level-overview.md @@ -0,0 +1,141 @@ +# High-level overview of the compiler source + +## Crate structure + +The main Rust repository consists of a `src` directory, under which +there live many crates. These crates contain the sources for the +standard library and the compiler. This document, of course, focuses +on the latter. + +Rustc consists of a number of crates, including `syntax`, +`rustc`, `rustc_back`, `rustc_trans`, `rustc_driver`, and +many more. The source for each crate can be found in a directory +like `src/libXXX`, where `XXX` is the crate name. + +(NB. The names and divisions of these crates are not set in +stone and may change over time -- for the time being, we tend towards +a finer-grained division to help with compilation time, though as +incremental improves that may change.) + +The dependency structure of these crates is roughly a diamond: + +``` + rustc_driver + / | \ + / | \ + / | \ + / v \ +rustc_trans rustc_borrowck ... rustc_metadata + \ | / + \ | / + \ | / + \ v / + rustc + | + v + syntax + / \ + / \ + syntax_pos syntax_ext +``` + +The `rustc_driver` crate, at the top of this lattice, is effectively +the "main" function for the rust compiler. It doesn't have much "real +code", but instead ties together all of the code defined in the other +crates and defines the overall flow of execution. (As we transition +more and more to the [query model](ty/maps/README.md), however, the +"flow" of compilation is becoming less centrally defined.) + +At the other extreme, the `rustc` crate defines the common and +pervasive data structures that all the rest of the compiler uses +(e.g., how to represent types, traits, and the program itself). It +also contains some amount of the compiler itself, although that is +relatively limited. + +Finally, all the crates in the bulge in the middle define the bulk of +the compiler -- they all depend on `rustc`, so that they can make use +of the various types defined there, and they export public routines +that `rustc_driver` will invoke as needed (more and more, what these +crates export are "query definitions", but those are covered later +on). + +Below `rustc` lie various crates that make up the parser and error +reporting mechanism. For historical reasons, these crates do not have +the `rustc_` prefix, but they are really just as much an internal part +of the compiler and not intended to be stable (though they do wind up +getting used by some crates in the wild; a practice we hope to +gradually phase out). + +Each crate has a `README.md` file that describes, at a high-level, +what it contains, and tries to give some kind of explanation (some +better than others). + +## The main stages of compilation + +The Rust compiler is in a bit of transition right now. It used to be a +purely "pass-based" compiler, where we ran a number of passes over the +entire program, and each did a particular check of transformation. We +are gradually replacing this pass-based code with an alternative setup +based on on-demand **queries**. In the query-model, we work backwards, +executing a *query* that expresses our ultimate goal (e.g., "compile +this crate"). This query in turn may make other queries (e.g., "get me +a list of all modules in the crate"). Those queries make other queries +that ultimately bottom out in the base operations, like parsing the +input, running the type-checker, and so forth. This on-demand model +permits us to do exciting things like only do the minimal amount of +work needed to type-check a single function. It also helps with +incremental compilation. (For details on defining queries, check out +`src/librustc/ty/maps/README.md`.) + +Regardless of the general setup, the basic operations that the +compiler must perform are the same. The only thing that changes is +whether these operations are invoked front-to-back, or on demand. In +order to compile a Rust crate, these are the general steps that we +take: + +1. **Parsing input** + - this processes the `.rs` files and produces the AST ("abstract syntax tree") + - the AST is defined in `syntax/ast.rs`. It is intended to match the lexical + syntax of the Rust language quite closely. +2. **Name resolution, macro expansion, and configuration** + - once parsing is complete, we process the AST recursively, resolving paths + and expanding macros. This same process also processes `#[cfg]` nodes, and hence + may strip things out of the AST as well. +3. **Lowering to HIR** + - Once name resolution completes, we convert the AST into the HIR, + or "high-level IR". The HIR is defined in `src/librustc/hir/`; that module also includes + the lowering code. + - The HIR is a lightly desugared variant of the AST. It is more processed than the + AST and more suitable for the analyses that follow. It is **not** required to match + the syntax of the Rust language. + - As a simple example, in the **AST**, we preserve the parentheses + that the user wrote, so `((1 + 2) + 3)` and `1 + 2 + 3` parse + into distinct trees, even though they are equivalent. In the + HIR, however, parentheses nodes are removed, and those two + expressions are represented in the same way. +3. **Type-checking and subsequent analyses** + - An important step in processing the HIR is to perform type + checking. This process assigns types to every HIR expression, + for example, and also is responsible for resolving some + "type-dependent" paths, such as field accesses (`x.f` -- we + can't know what field `f` is being accessed until we know the + type of `x`) and associated type references (`T::Item` -- we + can't know what type `Item` is until we know what `T` is). + - Type checking creates "side-tables" (`TypeckTables`) that include + the types of expressions, the way to resolve methods, and so forth. + - After type-checking, we can do other analyses, such as privacy checking. +4. **Lowering to MIR and post-processing** + - Once type-checking is done, we can lower the HIR into MIR ("middle IR"), which + is a **very** desugared version of Rust, well suited to the borrowck but also + certain high-level optimizations. +5. **Translation to LLVM and LLVM optimizations** + - From MIR, we can produce LLVM IR. + - LLVM then runs its various optimizations, which produces a number of `.o` files + (one for each "codegen unit"). +6. **Linking** + - Finally, those `.o` files are linked together. + + + + +The first thing you may wonder if From 3492c534fe72cb0964382b6d3fe933e69f3d7221 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 19 Jan 2018 06:42:50 -0500 Subject: [PATCH 008/648] remove the `chap-NNN-` convention from file names I don't want those showing up in the URLs. --- src/SUMMARY.md | 34 +++++++++---------- ...bout-this-guide.md => about-this-guide.md} | 0 ...vel-overview.md => high-level-overview.md} | 0 ...ap-080-hir-lowering.md => hir-lowering.md} | 0 ...ild-and-run.md => how-to-build-and-run.md} | 0 ...-macro-expansion.md => macro-expansion.md} | 0 ...ap-140-mir-borrowck.md => mir-borrowck.md} | 0 ...ir-construction.md => mir-construction.md} | 0 ...-optimizations.md => mir-optimizations.md} | 0 ...-name-resolution.md => name-resolution.md} | 0 ...-020-running-tests.md => running-tests.md} | 0 src/{chap-050-the-parser.md => the-parser.md} | 0 ...rait-resolution.md => trait-resolution.md} | 0 src/{chap-160-trans.md => trans.md} | 0 src/{chap-090-ty.md => ty.md} | 0 ...-120-type-checking.md => type-checking.md} | 0 ...00-type-inference.md => type-inference.md} | 0 ...chap-030-walkthrough.md => walkthrough.md} | 0 18 files changed, 17 insertions(+), 17 deletions(-) rename src/{chap-000-about-this-guide.md => about-this-guide.md} (100%) rename src/{chap-040-high-level-overview.md => high-level-overview.md} (100%) rename src/{chap-080-hir-lowering.md => hir-lowering.md} (100%) rename src/{chap-010-how-to-build-and-run.md => how-to-build-and-run.md} (100%) rename src/{chap-060-macro-expansion.md => macro-expansion.md} (100%) rename src/{chap-140-mir-borrowck.md => mir-borrowck.md} (100%) rename src/{chap-130-mir-construction.md => mir-construction.md} (100%) rename src/{chap-150-mir-optimizations.md => mir-optimizations.md} (100%) rename src/{chap-070-name-resolution.md => name-resolution.md} (100%) rename src/{chap-020-running-tests.md => running-tests.md} (100%) rename src/{chap-050-the-parser.md => the-parser.md} (100%) rename src/{chap-110-trait-resolution.md => trait-resolution.md} (100%) rename src/{chap-160-trans.md => trans.md} (100%) rename src/{chap-090-ty.md => ty.md} (100%) rename src/{chap-120-type-checking.md => type-checking.md} (100%) rename src/{chap-100-type-inference.md => type-inference.md} (100%) rename src/{chap-030-walkthrough.md => walkthrough.md} (100%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 1c87e26ce..b1dfa440d 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -1,20 +1,20 @@ # Summary -- [About this guide](./chap-000-about-this-guide.md) -- [How to build the compiler and run what you built](./chap-010-how-to-build-and-run.md) -- [Using the compiler testing framework](./chap-020-running-tests.md) -- [Walkthrough: a typical contribution](./chap-030-walkthrough.md) -- [High-level overview of the compiler source](./chap-040-high-level-overview.md) -- [The parser](./chap-050-the-parser.md) -- [Macro expansion](./chap-060-macro-expansion.md) -- [Name resolution](./chap-070-name-resolution.md) -- [HIR lowering](./chap-080-hir-lowering.md) -- [Representing types (`ty` module in depth)](./chap-090-ty.md) -- [Type inference](./chap-100-type-inference.md) -- [Trait resolution](./chap-110-trait-resolution.md) -- [Type checking](./chap-120-type-checking.md) -- [MIR construction](./chap-130-mir-construction.md) -- [MIR borrowck](./chap-140-mir-borrowck.md) -- [MIR optimizations](./chap-150-mir-optimizations.md) -- [trans: generating LLVM IR](./chap-160-trans.md) +- [About this guide](./about-this-guide.md) +- [How to build the compiler and run what you built](./how-to-build-and-run.md) +- [Using the compiler testing framework](./running-tests.md) +- [Walkthrough: a typical contribution](./walkthrough.md) +- [High-level overview of the compiler source](./high-level-overview.md) +- [The parser](./the-parser.md) +- [Macro expansion](./macro-expansion.md) +- [Name resolution](./name-resolution.md) +- [HIR lowering](./hir-lowering.md) +- [Representing types (`ty` module in depth)](./ty.md) +- [Type inference](./type-inference.md) +- [Trait resolution](./trait-resolution.md) +- [Type checking](./type-checking.md) +- [MIR construction](./mir-construction.md) +- [MIR borrowck](./mir-borrowck.md) +- [MIR optimizations](./mir-optimizations.md) +- [trans: generating LLVM IR](./trans.md) - [Glossary](./glossary.md) diff --git a/src/chap-000-about-this-guide.md b/src/about-this-guide.md similarity index 100% rename from src/chap-000-about-this-guide.md rename to src/about-this-guide.md diff --git a/src/chap-040-high-level-overview.md b/src/high-level-overview.md similarity index 100% rename from src/chap-040-high-level-overview.md rename to src/high-level-overview.md diff --git a/src/chap-080-hir-lowering.md b/src/hir-lowering.md similarity index 100% rename from src/chap-080-hir-lowering.md rename to src/hir-lowering.md diff --git a/src/chap-010-how-to-build-and-run.md b/src/how-to-build-and-run.md similarity index 100% rename from src/chap-010-how-to-build-and-run.md rename to src/how-to-build-and-run.md diff --git a/src/chap-060-macro-expansion.md b/src/macro-expansion.md similarity index 100% rename from src/chap-060-macro-expansion.md rename to src/macro-expansion.md diff --git a/src/chap-140-mir-borrowck.md b/src/mir-borrowck.md similarity index 100% rename from src/chap-140-mir-borrowck.md rename to src/mir-borrowck.md diff --git a/src/chap-130-mir-construction.md b/src/mir-construction.md similarity index 100% rename from src/chap-130-mir-construction.md rename to src/mir-construction.md diff --git a/src/chap-150-mir-optimizations.md b/src/mir-optimizations.md similarity index 100% rename from src/chap-150-mir-optimizations.md rename to src/mir-optimizations.md diff --git a/src/chap-070-name-resolution.md b/src/name-resolution.md similarity index 100% rename from src/chap-070-name-resolution.md rename to src/name-resolution.md diff --git a/src/chap-020-running-tests.md b/src/running-tests.md similarity index 100% rename from src/chap-020-running-tests.md rename to src/running-tests.md diff --git a/src/chap-050-the-parser.md b/src/the-parser.md similarity index 100% rename from src/chap-050-the-parser.md rename to src/the-parser.md diff --git a/src/chap-110-trait-resolution.md b/src/trait-resolution.md similarity index 100% rename from src/chap-110-trait-resolution.md rename to src/trait-resolution.md diff --git a/src/chap-160-trans.md b/src/trans.md similarity index 100% rename from src/chap-160-trans.md rename to src/trans.md diff --git a/src/chap-090-ty.md b/src/ty.md similarity index 100% rename from src/chap-090-ty.md rename to src/ty.md diff --git a/src/chap-120-type-checking.md b/src/type-checking.md similarity index 100% rename from src/chap-120-type-checking.md rename to src/type-checking.md diff --git a/src/chap-100-type-inference.md b/src/type-inference.md similarity index 100% rename from src/chap-100-type-inference.md rename to src/type-inference.md diff --git a/src/chap-030-walkthrough.md b/src/walkthrough.md similarity index 100% rename from src/chap-030-walkthrough.md rename to src/walkthrough.md From dfa328fc39a7724803e6f26518609e5b497529ef Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 19 Jan 2018 06:43:10 -0500 Subject: [PATCH 009/648] add in the trait resolution README from rustc --- src/trait-resolution.md | 484 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 484 insertions(+) diff --git a/src/trait-resolution.md b/src/trait-resolution.md index dd16aac62..58efbd050 100644 --- a/src/trait-resolution.md +++ b/src/trait-resolution.md @@ -1 +1,485 @@ # Trait resolution + +This document describes the general process and points out some non-obvious +things. + +**WARNING:** This material was moved verbatim from a rustc README, so +it may not "fit" the style of the guide until it is adapted. + +## Major concepts + +Trait resolution is the process of pairing up an impl with each +reference to a trait. So, for example, if there is a generic function like: + +```rust +fn clone_slice(x: &[T]) -> Vec { /*...*/ } +``` + +and then a call to that function: + +```rust +let v: Vec = clone_slice(&[1, 2, 3]) +``` + +it is the job of trait resolution to figure out (in which case) +whether there exists an impl of `isize : Clone` + +Note that in some cases, like generic functions, we may not be able to +find a specific impl, but we can figure out that the caller must +provide an impl. To see what I mean, consider the body of `clone_slice`: + +```rust +fn clone_slice(x: &[T]) -> Vec { + let mut v = Vec::new(); + for e in &x { + v.push((*e).clone()); // (*) + } +} +``` + +The line marked `(*)` is only legal if `T` (the type of `*e`) +implements the `Clone` trait. Naturally, since we don't know what `T` +is, we can't find the specific impl; but based on the bound `T:Clone`, +we can say that there exists an impl which the caller must provide. + +We use the term *obligation* to refer to a trait reference in need of +an impl. + +## Overview + +Trait resolution consists of three major parts: + +- SELECTION: Deciding how to resolve a specific obligation. For + example, selection might decide that a specific obligation can be + resolved by employing an impl which matches the self type, or by + using a parameter bound. In the case of an impl, Selecting one + obligation can create *nested obligations* because of where clauses + on the impl itself. It may also require evaluating those nested + obligations to resolve ambiguities. + +- FULFILLMENT: The fulfillment code is what tracks that obligations + are completely fulfilled. Basically it is a worklist of obligations + to be selected: once selection is successful, the obligation is + removed from the worklist and any nested obligations are enqueued. + +- COHERENCE: The coherence checks are intended to ensure that there + are never overlapping impls, where two impls could be used with + equal precedence. + +## Selection + +Selection is the process of deciding whether an obligation can be +resolved and, if so, how it is to be resolved (via impl, where clause, etc). +The main interface is the `select()` function, which takes an obligation +and returns a `SelectionResult`. There are three possible outcomes: + +- `Ok(Some(selection))` -- yes, the obligation can be resolved, and + `selection` indicates how. If the impl was resolved via an impl, + then `selection` may also indicate nested obligations that are required + by the impl. + +- `Ok(None)` -- we are not yet sure whether the obligation can be + resolved or not. This happens most commonly when the obligation + contains unbound type variables. + +- `Err(err)` -- the obligation definitely cannot be resolved due to a + type error, or because there are no impls that could possibly apply, + etc. + +The basic algorithm for selection is broken into two big phases: +candidate assembly and confirmation. + +### Candidate assembly + +Searches for impls/where-clauses/etc that might +possibly be used to satisfy the obligation. Each of those is called +a candidate. To avoid ambiguity, we want to find exactly one +candidate that is definitively applicable. In some cases, we may not +know whether an impl/where-clause applies or not -- this occurs when +the obligation contains unbound inference variables. + +The basic idea for candidate assembly is to do a first pass in which +we identify all possible candidates. During this pass, all that we do +is try and unify the type parameters. (In particular, we ignore any +nested where clauses.) Presuming that this unification succeeds, the +impl is added as a candidate. + +Once this first pass is done, we can examine the set of candidates. If +it is a singleton set, then we are done: this is the only impl in +scope that could possibly apply. Otherwise, we can winnow down the set +of candidates by using where clauses and other conditions. If this +reduced set yields a single, unambiguous entry, we're good to go, +otherwise the result is considered ambiguous. + +#### The basic process: Inferring based on the impls we see + +This process is easier if we work through some examples. Consider +the following trait: + +```rust +trait Convert { + fn convert(&self) -> Target; +} +``` + +This trait just has one method. It's about as simple as it gets. It +converts from the (implicit) `Self` type to the `Target` type. If we +wanted to permit conversion between `isize` and `usize`, we might +implement `Convert` like so: + +```rust +impl Convert for isize { /*...*/ } // isize -> usize +impl Convert for usize { /*...*/ } // usize -> isize +``` + +Now imagine there is some code like the following: + +```rust +let x: isize = ...; +let y = x.convert(); +``` + +The call to convert will generate a trait reference `Convert<$Y> for +isize`, where `$Y` is the type variable representing the type of +`y`. When we match this against the two impls we can see, we will find +that only one remains: `Convert for isize`. Therefore, we can +select this impl, which will cause the type of `$Y` to be unified to +`usize`. (Note that while assembling candidates, we do the initial +unifications in a transaction, so that they don't affect one another.) + +There are tests to this effect in src/test/run-pass: + + traits-multidispatch-infer-convert-source-and-target.rs + traits-multidispatch-infer-convert-target.rs + +#### Winnowing: Resolving ambiguities + +But what happens if there are multiple impls where all the types +unify? Consider this example: + +```rust +trait Get { + fn get(&self) -> Self; +} + +impl Get for T { + fn get(&self) -> T { *self } +} + +impl Get for Box { + fn get(&self) -> Box { box get_it(&**self) } +} +``` + +What happens when we invoke `get_it(&box 1_u16)`, for example? In this +case, the `Self` type is `Box` -- that unifies with both impls, +because the first applies to all types, and the second to all +boxes. In the olden days we'd have called this ambiguous. But what we +do now is do a second *winnowing* pass that considers where clauses +and attempts to remove candidates -- in this case, the first impl only +applies if `Box : Copy`, which doesn't hold. After winnowing, +then, we are left with just one candidate, so we can proceed. There is +a test of this in `src/test/run-pass/traits-conditional-dispatch.rs`. + +#### Matching + +The subroutines that decide whether a particular impl/where-clause/etc +applies to a particular obligation. At the moment, this amounts to +unifying the self types, but in the future we may also recursively +consider some of the nested obligations, in the case of an impl. + +#### Lifetimes and selection + +Because of how that lifetime inference works, it is not possible to +give back immediate feedback as to whether a unification or subtype +relationship between lifetimes holds or not. Therefore, lifetime +matching is *not* considered during selection. This is reflected in +the fact that subregion assignment is infallible. This may yield +lifetime constraints that will later be found to be in error (in +contrast, the non-lifetime-constraints have already been checked +during selection and can never cause an error, though naturally they +may lead to other errors downstream). + +#### Where clauses + +Besides an impl, the other major way to resolve an obligation is via a +where clause. The selection process is always given a *parameter +environment* which contains a list of where clauses, which are +basically obligations that can assume are satisfiable. We will iterate +over that list and check whether our current obligation can be found +in that list, and if so it is considered satisfied. More precisely, we +want to check whether there is a where-clause obligation that is for +the same trait (or some subtrait) and for which the self types match, +using the definition of *matching* given above. + +Consider this simple example: + +```rust +trait A1 { /*...*/ } +trait A2 : A1 { /*...*/ } + +trait B { /*...*/ } + +fn foo { /*...*/ } +``` + +Clearly we can use methods offered by `A1`, `A2`, or `B` within the +body of `foo`. In each case, that will incur an obligation like `X : +A1` or `X : A2`. The parameter environment will contain two +where-clauses, `X : A2` and `X : B`. For each obligation, then, we +search this list of where-clauses. To resolve an obligation `X:A1`, +we would note that `X:A2` implies that `X:A1`. + +### Confirmation + +Confirmation unifies the output type parameters of the trait with the +values found in the obligation, possibly yielding a type error. If we +return to our example of the `Convert` trait from the previous +section, confirmation is where an error would be reported, because the +impl specified that `T` would be `usize`, but the obligation reported +`char`. Hence the result of selection would be an error. + +### Selection during translation + +During type checking, we do not store the results of trait selection. +We simply wish to verify that trait selection will succeed. Then +later, at trans time, when we have all concrete types available, we +can repeat the trait selection. In this case, we do not consider any +where-clauses to be in scope. We know that therefore each resolution +will resolve to a particular impl. + +One interesting twist has to do with nested obligations. In general, in trans, +we only need to do a "shallow" selection for an obligation. That is, we wish to +identify which impl applies, but we do not (yet) need to decide how to select +any nested obligations. Nonetheless, we *do* currently do a complete resolution, +and that is because it can sometimes inform the results of type inference. That is, +we do not have the full substitutions in terms of the type variables of the impl available +to us, so we must run trait selection to figure everything out. + +Here is an example: + +```rust +trait Foo { /*...*/ } +impl> Foo for Vec { /*...*/ } + +impl Bar for isize { /*...*/ } +``` + +After one shallow round of selection for an obligation like `Vec +: Foo`, we would know which impl we want, and we would know that +`T=isize`, but we do not know the type of `U`. We must select the +nested obligation `isize : Bar` to find out that `U=usize`. + +It would be good to only do *just as much* nested resolution as +necessary. Currently, though, we just do a full resolution. + +# Higher-ranked trait bounds + +One of the more subtle concepts at work are *higher-ranked trait +bounds*. An example of such a bound is `for<'a> MyTrait<&'a isize>`. +Let's walk through how selection on higher-ranked trait references +works. + +## Basic matching and skolemization leaks + +Let's walk through the test `compile-fail/hrtb-just-for-static.rs` to see +how it works. The test starts with the trait `Foo`: + +```rust +trait Foo { + fn foo(&self, x: X) { } +} +``` + +Let's say we have a function `want_hrtb` that wants a type which +implements `Foo<&'a isize>` for any `'a`: + +```rust +fn want_hrtb() where T : for<'a> Foo<&'a isize> { ... } +``` + +Now we have a struct `AnyInt` that implements `Foo<&'a isize>` for any +`'a`: + +```rust +struct AnyInt; +impl<'a> Foo<&'a isize> for AnyInt { } +``` + +And the question is, does `AnyInt : for<'a> Foo<&'a isize>`? We want the +answer to be yes. The algorithm for figuring it out is closely related +to the subtyping for higher-ranked types (which is described in +`middle::infer::higher_ranked::doc`, but also in a [paper by SPJ] that +I recommend you read). + +1. Skolemize the obligation. +2. Match the impl against the skolemized obligation. +3. Check for skolemization leaks. + +[paper by SPJ]: http://research.microsoft.com/en-us/um/people/simonpj/papers/higher-rank/ + +So let's work through our example. The first thing we would do is to +skolemize the obligation, yielding `AnyInt : Foo<&'0 isize>` (here `'0` +represents skolemized region #0). Note that now have no quantifiers; +in terms of the compiler type, this changes from a `ty::PolyTraitRef` +to a `TraitRef`. We would then create the `TraitRef` from the impl, +using fresh variables for it's bound regions (and thus getting +`Foo<&'$a isize>`, where `'$a` is the inference variable for `'a`). Next +we relate the two trait refs, yielding a graph with the constraint +that `'0 == '$a`. Finally, we check for skolemization "leaks" -- a +leak is basically any attempt to relate a skolemized region to another +skolemized region, or to any region that pre-existed the impl match. +The leak check is done by searching from the skolemized region to find +the set of regions that it is related to in any way. This is called +the "taint" set. To pass the check, that set must consist *solely* of +itself and region variables from the impl. If the taint set includes +any other region, then the match is a failure. In this case, the taint +set for `'0` is `{'0, '$a}`, and hence the check will succeed. + +Let's consider a failure case. Imagine we also have a struct + +```rust +struct StaticInt; +impl Foo<&'static isize> for StaticInt; +``` + +We want the obligation `StaticInt : for<'a> Foo<&'a isize>` to be +considered unsatisfied. The check begins just as before. `'a` is +skolemized to `'0` and the impl trait reference is instantiated to +`Foo<&'static isize>`. When we relate those two, we get a constraint +like `'static == '0`. This means that the taint set for `'0` is `{'0, +'static}`, which fails the leak check. + +## Higher-ranked trait obligations + +Once the basic matching is done, we get to another interesting topic: +how to deal with impl obligations. I'll work through a simple example +here. Imagine we have the traits `Foo` and `Bar` and an associated impl: + +```rust +trait Foo { + fn foo(&self, x: X) { } +} + +trait Bar { + fn bar(&self, x: X) { } +} + +impl Foo for F + where F : Bar +{ +} +``` + +Now let's say we have a obligation `for<'a> Foo<&'a isize>` and we match +this impl. What obligation is generated as a result? We want to get +`for<'a> Bar<&'a isize>`, but how does that happen? + +After the matching, we are in a position where we have a skolemized +substitution like `X => &'0 isize`. If we apply this substitution to the +impl obligations, we get `F : Bar<&'0 isize>`. Obviously this is not +directly usable because the skolemized region `'0` cannot leak out of +our computation. + +What we do is to create an inverse mapping from the taint set of `'0` +back to the original bound region (`'a`, here) that `'0` resulted +from. (This is done in `higher_ranked::plug_leaks`). We know that the +leak check passed, so this taint set consists solely of the skolemized +region itself plus various intermediate region variables. We then walk +the trait-reference and convert every region in that taint set back to +a late-bound region, so in this case we'd wind up with `for<'a> F : +Bar<&'a isize>`. + +# Caching and subtle considerations therewith + +In general we attempt to cache the results of trait selection. This +is a somewhat complex process. Part of the reason for this is that we +want to be able to cache results even when all the types in the trait +reference are not fully known. In that case, it may happen that the +trait selection process is also influencing type variables, so we have +to be able to not only cache the *result* of the selection process, +but *replay* its effects on the type variables. + +## An example + +The high-level idea of how the cache works is that we first replace +all unbound inference variables with skolemized versions. Therefore, +if we had a trait reference `usize : Foo<$1>`, where `$n` is an unbound +inference variable, we might replace it with `usize : Foo<%0>`, where +`%n` is a skolemized type. We would then look this up in the cache. +If we found a hit, the hit would tell us the immediate next step to +take in the selection process: i.e., apply impl #22, or apply where +clause `X : Foo`. Let's say in this case there is no hit. +Therefore, we search through impls and where clauses and so forth, and +we come to the conclusion that the only possible impl is this one, +with def-id 22: + +```rust +impl Foo for usize { ... } // Impl #22 +``` + +We would then record in the cache `usize : Foo<%0> ==> +ImplCandidate(22)`. Next we would confirm `ImplCandidate(22)`, which +would (as a side-effect) unify `$1` with `isize`. + +Now, at some later time, we might come along and see a `usize : +Foo<$3>`. When skolemized, this would yield `usize : Foo<%0>`, just as +before, and hence the cache lookup would succeed, yielding +`ImplCandidate(22)`. We would confirm `ImplCandidate(22)` which would +(as a side-effect) unify `$3` with `isize`. + +## Where clauses and the local vs global cache + +One subtle interaction is that the results of trait lookup will vary +depending on what where clauses are in scope. Therefore, we actually +have *two* caches, a local and a global cache. The local cache is +attached to the `ParamEnv` and the global cache attached to the +`tcx`. We use the local cache whenever the result might depend on the +where clauses that are in scope. The determination of which cache to +use is done by the method `pick_candidate_cache` in `select.rs`. At +the moment, we use a very simple, conservative rule: if there are any +where-clauses in scope, then we use the local cache. We used to try +and draw finer-grained distinctions, but that led to a serious of +annoying and weird bugs like #22019 and #18290. This simple rule seems +to be pretty clearly safe and also still retains a very high hit rate +(~95% when compiling rustc). + +# Specialization + +Defined in the `specialize` module. + +The basic strategy is to build up a *specialization graph* during +coherence checking. Insertion into the graph locates the right place +to put an impl in the specialization hierarchy; if there is no right +place (due to partial overlap but no containment), you get an overlap +error. Specialization is consulted when selecting an impl (of course), +and the graph is consulted when propagating defaults down the +specialization hierarchy. + +You might expect that the specialization graph would be used during +selection -- i.e., when actually performing specialization. This is +not done for two reasons: + +- It's merely an optimization: given a set of candidates that apply, + we can determine the most specialized one by comparing them directly + for specialization, rather than consulting the graph. Given that we + also cache the results of selection, the benefit of this + optimization is questionable. + +- To build the specialization graph in the first place, we need to use + selection (because we need to determine whether one impl specializes + another). Dealing with this reentrancy would require some additional + mode switch for selection. Given that there seems to be no strong + reason to use the graph anyway, we stick with a simpler approach in + selection, and use the graph only for propagating default + implementations. + +Trait impl selection can succeed even when multiple impls can apply, +as long as they are part of the same specialization family. In that +case, it returns a *single* impl on success -- this is the most +specialized impl *known* to apply. However, if there are any inference +variables in play, the returned impl may not be the actual impl we +will use at trans time. Thus, we take special care to avoid projecting +associated types unless either (1) the associated type does not use +`default` and thus cannot be overridden or (2) all input types are +known concretely. From 761119ef83bd3a4340948bb67bfeb64e1ac486e5 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 19 Jan 2018 06:51:14 -0500 Subject: [PATCH 010/648] move over the `ty` README --- src/SUMMARY.md | 2 +- src/ty.md | 166 ++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 166 insertions(+), 2 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index b1dfa440d..f8b1f1d6e 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -9,7 +9,7 @@ - [Macro expansion](./macro-expansion.md) - [Name resolution](./name-resolution.md) - [HIR lowering](./hir-lowering.md) -- [Representing types (`ty` module in depth)](./ty.md) +- [The `ty` module: representing types](./ty.md) - [Type inference](./type-inference.md) - [Trait resolution](./trait-resolution.md) - [Type checking](./type-checking.md) diff --git a/src/ty.md b/src/ty.md index 1762ce037..8debb71c7 100644 --- a/src/ty.md +++ b/src/ty.md @@ -1 +1,165 @@ -# Representing types (`ty` module in depth) +# The `ty` module: representing types + +The `ty` module defines how the Rust compiler represents types +internally. It also defines the *typing context* (`tcx` or `TyCtxt`), +which is the central data structure in the compiler. + +## The tcx and how it uses lifetimes + +The `tcx` ("typing context") is the central data structure in the +compiler. It is the context that you use to perform all manner of +queries. The struct `TyCtxt` defines a reference to this shared context: + +```rust +tcx: TyCtxt<'a, 'gcx, 'tcx> +// -- ---- ---- +// | | | +// | | innermost arena lifetime (if any) +// | "global arena" lifetime +// lifetime of this reference +``` + +As you can see, the `TyCtxt` type takes three lifetime parameters. +These lifetimes are perhaps the most complex thing to understand about +the tcx. During Rust compilation, we allocate most of our memory in +**arenas**, which are basically pools of memory that get freed all at +once. When you see a reference with a lifetime like `'tcx` or `'gcx`, +you know that it refers to arena-allocated data (or data that lives as +long as the arenas, anyhow). + +We use two distinct levels of arenas. The outer level is the "global +arena". This arena lasts for the entire compilation: so anything you +allocate in there is only freed once compilation is basically over +(actually, when we shift to executing LLVM). + +To reduce peak memory usage, when we do type inference, we also use an +inner level of arena. These arenas get thrown away once type inference +is over. This is done because type inference generates a lot of +"throw-away" types that are not particularly interesting after type +inference completes, so keeping around those allocations would be +wasteful. + +Often, we wish to write code that explicitly asserts that it is not +taking place during inference. In that case, there is no "local" +arena, and all the types that you can access are allocated in the +global arena. To express this, the idea is to use the same lifetime +for the `'gcx` and `'tcx` parameters of `TyCtxt`. Just to be a touch +confusing, we tend to use the name `'tcx` in such contexts. Here is an +example: + +```rust +fn not_in_inference<'a, 'tcx>(tcx: TyCtxt<'a, 'tcx, 'tcx>, def_id: DefId) { + // ---- ---- + // Using the same lifetime here asserts + // that the innermost arena accessible through + // this reference *is* the global arena. +} +``` + +In contrast, if we want to code that can be usable during type inference, then you +need to declare a distinct `'gcx` and `'tcx` lifetime parameter: + +```rust +fn maybe_in_inference<'a, 'gcx, 'tcx>(tcx: TyCtxt<'a, 'gcx, 'tcx>, def_id: DefId) { + // ---- ---- + // Using different lifetimes here means that + // the innermost arena *may* be distinct + // from the global arena (but doesn't have to be). +} +``` + +### Allocating and working with types + +Rust types are represented using the `Ty<'tcx>` defined in the `ty` +module (not to be confused with the `Ty` struct from [the HIR]). This +is in fact a simple type alias for a reference with `'tcx` lifetime: + +```rust +pub type Ty<'tcx> = &'tcx TyS<'tcx>; +``` + +[the HIR]: ../hir/README.md + +You can basically ignore the `TyS` struct -- you will basically never +access it explicitly. We always pass it by reference using the +`Ty<'tcx>` alias -- the only exception I think is to define inherent +methods on types. Instances of `TyS` are only ever allocated in one of +the rustc arenas (never e.g. on the stack). + +One common operation on types is to **match** and see what kinds of +types they are. This is done by doing `match ty.sty`, sort of like this: + +```rust +fn test_type<'tcx>(ty: Ty<'tcx>) { + match ty.sty { + ty::TyArray(elem_ty, len) => { ... } + ... + } +} +``` + +The `sty` field (the origin of this name is unclear to me; perhaps +structural type?) is of type `TypeVariants<'tcx>`, which is an enum +defining all of the different kinds of types in the compiler. + +> NB: inspecting the `sty` field on types during type inference can be +> risky, as there may be inference variables and other things to +> consider, or sometimes types are not yet known that will become +> known later.). + +To allocate a new type, you can use the various `mk_` methods defined +on the `tcx`. These have names that correpond mostly to the various kinds +of type variants. For example: + +```rust +let array_ty = tcx.mk_array(elem_ty, len * 2); +``` + +These methods all return a `Ty<'tcx>` -- note that the lifetime you +get back is the lifetime of the innermost arena that this `tcx` has +access to. In fact, types are always canonicalized and interned (so we +never allocate exactly the same type twice) and are always allocated +in the outermost arena where they can be (so, if they do not contain +any inference variables or other "temporary" types, they will be +allocated in the global arena). However, the lifetime `'tcx` is always +a safe approximation, so that is what you get back. + +> NB. Because types are interned, it is possible to compare them for +> equality efficiently using `==` -- however, this is almost never what +> you want to do unless you happen to be hashing and looking for +> duplicates. This is because often in Rust there are multiple ways to +> represent the same type, particularly once inference is involved. If +> you are going to be testing for type equality, you probably need to +> start looking into the inference code to do it right. + +You can also find various common types in the `tcx` itself by accessing +`tcx.types.bool`, `tcx.types.char`, etc (see `CommonTypes` for more). + +### Beyond types: Other kinds of arena-allocated data structures + +In addition to types, there are a number of other arena-allocated data +structures that you can allocate, and which are found in this +module. Here are a few examples: + +- `Substs`, allocated with `mk_substs` -- this will intern a slice of types, often used to + specify the values to be substituted for generics (e.g., `HashMap` + would be represented as a slice `&'tcx [tcx.types.i32, tcx.types.u32]`). +- `TraitRef`, typically passed by value -- a **trait reference** + consists of a reference to a trait along with its various type + parameters (including `Self`), like `i32: Display` (here, the def-id + would reference the `Display` trait, and the substs would contain + `i32`). +- `Predicate` defines something the trait system has to prove (see `traits` module). + +### Import conventions + +Although there is no hard and fast rule, the `ty` module tends to be used like so: + +```rust +use ty::{self, Ty, TyCtxt}; +``` + +In particular, since they are so common, the `Ty` and `TyCtxt` types +are imported directly. Other types are often referenced with an +explicit `ty::` prefix (e.g., `ty::TraitRef<'tcx>`). But some modules +choose to import a larger or smaller set of names explicitly. From 74916aa83dea2a22593bfd12d870143614151854 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 19 Jan 2018 07:28:19 -0500 Subject: [PATCH 011/648] invoke `cargo` to install mdbook --- .travis.yml | 2 +- ci/install.sh | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index e1fcfa9bf..5978c61d8 100644 --- a/.travis.yml +++ b/.travis.yml @@ -3,8 +3,8 @@ language: rust cache: pip install: - - bash ci/install.sh - source ~/.cargo/env || true + - bash ci/install.sh script: - true diff --git a/ci/install.sh b/ci/install.sh index d9cb369e5..dd96daf17 100644 --- a/ci/install.sh +++ b/ci/install.sh @@ -1,6 +1,8 @@ #!/bin/bash set -ex +cargo install cargo install mdbook --vers "0.0.28" + if command -v ghp-import >/dev/null 2>&1; then pip install ghp-import -fi \ No newline at end of file +fi From 15ff0b7ef1f318fd49904aedf6f4b531612ed59b Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 19 Jan 2018 14:48:35 -0500 Subject: [PATCH 012/648] improvements to travis setup -- cache, use command -v --- .travis.yml | 4 +++- ci/install.sh | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 5978c61d8..36c3719f0 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,6 +1,8 @@ language: rust -cache: pip +cache: +- pip +- cargo install: - source ~/.cargo/env || true diff --git a/ci/install.sh b/ci/install.sh index dd96daf17..fac6488b6 100644 --- a/ci/install.sh +++ b/ci/install.sh @@ -1,7 +1,9 @@ #!/bin/bash set -ex -cargo install cargo install mdbook --vers "0.0.28" +if command -v mdbook >/dev/null 2>&1; then + cargo install mdbook --vers "0.0.28" +fi if command -v ghp-import >/dev/null 2>&1; then pip install ghp-import From 3c98b0f1894e4423d986ae9762c66dc015aa420a Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Sat, 20 Jan 2018 22:13:39 +0800 Subject: [PATCH 013/648] Started working on the parser chapter --- src/the-parser.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/src/the-parser.md b/src/the-parser.md index ab756895a..a43ae3398 100644 --- a/src/the-parser.md +++ b/src/the-parser.md @@ -1 +1,14 @@ -# The parser +# The Parser + +The parser is responsible for converting raw Rust source code into a structured +form which is easier for the compiler to work with, usually called an *Abstract +Syntax Tree*. The bulk of the parser lives in the [libsyntax] crate. + +The parsing process is made up of roughly 3 stages, + +- lexical analysis - turn a stream of characters into a stream of token trees +- macro expansion - run `proc-macros` and expand `macro_rules` macros +- parsing - turn the token trees into an AST + + +[libsyntax]: https://github.com/rust-lang/rust/tree/master/src/libsyntax From a86602e8f2cbc54946d19f99c9332bc85b068dea Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 23 Jan 2018 10:46:33 -0500 Subject: [PATCH 014/648] add an encrypted auth token --- .travis.yml | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/.travis.yml b/.travis.yml index 36c3719f0..4a2381fa7 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,19 +1,17 @@ language: rust - cache: - pip - cargo - install: - - source ~/.cargo/env || true - - bash ci/install.sh - +- source ~/.cargo/env || true +- bash ci/install.sh script: - - true - +- true after_success: - - bash ci/github_pages.sh - +- bash ci/github_pages.sh notifications: email: - on_success: never \ No newline at end of file + on_success: never +env: + global: + secure: EzIPKgJnOk8W0CPK2r6xQv02mV0L2NZJsoWlf3KEHs/ZSskzQEcN18V05/N6H6D4RazRu/UqtWqATV96vZHlmevWkU4PVYz/0UvHfPnu/TbEKBxY/BdJdLIuaNwXVIQ8EnRqVFcRUo/M9xlKk/8mlwVzhPRot8oJQmWDrOqZQgoABFFiU1/OktBz3BtTneR66hqs0rC1tNJyzNh+y2b0qn9Su+N3bp5AXDx5FvDyAKWUfPwP1sRh3xM7oF+0XCnjKKeMYENx7ZltKVQeYSCIHZjBnDIfKX2V1bwKHHrUMrQosJf4lmKsrvdP3sEYUQIIXtY80lA+18yDBhQCKIjh4JyuEisYDMYeYJc2WhUjYaS14WuqnyzRgMMf1EAIwBZgmcsLE2S31csHXXgTWSwwEPzC07erqEsc3nldSQWUbvV9EkFQRXNqMkvyXNt8BQDgSifVw+v0AUBIAr462YlbAN+dk/Ldnd++JPQ6kdrOmegE5B/Ukv+LwFzr9oog2V66TL2STMk8o5ffYWrFGijUfSbMLXIoPRxNj+jFMfAW346tyNCCVKqCexNZV4yUID0f+/zCzcL+DAK1c2hlO4vMQ2fIu2McNKk8RMrMwVsYZhr5jOIvRAkAuaOKZIY+RAsE1VF+G4iGY+UtT89hergHWAdDeFZB+z8OHr1uvaaTQC0= From 33c2b4c5f4621dbce5993b503df21ab67558ffa4 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 23 Jan 2018 10:50:15 -0500 Subject: [PATCH 015/648] make `github_pages.sh` more chatty --- ci/github_pages.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/ci/github_pages.sh b/ci/github_pages.sh index 603b280f9..f5dd7898b 100644 --- a/ci/github_pages.sh +++ b/ci/github_pages.sh @@ -7,4 +7,6 @@ BOOK_DIR=book if [ "$TRAVIS_BRANCH" = master -a "$TRAVIS_PULL_REQUEST" = false ]; then mdbook build ghp-import $BOOK_DIR -fi \ No newline at end of file +else + echo Skipping 'mdbook build' because this is not master or this is just a PR. +fi From a396e910dd694c47b5f0ea44f946c6136e69d9bb Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 23 Jan 2018 10:55:16 -0500 Subject: [PATCH 016/648] make install.sh tell us what it is installing and why --- ci/install.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/ci/install.sh b/ci/install.sh index fac6488b6..6cef274f8 100644 --- a/ci/install.sh +++ b/ci/install.sh @@ -2,9 +2,15 @@ set -ex if command -v mdbook >/dev/null 2>&1; then + echo "installing mdbook" cargo install mdbook --vers "0.0.28" +else + echo "mdbook already installed at $(which mdbook)" fi if command -v ghp-import >/dev/null 2>&1; then + echo "installing ghp-import" pip install ghp-import +else + echo "ghp-import already installed at $(which ghp-import)" fi From cce6a2808165fd28ed55cfaa6f73003d3b4fabe5 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 23 Jan 2018 10:58:54 -0500 Subject: [PATCH 017/648] invert the sense of `command -v` --- ci/install.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/ci/install.sh b/ci/install.sh index 6cef274f8..49a251d44 100644 --- a/ci/install.sh +++ b/ci/install.sh @@ -2,15 +2,15 @@ set -ex if command -v mdbook >/dev/null 2>&1; then + echo "mdbook already installed at $(command -v mdbook)" +else echo "installing mdbook" cargo install mdbook --vers "0.0.28" -else - echo "mdbook already installed at $(which mdbook)" fi if command -v ghp-import >/dev/null 2>&1; then + echo "ghp-import already installed at $(which ghp-import)" +else echo "installing ghp-import" pip install ghp-import -else - echo "ghp-import already installed at $(which ghp-import)" fi From 11bb542a9c205ec00f9e86bab352b5bb803130be Mon Sep 17 00:00:00 2001 From: Phil Ellison Date: Tue, 23 Jan 2018 20:04:27 +0000 Subject: [PATCH 018/648] Copy contents of README.md from librustc/hir --- src/hir-lowering.md | 118 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+) diff --git a/src/hir-lowering.md b/src/hir-lowering.md index b2b6a7cdf..e28bb4cd4 100644 --- a/src/hir-lowering.md +++ b/src/hir-lowering.md @@ -1 +1,119 @@ # HIR lowering + +The HIR -- "High-level IR" -- is the primary IR used in most of +rustc. It is a desugared version of the "abstract syntax tree" (AST) +that is generated after parsing, macro expansion, and name resolution +have completed. Many parts of HIR resemble Rust surface syntax quite +closely, with the exception that some of Rust's expression forms have +been desugared away (as an example, `for` loops are converted into a +`loop` and do not appear in the HIR). + +This chapter covers the main concepts of the HIR. + +### Out-of-band storage and the `Crate` type + +The top-level data-structure in the HIR is the `Crate`, which stores +the contents of the crate currently being compiled (we only ever +construct HIR for the current crate). Whereas in the AST the crate +data structure basically just contains the root module, the HIR +`Crate` structure contains a number of maps and other things that +serve to organize the content of the crate for easier access. + +For example, the contents of individual items (e.g., modules, +functions, traits, impls, etc) in the HIR are not immediately +accessible in the parents. So, for example, if had a module item `foo` +containing a function `bar()`: + +``` +mod foo { + fn bar() { } +} +``` + +Then in the HIR the representation of module `foo` (the `Mod` +stuct) would have only the **`ItemId`** `I` of `bar()`. To get the +details of the function `bar()`, we would lookup `I` in the +`items` map. + +One nice result from this representation is that one can iterate +over all items in the crate by iterating over the key-value pairs +in these maps (without the need to trawl through the IR in total). +There are similar maps for things like trait items and impl items, +as well as "bodies" (explained below). + +The other reason to setup the representation this way is for better +integration with incremental compilation. This way, if you gain access +to a `&hir::Item` (e.g. for the mod `foo`), you do not immediately +gain access to the contents of the function `bar()`. Instead, you only +gain access to the **id** for `bar()`, and you must invoke some +function to lookup the contents of `bar()` given its id; this gives us +a chance to observe that you accessed the data for `bar()` and record +the dependency. + +### Identifiers in the HIR + +Most of the code that has to deal with things in HIR tends not to +carry around references into the HIR, but rather to carry around +*identifier numbers* (or just "ids"). Right now, you will find four +sorts of identifiers in active use: + +- `DefId`, which primarily names "definitions" or top-level items. + - You can think of a `DefId` as being shorthand for a very explicit + and complete path, like `std::collections::HashMap`. However, + these paths are able to name things that are not nameable in + normal Rust (e.g., impls), and they also include extra information + about the crate (such as its version number, as two versions of + the same crate can co-exist). + - A `DefId` really consists of two parts, a `CrateNum` (which + identifies the crate) and a `DefIndex` (which indixes into a list + of items that is maintained per crate). +- `HirId`, which combines the index of a particular item with an + offset within that item. + - the key point of a `HirId` is that it is *relative* to some item (which is named + via a `DefId`). +- `BodyId`, this is an absolute identifier that refers to a specific + body (definition of a function or constant) in the crate. It is currently + effectively a "newtype'd" `NodeId`. +- `NodeId`, which is an absolute id that identifies a single node in the HIR tree. + - While these are still in common use, **they are being slowly phased out**. + - Since they are absolute within the crate, adding a new node + anywhere in the tree causes the node-ids of all subsequent code in + the crate to change. This is terrible for incremental compilation, + as you can perhaps imagine. + +### HIR Map + +Most of the time when you are working with the HIR, you will do so via +the **HIR Map**, accessible in the tcx via `tcx.hir` (and defined in +the `hir::map` module). The HIR map contains a number of methods to +convert between ids of various kinds and to lookup data associated +with a HIR node. + +For example, if you have a `DefId`, and you would like to convert it +to a `NodeId`, you can use `tcx.hir.as_local_node_id(def_id)`. This +returns an `Option` -- this will be `None` if the def-id +refers to something outside of the current crate (since then it has no +HIR node), but otherwise returns `Some(n)` where `n` is the node-id of +the definition. + +Similarly, you can use `tcx.hir.find(n)` to lookup the node for a +`NodeId`. This returns a `Option>`, where `Node` is an enum +defined in the map; by matching on this you can find out what sort of +node the node-id referred to and also get a pointer to the data +itself. Often, you know what sort of node `n` is -- e.g., if you know +that `n` must be some HIR expression, you can do +`tcx.hir.expect_expr(n)`, which will extract and return the +`&hir::Expr`, panicking if `n` is not in fact an expression. + +Finally, you can use the HIR map to find the parents of nodes, via +calls like `tcx.hir.get_parent_node(n)`. + +### HIR Bodies + +A **body** represents some kind of executable code, such as the body +of a function/closure or the definition of a constant. Bodies are +associated with an **owner**, which is typically some kind of item +(e.g., a `fn()` or `const`), but could also be a closure expression +(e.g., `|x, y| x + y`). You can use the HIR map to find the body +associated with a given def-id (`maybe_body_owned_by()`) or to find +the owner of a body (`body_owner_def_id()`). \ No newline at end of file From 952ab0e050146b25e2c0f109a5e8330074ec6cf4 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 23 Jan 2018 16:36:56 -0500 Subject: [PATCH 019/648] try pip install --user --- ci/install.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install.sh b/ci/install.sh index 49a251d44..2d32caaa9 100644 --- a/ci/install.sh +++ b/ci/install.sh @@ -12,5 +12,5 @@ if command -v ghp-import >/dev/null 2>&1; then echo "ghp-import already installed at $(which ghp-import)" else echo "installing ghp-import" - pip install ghp-import + pip install --user ghp-import fi From 3a583ec819d9fcfd13106433ecc4daf71cd381f2 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 23 Jan 2018 17:12:26 -0500 Subject: [PATCH 020/648] add `-p` flag to `ghp-import` so that it pushes --- ci/github_pages.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/github_pages.sh b/ci/github_pages.sh index f5dd7898b..b0dc26902 100644 --- a/ci/github_pages.sh +++ b/ci/github_pages.sh @@ -6,7 +6,7 @@ BOOK_DIR=book # Only upload the built book to github pages if it's a commit to master if [ "$TRAVIS_BRANCH" = master -a "$TRAVIS_PULL_REQUEST" = false ]; then mdbook build - ghp-import $BOOK_DIR + ghp-import -p $BOOK_DIR else echo Skipping 'mdbook build' because this is not master or this is just a PR. fi From 1e6fdb45a247910c04ed9d574a4f4dde5b932b2d Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 23 Jan 2018 21:45:25 -0500 Subject: [PATCH 021/648] try to use travis's builtin deploy workflow --- .travis.yml | 7 +++++++ ci/github_pages.sh | 1 - 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 4a2381fa7..7ecd57fd0 100644 --- a/.travis.yml +++ b/.travis.yml @@ -15,3 +15,10 @@ notifications: env: global: secure: EzIPKgJnOk8W0CPK2r6xQv02mV0L2NZJsoWlf3KEHs/ZSskzQEcN18V05/N6H6D4RazRu/UqtWqATV96vZHlmevWkU4PVYz/0UvHfPnu/TbEKBxY/BdJdLIuaNwXVIQ8EnRqVFcRUo/M9xlKk/8mlwVzhPRot8oJQmWDrOqZQgoABFFiU1/OktBz3BtTneR66hqs0rC1tNJyzNh+y2b0qn9Su+N3bp5AXDx5FvDyAKWUfPwP1sRh3xM7oF+0XCnjKKeMYENx7ZltKVQeYSCIHZjBnDIfKX2V1bwKHHrUMrQosJf4lmKsrvdP3sEYUQIIXtY80lA+18yDBhQCKIjh4JyuEisYDMYeYJc2WhUjYaS14WuqnyzRgMMf1EAIwBZgmcsLE2S31csHXXgTWSwwEPzC07erqEsc3nldSQWUbvV9EkFQRXNqMkvyXNt8BQDgSifVw+v0AUBIAr462YlbAN+dk/Ldnd++JPQ6kdrOmegE5B/Ukv+LwFzr9oog2V66TL2STMk8o5ffYWrFGijUfSbMLXIoPRxNj+jFMfAW346tyNCCVKqCexNZV4yUID0f+/zCzcL+DAK1c2hlO4vMQ2fIu2McNKk8RMrMwVsYZhr5jOIvRAkAuaOKZIY+RAsE1VF+G4iGY+UtT89hergHWAdDeFZB+z8OHr1uvaaTQC0= +deploy: + provider: pages + skip-cleanup: true + github-token: $GITHUB_TOKEN + local-dir: book + on: + branch: master \ No newline at end of file diff --git a/ci/github_pages.sh b/ci/github_pages.sh index b0dc26902..ffd89ad52 100644 --- a/ci/github_pages.sh +++ b/ci/github_pages.sh @@ -6,7 +6,6 @@ BOOK_DIR=book # Only upload the built book to github pages if it's a commit to master if [ "$TRAVIS_BRANCH" = master -a "$TRAVIS_PULL_REQUEST" = false ]; then mdbook build - ghp-import -p $BOOK_DIR else echo Skipping 'mdbook build' because this is not master or this is just a PR. fi From 67da39e4b354cc5f2f367d730e90df6efa502aca Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 24 Jan 2018 04:17:48 -0500 Subject: [PATCH 022/648] try adjusting secure password again --- .travis.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 7ecd57fd0..2d107794f 100644 --- a/.travis.yml +++ b/.travis.yml @@ -14,11 +14,11 @@ notifications: on_success: never env: global: - secure: EzIPKgJnOk8W0CPK2r6xQv02mV0L2NZJsoWlf3KEHs/ZSskzQEcN18V05/N6H6D4RazRu/UqtWqATV96vZHlmevWkU4PVYz/0UvHfPnu/TbEKBxY/BdJdLIuaNwXVIQ8EnRqVFcRUo/M9xlKk/8mlwVzhPRot8oJQmWDrOqZQgoABFFiU1/OktBz3BtTneR66hqs0rC1tNJyzNh+y2b0qn9Su+N3bp5AXDx5FvDyAKWUfPwP1sRh3xM7oF+0XCnjKKeMYENx7ZltKVQeYSCIHZjBnDIfKX2V1bwKHHrUMrQosJf4lmKsrvdP3sEYUQIIXtY80lA+18yDBhQCKIjh4JyuEisYDMYeYJc2WhUjYaS14WuqnyzRgMMf1EAIwBZgmcsLE2S31csHXXgTWSwwEPzC07erqEsc3nldSQWUbvV9EkFQRXNqMkvyXNt8BQDgSifVw+v0AUBIAr462YlbAN+dk/Ldnd++JPQ6kdrOmegE5B/Ukv+LwFzr9oog2V66TL2STMk8o5ffYWrFGijUfSbMLXIoPRxNj+jFMfAW346tyNCCVKqCexNZV4yUID0f+/zCzcL+DAK1c2hlO4vMQ2fIu2McNKk8RMrMwVsYZhr5jOIvRAkAuaOKZIY+RAsE1VF+G4iGY+UtT89hergHWAdDeFZB+z8OHr1uvaaTQC0= + secure: YQX/AWq5KsvAFYqcCK6c1DmOZX9EMrecBM5qnc4uE2HvEBS+x0l8xatI2Nv8U9eiasZYfsqmHn0ANvxu6e4oqL15m4cVsdliCzdkrPsDapxTnwwJvMQg+yHZiEd5BPlaDQt/wYvP8QBXgQsXoAJKrfAS+BFsowBFHt/LOFOunbAQrtQZqwqrnI6+xh+2TRMckws/VcTLRqwl3pyEyfacJhbbv1V3gJh7Y17hELsgsP7+7cMXT0bK6dtf7a9vne9Hsm5fw7VeMKBn1/dJ82fyEK6HHjkjdw1/OoY35YVyNZ/9ZxP2u1ClEXzCRJQ2CvKr8Tuoh/AuoL0pwrfhOTaOuWU0QZT4QBqjTimsgBLqiJicMiSndgsXinLWvlDqrMS1XfleqCKqAQy9AJTCR1LnwR90/HRxfE5YDAL/mbc0Su4jj+l5Zv3UE8vUqFE34E/jzip17JkDT5aMkl4bgW65lqJE7SLWl7gXT7eYbPEtQZoucR1hkSsBu/4YTvcxSlD98spWZ68mWwYyjLJSQDES+GefUnHJ/RbBVl9pW+sL7jXJ+kZ/NBCtCIgrkGchudEMDEvS6rcOzwCejxqL1of0jYHGopkBXSVHOPneWIdNeKXwBZA9hp0yKh0sWwrKHrA3wYhS/kF9uO19l/RnSTXAfApYR/yJUbYliuMJYCgNeKE= deploy: provider: pages skip-cleanup: true github-token: $GITHUB_TOKEN local-dir: book on: - branch: master \ No newline at end of file + branch: master From c44964bd148d23a68f560a07c6d88117c564546a Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 24 Jan 2018 09:11:24 -0500 Subject: [PATCH 023/648] how to build and run compiler, first shot --- src/how-to-build-and-run.md | 117 ++++++++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 505836094..de8a93bdd 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -1 +1,118 @@ # How to build the compiler and run what you built + +The compiler is built using a tool called `x.py`. You will need to +have Python installed to run it. But before we get to that, if you're going to +be hacking on rustc, you'll want to tweak the configuration of the compiler. The default +configuration is oriented towards running the compiler as a user, not a developer. + +### Create a config.toml + +To start, copy [`config.toml.example`] to `config.toml`: + +[`config.toml.example`]: https://github.com/rust-lang/rust/blob/master/config.toml.example + +```bash +> cd $RUST_CHECKOUT +> cp config.toml.example config.toml +``` + +Then you will want to open up the file and change the following +settings (and possibly others, such as `llvm.ccache`): + +``` +[llvm] +# Enables LLVM assertions, which will check that the LLVM bitcode generated +# by the compiler is internally consistent. These are particularly helpful +# if you edit `trans`. +assertions = true + +[rust] +# This enables some assertions, but more importantly it enables the `debug!` logging +# macros that are essential for debugging rustc. +debug-assertions = true + +# This will make your build more parallel; it costs a bit of runtime +# performance perhaps (less inlining) but it's worth it. +codegen-units = 0 + +# I always enable full debuginfo, though debuginfo-lines is more important. +debuginfo = true + +# Gives you line numbers for backtraces. +debuginfo-lines = true + +# Using the system allocator (instead of jemalloc) means that tools +# like valgrind and memcache work better. +use-jemalloc = false +``` + +### Running x.py and building a stage1 compiler + +Once you've created a config.toml, you are now ready to run +`x.py`. There are a lot of options here, but let's start with what is +probably the best "go to" command for building a local rust: + +``` +./x.py build --incremental --stage 1 src/libstd +``` + +What this command will do is the following: + +- Using the beta compiler (also called stage 0), it will build the + standard library and rustc from the `src` directory. The resulting + compiler is called the "stage 1" compiler. + - During this build, the `--incremental` switch enables incremental + compilation, so that if you later rebuild after editing things in + `src`, you can save a bit of time. +- Using this stage 1 compiler, it will build the standard library. + (this is what the `src/libstd`) means. + +This is just a subset of the full rustc build. The **full** rustc build (what you +get if you just say `./x.py build`) has quite a few more steps: + +- Build stage1 rustc with stage0 compiler +- Build libstd with stage1 compiler (up to here is the same) +- Build stage2 rustc with stage1 compiler (this part is new) +- Build libstd with stage2 compiler +- Build librustdoc and a bunch of other things + +### Creating a rustup toolchain + +Once you have successfully built rustc, you will have created a bunch +of files in your `build` directory. In order to actually run the +resulting rustc, we recommend creating rustup toolchains. The first +one will run the stage1 compiler (which we built above). The second +will execute the stage2 compiler (which we did not build, but which +you will likely build at some point). + +``` +> rustup toolchain link stage1 build//stage1 +> rustup toolchain link stage2 build//stage2 +``` + +Now you can run the rustc you built with. If you run with `-vV`, you +should see a version number ending in `-dev`, indicating a build from +your local environment: + +``` +> rustc +stage1 -vV +rustc 1.25.0-dev +binary: rustc +commit-hash: unknown +commit-date: unknown +host: x86_64-unknown-linux-gnu +release: 1.25.0-dev +LLVM version: 4.0 +``` + +### Other x.py commands + +Here are a few other useful x.py commands. We'll cover some of them in detail in other sections: + +- Building things: + - `./x.py clean` -- clean up the build directory (`rm -rf build` works too, but then you have to rebuild LLVM) + - `./x.py build --stage 1` -- builds everything using the stage 1 compiler, not just up to libstd + - `./x.py build` -- builds the stage2 compiler +- Running tests (see the section [running tests](./running-tests.html) for more details): + - `./x.py test --stage 1 src/libstd` -- runs the `#[test]` tests from libstd + - `./x.py test --stage 1 src/test/run-pass` -- runs the `run-pass` test suite From b2a850faf0fd2e5b246ea2b7938ea288bfbca2a5 Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Thu, 25 Jan 2018 00:30:52 +0800 Subject: [PATCH 024/648] Mentioned the main players in the parser --- src/the-parser.md | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/src/the-parser.md b/src/the-parser.md index a43ae3398..9a01d8a36 100644 --- a/src/the-parser.md +++ b/src/the-parser.md @@ -1,14 +1,42 @@ # The Parser The parser is responsible for converting raw Rust source code into a structured -form which is easier for the compiler to work with, usually called an *Abstract -Syntax Tree*. The bulk of the parser lives in the [libsyntax] crate. +form which is easier for the compiler to work with, usually called an [*Abstract +Syntax Tree*][ast]. An AST mirrors the structure of a Rust program in memory, +using a `Span` to link a particular AST node back to its source text. -The parsing process is made up of roughly 3 stages, +The bulk of the parser lives in the [libsyntax] crate. + +Like most parsers, the parsing process is composed of two main steps, - lexical analysis - turn a stream of characters into a stream of token trees -- macro expansion - run `proc-macros` and expand `macro_rules` macros - parsing - turn the token trees into an AST +The `syntax` crate contains several main players, + +- a [`CodeMap`] for mapping AST nodes to their source code +- the [ast module] contains types corresponding to each AST node +- a [`StringReader`] for lexing source code into tokens +- the [parser module] and [`Parser`] struct are in charge of actually parsing + tokens into AST nodes, +- and a [visit module] for walking the AST and inspecting or mutating the AST + nodes. + +The main entrypoint to the parser is via the various `parse_*` functions +in the [parser module]. They let you do things like turn a filemap into a +token stream, create a parser from the token stream, and then execute the +parser to get a `Crate` (the root AST node). + +To minimise the amount of copying that is done, both the `StringReader` and +`Parser` have lifetimes which bind them to the parent `ParseSess`. This contains +all the information needed while parsing, as well as the `CodeMap` itself. [libsyntax]: https://github.com/rust-lang/rust/tree/master/src/libsyntax +[rustc_errors]: https://github.com/rust-lang/rust/tree/master/src/librustc_errors +[ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree +[`CodeMap`]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs +[ast module]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/ast.rs +[parser module]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/parse +[`Parser`]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/parser.rs +[`StringReader`]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/lexer/mod.rs +[visit module]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/visit.rs \ No newline at end of file From 3b142e54c23bf615eda3de5d1cd42289f86bcab0 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Thu, 25 Jan 2018 09:40:07 -0500 Subject: [PATCH 025/648] fix nits --- src/how-to-build-and-run.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index de8a93bdd..f685f569c 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -19,7 +19,7 @@ To start, copy [`config.toml.example`] to `config.toml`: Then you will want to open up the file and change the following settings (and possibly others, such as `llvm.ccache`): -``` +```toml [llvm] # Enables LLVM assertions, which will check that the LLVM bitcode generated # by the compiler is internally consistent. These are particularly helpful @@ -53,7 +53,7 @@ Once you've created a config.toml, you are now ready to run probably the best "go to" command for building a local rust: ``` -./x.py build --incremental --stage 1 src/libstd +./x.py build -i --stage 1 src/libstd ``` What this command will do is the following: @@ -61,7 +61,7 @@ What this command will do is the following: - Using the beta compiler (also called stage 0), it will build the standard library and rustc from the `src` directory. The resulting compiler is called the "stage 1" compiler. - - During this build, the `--incremental` switch enables incremental + - During this build, the `-i` (or `--incremental`) switch enables incremental compilation, so that if you later rebuild after editing things in `src`, you can save a bit of time. - Using this stage 1 compiler, it will build the standard library. @@ -72,7 +72,8 @@ get if you just say `./x.py build`) has quite a few more steps: - Build stage1 rustc with stage0 compiler - Build libstd with stage1 compiler (up to here is the same) -- Build stage2 rustc with stage1 compiler (this part is new) +- Build rustc from `src` again, this time with the stage1 compiler (this part is new) + - The resulting compiler here is called the "stage2" compiler - Build libstd with stage2 compiler - Build librustdoc and a bunch of other things @@ -83,7 +84,8 @@ of files in your `build` directory. In order to actually run the resulting rustc, we recommend creating rustup toolchains. The first one will run the stage1 compiler (which we built above). The second will execute the stage2 compiler (which we did not build, but which -you will likely build at some point). +you will likely need to build at some point; for example, if you want +to run the entire test suite). ``` > rustup toolchain link stage1 build//stage1 From 1627505cab834ed269b8b28a5772731b21067a91 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Thu, 25 Jan 2018 15:56:06 -0600 Subject: [PATCH 026/648] Start macro expansion chapter --- src/macro-expansion.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/src/macro-expansion.md b/src/macro-expansion.md index 12b95cb6c..7ab1c35bc 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -1 +1,12 @@ # Macro expansion + +Macro expansion happens during parsing. `rustc` has two parsers, in fact: the +normal Rust parser, and the macro parser. During the parsing phase, the normal +Rust parser will call into the macro parser when it encounters a macro. The +macro parser, in turn, may call back out to the Rust parser when it needs to +bind a metavariable (e.g. `$expr`). There are a few aspects of this system to be +explained. The code for macro expansion is in `src/libsyntax/ext/tt/`. + +TODO: explain parsing of macro definitions + +TODO: explain parsing of macro invokations + macro expansion From 4992b476289894fcd4de62a957bf52ea35af5491 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Thu, 25 Jan 2018 17:56:33 -0600 Subject: [PATCH 027/648] Add a bit about macro expansion --- src/src/chap-060-macro-expansion.md | 66 +++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 src/src/chap-060-macro-expansion.md diff --git a/src/src/chap-060-macro-expansion.md b/src/src/chap-060-macro-expansion.md new file mode 100644 index 000000000..77f764d6a --- /dev/null +++ b/src/src/chap-060-macro-expansion.md @@ -0,0 +1,66 @@ +# Macro expansion + +Macro expansion happens during parsing. `rustc` has two parsers, in fact: the +normal Rust parser, and the macro parser. During the parsing phase, the normal +Rust parser will call into the macro parser when it encounters a macro. The +macro parser, in turn, may call back out to the Rust parser when it needs to +bind a metavariable (e.g. `$my_expr`). There are a few aspects of this system to +be explained. The code for macro expansion is in `src/libsyntax/ext/tt/`. + +### The macro parser + +Basically, the macro parser is like an NFA-based regex parser. It uses an +algorithm similar in spirit to the [Earley parsing +algorithm](https://en.wikipedia.org/wiki/Earley_parser). The macro parser is +defined in `src/libsyntax/ext/tt/macro_parser.rs`. + +In a traditional NFA-based parser, one common approach is to have some pattern +which we are trying to match an input against. Moreover, we may try to capture +some portion of the input and bind it to variable in the pattern. For example: +suppose we have a pattern (borrowing Rust macro syntax) such as `a $b:ident a` +-- that is, an `a` token followed by an `ident` token followed by another `a` +token. Given an input `a foo a`, the _metavariable_ `$b` would bind to the +`ident` `foo`. On the other hand, an input `a foo b` would be rejected as a +parse failure because the pattern `a a` cannot match `a foo b` (or as +the compiler would put it, "no rules expected token `b`"). + +The macro parser does pretty much exactly that with one exception: in order to +parse different types of metavariables, such as `ident`, `block`, `expr`, etc., +the macro parser must sometimes call back to the normal Rust parser. + +Interestingly, both definitions and invokations of macros are parsed using the +macro parser. This is extremely non-intuitive and self-referential. The code to +parse macro _definitions_ is in `src/libsyntax/ext/tt/macro_rules.rs`. It +defines the pattern for matching for a macro definition as `$( $lhs:tt => +$rhs:tt );+`. In other words, a `macro_rules` defintion should have in its body +at least one occurence of a token tree followed by `=>` followed by another +token tree. When the compiler comes to a `macro_rules` definition, it uses this +pattern to match the two token trees per rule in the definition of the macro +_using the macro parser itself_. + +When the compiler comes to a macro invokation, it needs to parse that +invokation. This is also known as _macro expansion_. The same NFA-based macro +parser is used that is described above. Notably, the "pattern" (or _matcher_) +used is the first token tree extracted from the rules of the macro _definition_. +In other words, given some pattern described by the _definition_ of the macro, +we want to match the contents of the _invokation_ of the macro. + +The algorithm is exactly the same, but when the macro parser comes to a place in +the current matcher where it needs to match a _non-terminal_ (i.e. a +metavariable), it calls back to the normal Rust parser to get the contents of +that non-terminal. Then, the macro parser proceeds in parsing as normal. + +For more information about the macro parser's implementation, see the comments +in `src/libsyntax/ext/tt/macro_parser.rs`. + +### Hygiene + +TODO + +### Procedural Macros + +TODO + +### Custom Derive + +TODO From ba3dd183e615682ab515c57de91cf3d32e891fe1 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Thu, 25 Jan 2018 17:58:04 -0600 Subject: [PATCH 028/648] Oops rename --- src/macro-expansion.md | 62 +++++++++++++++++++++++++-- src/src/chap-060-macro-expansion.md | 66 ----------------------------- 2 files changed, 58 insertions(+), 70 deletions(-) delete mode 100644 src/src/chap-060-macro-expansion.md diff --git a/src/macro-expansion.md b/src/macro-expansion.md index 7ab1c35bc..77f764d6a 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -4,9 +4,63 @@ Macro expansion happens during parsing. `rustc` has two parsers, in fact: the normal Rust parser, and the macro parser. During the parsing phase, the normal Rust parser will call into the macro parser when it encounters a macro. The macro parser, in turn, may call back out to the Rust parser when it needs to -bind a metavariable (e.g. `$expr`). There are a few aspects of this system to be -explained. The code for macro expansion is in `src/libsyntax/ext/tt/`. +bind a metavariable (e.g. `$my_expr`). There are a few aspects of this system to +be explained. The code for macro expansion is in `src/libsyntax/ext/tt/`. -TODO: explain parsing of macro definitions +### The macro parser -TODO: explain parsing of macro invokations + macro expansion +Basically, the macro parser is like an NFA-based regex parser. It uses an +algorithm similar in spirit to the [Earley parsing +algorithm](https://en.wikipedia.org/wiki/Earley_parser). The macro parser is +defined in `src/libsyntax/ext/tt/macro_parser.rs`. + +In a traditional NFA-based parser, one common approach is to have some pattern +which we are trying to match an input against. Moreover, we may try to capture +some portion of the input and bind it to variable in the pattern. For example: +suppose we have a pattern (borrowing Rust macro syntax) such as `a $b:ident a` +-- that is, an `a` token followed by an `ident` token followed by another `a` +token. Given an input `a foo a`, the _metavariable_ `$b` would bind to the +`ident` `foo`. On the other hand, an input `a foo b` would be rejected as a +parse failure because the pattern `a a` cannot match `a foo b` (or as +the compiler would put it, "no rules expected token `b`"). + +The macro parser does pretty much exactly that with one exception: in order to +parse different types of metavariables, such as `ident`, `block`, `expr`, etc., +the macro parser must sometimes call back to the normal Rust parser. + +Interestingly, both definitions and invokations of macros are parsed using the +macro parser. This is extremely non-intuitive and self-referential. The code to +parse macro _definitions_ is in `src/libsyntax/ext/tt/macro_rules.rs`. It +defines the pattern for matching for a macro definition as `$( $lhs:tt => +$rhs:tt );+`. In other words, a `macro_rules` defintion should have in its body +at least one occurence of a token tree followed by `=>` followed by another +token tree. When the compiler comes to a `macro_rules` definition, it uses this +pattern to match the two token trees per rule in the definition of the macro +_using the macro parser itself_. + +When the compiler comes to a macro invokation, it needs to parse that +invokation. This is also known as _macro expansion_. The same NFA-based macro +parser is used that is described above. Notably, the "pattern" (or _matcher_) +used is the first token tree extracted from the rules of the macro _definition_. +In other words, given some pattern described by the _definition_ of the macro, +we want to match the contents of the _invokation_ of the macro. + +The algorithm is exactly the same, but when the macro parser comes to a place in +the current matcher where it needs to match a _non-terminal_ (i.e. a +metavariable), it calls back to the normal Rust parser to get the contents of +that non-terminal. Then, the macro parser proceeds in parsing as normal. + +For more information about the macro parser's implementation, see the comments +in `src/libsyntax/ext/tt/macro_parser.rs`. + +### Hygiene + +TODO + +### Procedural Macros + +TODO + +### Custom Derive + +TODO diff --git a/src/src/chap-060-macro-expansion.md b/src/src/chap-060-macro-expansion.md deleted file mode 100644 index 77f764d6a..000000000 --- a/src/src/chap-060-macro-expansion.md +++ /dev/null @@ -1,66 +0,0 @@ -# Macro expansion - -Macro expansion happens during parsing. `rustc` has two parsers, in fact: the -normal Rust parser, and the macro parser. During the parsing phase, the normal -Rust parser will call into the macro parser when it encounters a macro. The -macro parser, in turn, may call back out to the Rust parser when it needs to -bind a metavariable (e.g. `$my_expr`). There are a few aspects of this system to -be explained. The code for macro expansion is in `src/libsyntax/ext/tt/`. - -### The macro parser - -Basically, the macro parser is like an NFA-based regex parser. It uses an -algorithm similar in spirit to the [Earley parsing -algorithm](https://en.wikipedia.org/wiki/Earley_parser). The macro parser is -defined in `src/libsyntax/ext/tt/macro_parser.rs`. - -In a traditional NFA-based parser, one common approach is to have some pattern -which we are trying to match an input against. Moreover, we may try to capture -some portion of the input and bind it to variable in the pattern. For example: -suppose we have a pattern (borrowing Rust macro syntax) such as `a $b:ident a` --- that is, an `a` token followed by an `ident` token followed by another `a` -token. Given an input `a foo a`, the _metavariable_ `$b` would bind to the -`ident` `foo`. On the other hand, an input `a foo b` would be rejected as a -parse failure because the pattern `a a` cannot match `a foo b` (or as -the compiler would put it, "no rules expected token `b`"). - -The macro parser does pretty much exactly that with one exception: in order to -parse different types of metavariables, such as `ident`, `block`, `expr`, etc., -the macro parser must sometimes call back to the normal Rust parser. - -Interestingly, both definitions and invokations of macros are parsed using the -macro parser. This is extremely non-intuitive and self-referential. The code to -parse macro _definitions_ is in `src/libsyntax/ext/tt/macro_rules.rs`. It -defines the pattern for matching for a macro definition as `$( $lhs:tt => -$rhs:tt );+`. In other words, a `macro_rules` defintion should have in its body -at least one occurence of a token tree followed by `=>` followed by another -token tree. When the compiler comes to a `macro_rules` definition, it uses this -pattern to match the two token trees per rule in the definition of the macro -_using the macro parser itself_. - -When the compiler comes to a macro invokation, it needs to parse that -invokation. This is also known as _macro expansion_. The same NFA-based macro -parser is used that is described above. Notably, the "pattern" (or _matcher_) -used is the first token tree extracted from the rules of the macro _definition_. -In other words, given some pattern described by the _definition_ of the macro, -we want to match the contents of the _invokation_ of the macro. - -The algorithm is exactly the same, but when the macro parser comes to a place in -the current matcher where it needs to match a _non-terminal_ (i.e. a -metavariable), it calls back to the normal Rust parser to get the contents of -that non-terminal. Then, the macro parser proceeds in parsing as normal. - -For more information about the macro parser's implementation, see the comments -in `src/libsyntax/ext/tt/macro_parser.rs`. - -### Hygiene - -TODO - -### Procedural Macros - -TODO - -### Custom Derive - -TODO From 458685bbb802c2df00a8422f756c64059885f875 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 26 Jan 2018 09:11:52 -0500 Subject: [PATCH 029/648] rename the hir chapter to The HIR --- src/SUMMARY.md | 2 +- src/{hir-lowering.md => hir.md} | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) rename src/{hir-lowering.md => hir.md} (98%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index f8b1f1d6e..2e3b34386 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -8,7 +8,7 @@ - [The parser](./the-parser.md) - [Macro expansion](./macro-expansion.md) - [Name resolution](./name-resolution.md) -- [HIR lowering](./hir-lowering.md) +- [The HIR (High-level IR)](./hir.md) - [The `ty` module: representing types](./ty.md) - [Type inference](./type-inference.md) - [Trait resolution](./trait-resolution.md) diff --git a/src/hir-lowering.md b/src/hir.md similarity index 98% rename from src/hir-lowering.md rename to src/hir.md index e28bb4cd4..5d5e273c4 100644 --- a/src/hir-lowering.md +++ b/src/hir.md @@ -1,4 +1,4 @@ -# HIR lowering +# The HIR The HIR -- "High-level IR" -- is the primary IR used in most of rustc. It is a desugared version of the "abstract syntax tree" (AST) @@ -116,4 +116,4 @@ associated with an **owner**, which is typically some kind of item (e.g., a `fn()` or `const`), but could also be a closure expression (e.g., `|x, y| x + y`). You can use the HIR map to find the body associated with a given def-id (`maybe_body_owned_by()`) or to find -the owner of a body (`body_owner_def_id()`). \ No newline at end of file +the owner of a body (`body_owner_def_id()`). From 40daff36d4d8ac58b1ff7842abff7ece1477be8e Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 26 Jan 2018 09:20:01 -0500 Subject: [PATCH 030/648] move over the query chapter from src/librustc/ty/maps --- src/SUMMARY.md | 2 + src/query.md | 314 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 316 insertions(+) create mode 100644 src/query.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 2e3b34386..e4bc24283 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -5,6 +5,8 @@ - [Using the compiler testing framework](./running-tests.md) - [Walkthrough: a typical contribution](./walkthrough.md) - [High-level overview of the compiler source](./high-level-overview.md) +- [Queries: demand-driven compilation](./query.md) + - [Incremental compilation](./incremental-compilation.md) - [The parser](./the-parser.md) - [Macro expansion](./macro-expansion.md) - [Name resolution](./name-resolution.md) diff --git a/src/query.md b/src/query.md new file mode 100644 index 000000000..65d651307 --- /dev/null +++ b/src/query.md @@ -0,0 +1,314 @@ +# Queries: demand-driven compilation + +As described in [the high-level overview of the compiler][hl], the +Rust compiler is current transitioning from a traditional "pass-based" +setup to a "demand-driven" system. **The Compiler Query System is the +key to our new demand-driven organization.** The idea is pretty +simple. You have various queries that compute things about the input +-- for example, there is a query called `type_of(def_id)` that, given +the def-id of some item, will compute the type of that item and return +it to you. + +[hl]: high-level-overview.html + +Query execution is **memoized** -- so the first time you invoke a +query, it will go do the computation, but the next time, the result is +returned from a hashtable. Moreover, query execution fits nicely into +**incremental computation**; the idea is roughly that, when you do a +query, the result **may** be returned to you by loading stored data +from disk (but that's a separate topic we won't discuss further here). + +The overall vision is that, eventually, the entire compiler +control-flow will be query driven. There will effectively be one +top-level query ("compile") that will run compilation on a crate; this +will in turn demand information about that crate, starting from the +*end*. For example: + +- This "compile" query might demand to get a list of codegen-units + (i.e., modules that need to be compiled by LLVM). +- But computing the list of codegen-units would invoke some subquery + that returns the list of all modules defined in the Rust source. +- That query in turn would invoke something asking for the HIR. +- This keeps going further and further back until we wind up doing the + actual parsing. + +However, that vision is not fully realized. Still, big chunks of the +compiler (for example, generating MIR) work exactly like this. + +### Invoking queries + +To invoke a query is simple. The tcx ("type context") offers a method +for each defined query. So, for example, to invoke the `type_of` +query, you would just do this: + +```rust +let ty = tcx.type_of(some_def_id); +``` + +### Cycles between queries + +Currently, cycles during query execution should always result in a +compilation error. Typically, they arise because of illegal programs +that contain cyclic references they shouldn't (though sometimes they +arise because of compiler bugs, in which case we need to factor our +queries in a more fine-grained fashion to avoid them). + +However, it is nonetheless often useful to *recover* from a cycle +(after reporting an error, say) and try to soldier on, so as to give a +better user experience. In order to recover from a cycle, you don't +get to use the nice method-call-style syntax. Instead, you invoke +using the `try_get` method, which looks roughly like this: + +```rust +use ty::maps::queries; +... +match queries::type_of::try_get(tcx, DUMMY_SP, self.did) { + Ok(result) => { + // no cycle occurred! You can use `result` + } + Err(err) => { + // A cycle occurred! The error value `err` is a `DiagnosticBuilder`, + // meaning essentially an "in-progress", not-yet-reported error message. + // See below for more details on what to do here. + } +} +``` + +So, if you get back an `Err` from `try_get`, then a cycle *did* occur. This means that +you must ensure that a compiler error message is reported. You can do that in two ways: + +The simplest is to invoke `err.emit()`. This will emit the cycle error to the user. + +However, often cycles happen because of an illegal program, and you +know at that point that an error either already has been reported or +will be reported due to this cycle by some other bit of code. In that +case, you can invoke `err.cancel()` to not emit any error. It is +traditional to then invoke: + +``` +tcx.sess.delay_span_bug(some_span, "some message") +``` + +`delay_span_bug()` is a helper that says: we expect a compilation +error to have happened or to happen in the future; so, if compilation +ultimately succeeds, make an ICE with the message `"some +message"`. This is basically just a precaution in case you are wrong. + +### How the compiler executes a query + +So you may be wondering what happens when you invoke a query +method. The answer is that, for each query, the compiler maintains a +cache -- if your query has already been executed, then, the answer is +simple: we clone the return value out of the cache and return it +(therefore, you should try to ensure that the return types of queries +are cheaply cloneable; insert a `Rc` if necessary). + +#### Providers + +If, however, the query is *not* in the cache, then the compiler will +try to find a suitable **provider**. A provider is a function that has +been defined and linked into the compiler somewhere that contains the +code to compute the result of the query. + +**Providers are defined per-crate.** The compiler maintains, +internally, a table of providers for every crate, at least +conceptually. Right now, there are really two sets: the providers for +queries about the **local crate** (that is, the one being compiled) +and providers for queries about **external crates** (that is, +dependencies of the local crate). Note that what determines the crate +that a query is targeting is not the *kind* of query, but the *key*. +For example, when you invoke `tcx.type_of(def_id)`, that could be a +local query or an external query, depending on what crate the `def_id` +is referring to (see the `self::keys::Key` trait for more information +on how that works). + +Providers always have the same signature: + +```rust +fn provider<'cx, 'tcx>(tcx: TyCtxt<'cx, 'tcx, 'tcx>, + key: QUERY_KEY) + -> QUERY_RESULT +{ + ... +} +``` + +Providers take two arguments: the `tcx` and the query key. Note also +that they take the *global* tcx (i.e., they use the `'tcx` lifetime +twice), rather than taking a tcx with some active inference context. +They return the result of the query. + +#### How providers are setup + +When the tcx is created, it is given the providers by its creator using +the `Providers` struct. This struct is generate by the macros here, but it +is basically a big list of function pointers: + +```rust +struct Providers { + type_of: for<'cx, 'tcx> fn(TyCtxt<'cx, 'tcx, 'tcx>, DefId) -> Ty<'tcx>, + ... +} +``` + +At present, we have one copy of the struct for local crates, and one +for external crates, though the plan is that we may eventually have +one per crate. + +These `Provider` structs are ultimately created and populated by +`librustc_driver`, but it does this by distributing the work +throughout the other `rustc_*` crates. This is done by invoking +various `provide` functions. These functions tend to look something +like this: + +```rust +pub fn provide(providers: &mut Providers) { + *providers = Providers { + type_of, + ..*providers + }; +} +``` + +That is, they take an `&mut Providers` and mutate it in place. Usually +we use the formulation above just because it looks nice, but you could +as well do `providers.type_of = type_of`, which would be equivalent. +(Here, `type_of` would be a top-level function, defined as we saw +before.) So, if we want to add a provider for some other query, +let's call it `fubar`, into the crate above, we might modify the `provide()` +function like so: + +```rust +pub fn provide(providers: &mut Providers) { + *providers = Providers { + type_of, + fubar, + ..*providers + }; +} + +fn fubar<'cx, 'tcx>(tcx: TyCtxt<'cx, 'tcx>, key: DefId) -> Fubar<'tcx> { .. } +``` + +NB. Most of the `rustc_*` crates only provide **local +providers**. Almost all **extern providers** wind up going through the +[`rustc_metadata` crate][rustc_metadata], which loads the information from the crate +metadata. But in some cases there are crates that provide queries for +*both* local and external crates, in which case they define both a +`provide` and a `provide_extern` function that `rustc_driver` can +invoke. + +[rustc_metadata]: https://github.com/rust-lang/rust/tree/master/src/librustc_metadata + +### Adding a new kind of query + +So suppose you want to add a new kind of query, how do you do so? +Well, defining a query takes place in two steps: + +1. first, you have to specify the query name and arguments; and then, +2. you have to supply query providers where needed. + +To specify the query name and arguments, you simply add an entry to +the big macro invocation in +[`src/librustc/ty/maps/mod.rs`][maps-mod]. This will probably have +changed by the time you read this README, but at present it looks +something like: + +[maps-mod]: https://github.com/rust-lang/rust/blob/master/src/librustc/ty/maps/mod.rs + +``` +define_maps! { <'tcx> + /// Records the type of every item. + [] fn type_of: TypeOfItem(DefId) -> Ty<'tcx>, + + ... +} +``` + +Each line of the macro defines one query. The name is broken up like this: + +``` +[] fn type_of: TypeOfItem(DefId) -> Ty<'tcx>, +^^ ^^^^^^^ ^^^^^^^^^^ ^^^^^ ^^^^^^^^ +| | | | | +| | | | result type of query +| | | query key type +| | dep-node constructor +| name of query +query flags +``` + +Let's go over them one by one: + +- **Query flags:** these are largely unused right now, but the intention + is that we'll be able to customize various aspects of how the query is + processed. +- **Name of query:** the name of the query method + (`tcx.type_of(..)`). Also used as the name of a struct + (`ty::maps::queries::type_of`) that will be generated to represent + this query. +- **Dep-node constructor:** indicates the constructor function that + connects this query to incremental compilation. Typically, this is a + `DepNode` variant, which can be added by modifying the + `define_dep_nodes!` macro invocation in + [`librustc/dep_graph/dep_node.rs`][dep-node]. + - However, sometimes we use a custom function, in which case the + name will be in snake case and the function will be defined at the + bottom of the file. This is typically used when the query key is + not a def-id, or just not the type that the dep-node expects. +- **Query key type:** the type of the argument to this query. + This type must implement the `ty::maps::keys::Key` trait, which + defines (for example) how to map it to a crate, and so forth. +- **Result type of query:** the type produced by this query. This type + should (a) not use `RefCell` or other interior mutability and (b) be + cheaply cloneable. Interning or using `Rc` or `Arc` is recommended for + non-trivial data types. + - The one exception to those rules is the `ty::steal::Steal` type, + which is used to cheaply modify MIR in place. See the definition + of `Steal` for more details. New uses of `Steal` should **not** be + added without alerting `@rust-lang/compiler`. + +[dep-node]: https://github.com/rust-lang/rust/blob/master/src/librustc/dep_graph/dep_node.rs + +So, to add a query: + +- Add an entry to `define_maps!` using the format above. +- Possibly add a corresponding entry to the dep-node macro. +- Link the provider by modifying the appropriate `provide` method; + or add a new one if needed and ensure that `rustc_driver` is invoking it. + +#### Query structs and descriptions + +For each kind, the `define_maps` macro will generate a "query struct" +named after the query. This struct is a kind of a place-holder +describing the query. Each such struct implements the +`self::config::QueryConfig` trait, which has associated types for the +key/value of that particular query. Basically the code generated looks something +like this: + +```rust +// Dummy struct representing a particular kind of query: +pub struct type_of<'tcx> { phantom: PhantomData<&'tcx ()> } + +impl<'tcx> QueryConfig for type_of<'tcx> { + type Key = DefId; + type Value = Ty<'tcx>; +} +``` + +There is an additional trait that you may wish to implement called +`self::config::QueryDescription`. This trait is used during cycle +errors to give a "human readable" name for the query, so that we can +summarize what was happening when the cycle occurred. Implementing +this trait is optional if the query key is `DefId`, but if you *don't* +implement it, you get a pretty generic error ("processing `foo`..."). +You can put new impls into the `config` module. They look something like this: + +```rust +impl<'tcx> QueryDescription for queries::type_of<'tcx> { + fn describe(tcx: TyCtxt, key: DefId) -> String { + format!("computing the type of `{}`", tcx.item_path_str(key)) + } +} +``` + From 858dfdf054c898b3966c6360a3db05428e01f304 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 26 Jan 2018 14:41:56 -0600 Subject: [PATCH 031/648] Updated macros to address Niko's comments --- src/macro-expansion.md | 172 +++++++++++++++++++++++++++++++---------- 1 file changed, 130 insertions(+), 42 deletions(-) diff --git a/src/macro-expansion.md b/src/macro-expansion.md index 77f764d6a..dd735ed23 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -2,56 +2,136 @@ Macro expansion happens during parsing. `rustc` has two parsers, in fact: the normal Rust parser, and the macro parser. During the parsing phase, the normal -Rust parser will call into the macro parser when it encounters a macro. The -macro parser, in turn, may call back out to the Rust parser when it needs to -bind a metavariable (e.g. `$my_expr`). There are a few aspects of this system to -be explained. The code for macro expansion is in `src/libsyntax/ext/tt/`. +Rust parser will call into the macro parser when it encounters a macro +definition or macro invocation (TODO: verify). The macro parser, in turn, may +call back out to the Rust parser when it needs to bind a metavariable (e.g. +`$my_expr`) while parsing the contents of a macro invocation. The code for macro +expansion is in [`src/libsyntax/ext/tt/`][code_dir]. This chapter aims to +explain how macro expansion works. + +### Example + +It's helpful to have an example to refer to. For the remainder of this chapter, +whenever we refer to the "example _definition_", we mean the following: + +```rust +macro_rules! printer { + (print $mvar:ident) => { + println!("{}", $mvar); + } + (print twice $mvar:ident) => { + println!("{}", $mvar); + println!("{}", $mvar); + } +} +``` + +`$mvar` is called a _metavariable_. Unlike normal variables, rather than binding +to a value in a computation, a metavariable binds _at compile time_ to a tree of +_tokens_. A _token_ zero or more symbols that together have some meaning. For +example, in our example definition, `print`, `$mvar`, `=>`, `{` are all tokens +(though that's not an exhaustive list). There are also other special tokens, +such as `EOF`, which indicates that there are no more tokens. The process of +producing a stream of tokens from the raw bytes of the source file is called +_lexing_. For more information about _lexing_, see the [Parsing +chapter][parsing] of this book. + +Whenever we refer to the "example _invocation_", we mean the following snippet: + +```rust +printer!(print foo); // Assume `foo` is a variable defined somewhere else... +``` + +The process of expanding the macro invocation into the syntax tree +`println!("{}", foo)` and then expanding that into a call to `Display::fmt` is +called _macro expansion_, it is the topic of this chapter. ### The macro parser +There are two parts to macro expansion: parsing the definition and parsing the +invocations. Interestingly, both are done by the macro parser. + Basically, the macro parser is like an NFA-based regex parser. It uses an algorithm similar in spirit to the [Earley parsing algorithm](https://en.wikipedia.org/wiki/Earley_parser). The macro parser is -defined in `src/libsyntax/ext/tt/macro_parser.rs`. - -In a traditional NFA-based parser, one common approach is to have some pattern -which we are trying to match an input against. Moreover, we may try to capture -some portion of the input and bind it to variable in the pattern. For example: -suppose we have a pattern (borrowing Rust macro syntax) such as `a $b:ident a` --- that is, an `a` token followed by an `ident` token followed by another `a` -token. Given an input `a foo a`, the _metavariable_ `$b` would bind to the -`ident` `foo`. On the other hand, an input `a foo b` would be rejected as a -parse failure because the pattern `a a` cannot match `a foo b` (or as -the compiler would put it, "no rules expected token `b`"). - -The macro parser does pretty much exactly that with one exception: in order to -parse different types of metavariables, such as `ident`, `block`, `expr`, etc., -the macro parser must sometimes call back to the normal Rust parser. - -Interestingly, both definitions and invokations of macros are parsed using the -macro parser. This is extremely non-intuitive and self-referential. The code to -parse macro _definitions_ is in `src/libsyntax/ext/tt/macro_rules.rs`. It -defines the pattern for matching for a macro definition as `$( $lhs:tt => -$rhs:tt );+`. In other words, a `macro_rules` defintion should have in its body -at least one occurence of a token tree followed by `=>` followed by another -token tree. When the compiler comes to a `macro_rules` definition, it uses this -pattern to match the two token trees per rule in the definition of the macro -_using the macro parser itself_. - -When the compiler comes to a macro invokation, it needs to parse that -invokation. This is also known as _macro expansion_. The same NFA-based macro -parser is used that is described above. Notably, the "pattern" (or _matcher_) -used is the first token tree extracted from the rules of the macro _definition_. -In other words, given some pattern described by the _definition_ of the macro, -we want to match the contents of the _invokation_ of the macro. - -The algorithm is exactly the same, but when the macro parser comes to a place in -the current matcher where it needs to match a _non-terminal_ (i.e. a -metavariable), it calls back to the normal Rust parser to get the contents of -that non-terminal. Then, the macro parser proceeds in parsing as normal. +defined in [`src/libsyntax/ext/tt/macro_parser.rs`][code_mp]. + +The interface of the macro parser is as follows (this is slightly simplified): + +```rust +fn parse( + sess: ParserSession, + tts: TokenStream, + ms: &[TokenTree] +) -> NamedParseResult +``` + +In this interface: + +- `sess` is a "parsing session", which keeps track of some metadata. Most + notably, this is used to keep track of errors that are generated so they can + be reported to the user. +- `tts` is a stream of tokens. The macro parser's job is to consume the raw + stream of tokens and output a binding of metavariables to corresponding token + trees. +- `ms` a _matcher_. This is a sequence of token trees that we want to match + `tts` against. + +In the analogy of a regex parser, `tts` is the input and we are matching it +against the pattern `ms`. Using our examples, `tts` could be the stream of +tokens containing the inside of the example invocation `print foo`, while `ms` +might be the sequence of token (trees) `print $mvar:ident`. + +The output of the parser is a `NamedParserResult`, which indicates which of +three cases has occured: + +- Success: `tts` matches the given matcher `ms`, and we have produced a binding + from metavariables to the corresponding token trees. +- Failure: `tts` does not match `ms`. This results in an error message such as + "No rule expected token _blah_". +- Error: some fatal error has occured _in the parser_. For example, this happens + if there are more than one pattern match, since that indicates the macro is + ambiguous. + +The full interface is defined [here][code_parse_int]. + +The macro parser does pretty much exactly the same as a normal regex parser with +one exception: in order to parse different types of metavariables, such as +`ident`, `block`, `expr`, etc., the macro parser must sometimes call back to the +normal Rust parser. + +As mentioned above, both definitions and invocations of macros are parsed using +the macro parser. This is extremely non-intuitive and self-referential. The code +to parse macro _definitions_ is in +[`src/libsyntax/ext/tt/macro_rules.rs`][code_mr]. It defines the pattern for +matching for a macro definition as `$( $lhs:tt => $rhs:tt );+`. In other words, +a `macro_rules` defintion should have in its body at least one occurence of a +token tree followed by `=>` followed by another token tree. When the compiler +comes to a `macro_rules` definition, it uses this pattern to match the two token +trees per rule in the definition of the macro _using the macro parser itself_. +In our example definition, the metavariable `$lhs` would match the patterns of +both arms: `(print $mvar:ident)` and `(print twice $mvar:ident)`. And `$rhs` +would match the bodies of both arms: `{ println!("{}", $mvar); }` and `{ +println!("{}", $mvar); println!("{}", $mvar); }`. The parser would keep this +knowledge around for when it needs to expand a macro invocation. + +When the compiler comes to a macro invocation, it parses that invocation using +the same NFA-based macro parser that is described above. However, the matcher +used is the first token tree (`$lhs`) extracted from the arms of the macro +_definition_. Using our example, we would try to match the token stream `print +foo` from the invocation against the matchers `print $mvar:ident` and `print +twice $mvar:ident` that we previously extracted from the definition. The +algorithm is exactly the same, but when the macro parser comes to a place in the +current matcher where it needs to match a _non-terminal_ (e.g. `$mvar:ident`), +it calls back to the normal Rust parser to get the contents of that +non-terminal. In this case, the Rust parser would look for an `ident` token, +which it finds (`foo`) and returns to the macro parser. Then, the macro parser +proceeds in parsing as normal. Also, note that exactly one of the matchers from +the various arms should match the invocation (otherwise, the macro is +ambiguous). For more information about the macro parser's implementation, see the comments -in `src/libsyntax/ext/tt/macro_parser.rs`. +in [`src/libsyntax/ext/tt/macro_parser.rs`][code_mp]. ### Hygiene @@ -64,3 +144,11 @@ TODO ### Custom Derive TODO + + + +[code_dir]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt +[code_mp]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt/macro_parser.rs +[code_mp]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt/macro_rules.rs +[code_parse_int]: https://github.com/rust-lang/rust/blob/a97cd17f5d71fb4ec362f4fbd79373a6e7ed7b82/src/libsyntax/ext/tt/macro_parser.rs#L421 +[parsing]: ./the-parser.md From 65263b6752355e52e616e90ac2d7d8f31e327a93 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 29 Jan 2018 09:59:47 -0500 Subject: [PATCH 032/648] breakup the MIR section and add an incremental compilation section --- src/SUMMARY.md | 9 ++- src/incremental-compilation.md | 139 +++++++++++++++++++++++++++++++++ src/mir.md | 1 + 3 files changed, 145 insertions(+), 4 deletions(-) create mode 100644 src/incremental-compilation.md create mode 100644 src/mir.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index e4bc24283..8e18969a1 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -6,7 +6,7 @@ - [Walkthrough: a typical contribution](./walkthrough.md) - [High-level overview of the compiler source](./high-level-overview.md) - [Queries: demand-driven compilation](./query.md) - - [Incremental compilation](./incremental-compilation.md) + - [Incremental compilation](./incremental-compilation.md) - [The parser](./the-parser.md) - [Macro expansion](./macro-expansion.md) - [Name resolution](./name-resolution.md) @@ -15,8 +15,9 @@ - [Type inference](./type-inference.md) - [Trait resolution](./trait-resolution.md) - [Type checking](./type-checking.md) -- [MIR construction](./mir-construction.md) -- [MIR borrowck](./mir-borrowck.md) -- [MIR optimizations](./mir-optimizations.md) +- [The MIR (Mid-level IR)](./mir.md) + - [MIR construction](./mir-construction.md) + - [MIR borrowck](./mir-borrowck.md) + - [MIR optimizations](./mir-optimizations.md) - [trans: generating LLVM IR](./trans.md) - [Glossary](./glossary.md) diff --git a/src/incremental-compilation.md b/src/incremental-compilation.md new file mode 100644 index 000000000..23910c5b3 --- /dev/null +++ b/src/incremental-compilation.md @@ -0,0 +1,139 @@ +# Incremental compilation + +The incremental compilation scheme is, in essence, a surprisingly +simple extension to the overall query system. We'll start by describing +a slightly simplified variant of the real thing, the "basic algorithm", and then describe +some possible improvements. + +## The basic algorithm + +The basic algorithm is +called the **red-green** algorithm[^salsa]. The high-level idea is +that, after each run of the compiler, we will save the results of all +the queries that we do, as well as the **query DAG**. The +**query DAG** is a [DAG] that indices which queries executed which +other queries. So for example there would be an edge from a query Q1 +to another query Q2 if computing Q1 required computing Q2 (note that +because queries cannot depend on themselves, this results in a DAG and +not a general graph). + +[DAG]: https://en.wikipedia.org/wiki/Directed_acyclic_graph + +On the next run of the compiler, then, we can sometimes reuse these +query results to avoid re-executing a query. We do this by assigning +every query a **color**: + +- If a query is colored **red**, that means that its result during + this compilation has **changed** from the previous compilation. +- If a query is colored **green**, that means that its result is + the **same** as the previous compilation. + +There are two key insights here: + +- First, if all the inputs to query Q are colored green, then the + query Q **must** result in the same value as last time and hence + need not be re-executed (or else the compiler is not deterministic). +- Second, even if some inputs to a query changes, it may be that it + **still** produces the same result as the previous compilation. In + particular, the query may only use part of its input. + - Therefore, after executing a query, we always check whether it + produced the same result as the previous time. **If it did,** we + can still mark the query as green, and hence avoid re-executing + dependent queries. + +### The try-mark-green algorithm + +The core of the incremental compilation is an algorithm called +"try-mark-green". It has the job of determining the color of a given +query Q (which must not yet have been executed). In cases where Q has +red inputs, determining Q's color may involve re-executing Q so that +we can compare its output; but if all of Q's inputs are green, then we +can determine that Q must be green without re-executing it or inspect +its value what-so-ever. In the compiler, this allows us to avoid +deserializing the result from disk when we don't need it, and -- in +fact -- enables us to sometimes skip *serializing* the result as well +(see the refinements section below). + +Try-mark-green works as follows: + +- First check if there is the query Q was executed during the previous + compilation. + - If not, we can just re-execute the query as normal, and assign it the + color of red. +- If yes, then load the 'dependent queries' that Q +- If there is a saved result, then we load the `reads(Q)` vector from the + query DAG. The "reads" is the set of queries that Q executed during + its execution. + - For each query R that in `reads(Q)`, we recursively demand the color + of R using try-mark-green. + - Note: it is important that we visit each node in `reads(Q)` in same order + as they occurred in the original compilation. See [the section on the query DAG below](#dag). + - If **any** of the nodes in `reads(Q)` wind up colored **red**, then Q is dirty. + - We re-execute Q and compare the hash of its result to the hash of the result + from the previous compilation. + - If the hash has not changed, we can mark Q as **green** and return. + - Otherwise, **all** of the nodes in `reads(Q)` must be **green**. In that case, + we can color Q as **green** and return. + + + +### The query DAG + +The query DAG code is stored in +[`src/librustc/dep_graph`][dep_graph]. Construction of the DAG is done +by instrumenting the query execution. + +One key point is that the query DAG also tracks ordering; that is, for +each query Q, we noy only track the queries that Q reads, we track the +**order** in which they were read. This allows try-mark-green to walk +those queries back in the same order. This is important because once a subquery comes back as red, +we can no longer be sure that Q will continue along the same path as before. +That is, imagine a query like this: + +```rust,ignore +fn main_query(tcx) { + if tcx.subquery1() { + tcx.subquery2() + } else { + tcx.subquery3() + } +} +``` + +Now imagine that in the first compilation, `main_query` starts by +executing `subquery1`, and this returns true. In that case, the next +query `main_query` executes will be `subquery2`, and `subquery3` will +not be executed at all. + +But now imagine that in the **next** compilation, the input has +changed such that `subquery` returns **false**. In this case, `subquery2` would never +execute. If try-mark-green were to visit `reads(main_query)` out of order, +however, it might have visited `subquery2` before `subquery1`, and hence executed it. +This can lead to ICEs and other problems in the compiler. + +[dep_graph]: https://github.com/rust-lang/rust/tree/master/src/librustc/dep_graph + +## Improvements to the basic algorithm + +In the description basic algorithm, we said that at the end of +compilation we would save the results of all the queries that were +performed. In practice, this can be quite wasteful -- many of those +results are very cheap to recompute, and serializing + deserializing +them is not a particular win. In practice, what we would do is to save +**the hashes** of all the subqueries that we performed. Then, in select cases, +we **also** save the results. + +This is why the incremental algorithm separates computing the +**color** of a node, which often does not require its value, from +computing the **result** of a node. Computing the result is done via a simple algorithm +like so: + +- Check if a saved result for Q is available. If so, compute the color of Q. + If Q is green, deserialize and return the saved result. +- Otherwise, execute Q. + - We can then compare the hash of the result and color Q as green if + it did not change. + +# Footnotes + +[^salsa]: I have long wanted to rename it to the Salsa algorithm, but it never caught on. -@nikomatsakis diff --git a/src/mir.md b/src/mir.md new file mode 100644 index 000000000..2be6a2e1a --- /dev/null +++ b/src/mir.md @@ -0,0 +1 @@ +# The MIR (Mid-level IR) From 0414ffee78512cb31082e979f4a53aa38ff22de4 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 29 Jan 2018 10:25:44 -0500 Subject: [PATCH 033/648] create links in the glossary --- src/glossary.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/src/glossary.md b/src/glossary.md index b66e17ea3..338c9e93f 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -9,23 +9,24 @@ AST | the abstract syntax tree produced by the syntax crate codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. cx | we tend to use "cx" as an abbrevation for context. See also `tcx`, `infcx`, etc. DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. -HIR | the High-level IR, created by lowering and desugaring the AST. See `librustc/hir`. +HIR | the High-level IR, created by lowering and desugaring the AST ([see more](hir.html)) HirId | identifies a particular node in the HIR by combining a def-id with an "intra-definition offset". -'gcx | the lifetime of the global arena (see `librustc/ty`). +'gcx | the lifetime of the global arena ([see more](ty.html)) generics | the set of generic type parameters defined on a type or item ICE | internal compiler error. When the compiler crashes. infcx | the inference context (see `librustc/infer`) MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans. Defined in the `src/librustc/mir/` module, but much of the code that manipulates it is found in `src/librustc_mir`. -obligation | something that must be proven by the trait system; see `librustc/traits`. +obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) local crate | the crate currently being compiled. node-id or NodeId | an index identifying a particular node in the AST or HIR; gradually being phased out and replaced with `HirId`. -query | perhaps some sub-computation during compilation; see `librustc/maps`. -provider | the function that executes a query; see `librustc/maps`. +query | perhaps some sub-computation during compilation ([see more](query.html)) +provider | the function that executes a query ([see more](query.html)) sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. substs | the substitutions for a given generic type or item (e.g., the `i32`, `u32` in `HashMap`) -tcx | the "typing context", main data structure of the compiler (see `librustc/ty`). +tcx | the "typing context", main data structure of the compiler ([see more](ty.html)) +'tcx | the lifetime of the currently active inference context ([see more](ty.html)) trans | the code to translate MIR into LLVM IR. -trait reference | a trait and values for its type parameters (see `librustc/ty`). -ty | the internal representation of a type (see `librustc/ty`). +trait reference | a trait and values for its type parameters ([see more](ty.html)). +ty | the internal representation of a type ([see more](ty.html)). From bf775929af659d3e25dd0458b8d8465db8378b8f Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 29 Jan 2018 10:26:35 -0500 Subject: [PATCH 034/648] link glossary to mir --- src/glossary.md | 2 +- src/mir.md | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/src/glossary.md b/src/glossary.md index 338c9e93f..3202e5f4c 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -15,7 +15,7 @@ HirId | identifies a particular node in the HIR by combining generics | the set of generic type parameters defined on a type or item ICE | internal compiler error. When the compiler crashes. infcx | the inference context (see `librustc/infer`) -MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans. Defined in the `src/librustc/mir/` module, but much of the code that manipulates it is found in `src/librustc_mir`. +MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) local crate | the crate currently being compiled. node-id or NodeId | an index identifying a particular node in the AST or HIR; gradually being phased out and replaced with `HirId`. diff --git a/src/mir.md b/src/mir.md index 2be6a2e1a..eeba68472 100644 --- a/src/mir.md +++ b/src/mir.md @@ -1 +1,6 @@ # The MIR (Mid-level IR) + +TODO + +Defined in the `src/librustc/mir/` module, but much of the code that +manipulates it is found in `src/librustc_mir`. From dee42c16732550a81cc7aa287b2404afd8cfa8b3 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 29 Jan 2018 11:20:46 -0600 Subject: [PATCH 035/648] Rewrite 'tokens' para... --- src/macro-expansion.md | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/src/macro-expansion.md b/src/macro-expansion.md index dd735ed23..da4615128 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -26,15 +26,21 @@ macro_rules! printer { } ``` -`$mvar` is called a _metavariable_. Unlike normal variables, rather than binding -to a value in a computation, a metavariable binds _at compile time_ to a tree of -_tokens_. A _token_ zero or more symbols that together have some meaning. For -example, in our example definition, `print`, `$mvar`, `=>`, `{` are all tokens -(though that's not an exhaustive list). There are also other special tokens, -such as `EOF`, which indicates that there are no more tokens. The process of -producing a stream of tokens from the raw bytes of the source file is called -_lexing_. For more information about _lexing_, see the [Parsing -chapter][parsing] of this book. +`$mvar` is called a _metavariable_. Unlike normal variables, rather than +binding to a value in a computation, a metavariable binds _at compile time_ to +a tree of _tokens_. A _token_ is a single "unit" of the grammar, such as an +identifier (e.g., `foo`) or punctuation (e.g., `=>`). There are also other +special tokens, such as `EOF`, which indicates that there are no more tokens. +Token trees resulting from paired parentheses-like characters (`(`...`)`, +`[`...`]`, and `{`...`}`) -- they include the open and close and all the tokens +in between (we do require that parentheses-like characters be balanced). Having +macro expansion operate on token streams rather than the raw bytes of a source +file abstracts away a lot of complexity. The macro expander (and much of the +rest of the compiler) doesn't really care that much about the exact line and +column of some syntactic construct in the code; it cares about what constructs +are used in the code. Using tokens allows us to care about _what_ without +worrying about _where_. For more information about tokens, see the +[Parsing][parsing] chapter of this book. Whenever we refer to the "example _invocation_", we mean the following snippet: @@ -44,7 +50,7 @@ printer!(print foo); // Assume `foo` is a variable defined somewhere else... The process of expanding the macro invocation into the syntax tree `println!("{}", foo)` and then expanding that into a call to `Display::fmt` is -called _macro expansion_, it is the topic of this chapter. +called _macro expansion_, and it is the topic of this chapter. ### The macro parser From ed50f8d5d7e4ee786578de54a49868edfbbe8c97 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Mon, 29 Jan 2018 11:37:40 -0600 Subject: [PATCH 036/648] I'm guessing this was a typo? --- src/high-level-overview.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/high-level-overview.md b/src/high-level-overview.md index b8d75bff6..55b596a2a 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -134,8 +134,3 @@ take: (one for each "codegen unit"). 6. **Linking** - Finally, those `.o` files are linked together. - - - - -The first thing you may wonder if From 82da67a7519723b5cf3e071f6cdfca0c6114bc83 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 29 Jan 2018 11:50:34 -0600 Subject: [PATCH 037/648] Corrected relationship of macro and rust parsers --- src/macro-expansion.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/src/macro-expansion.md b/src/macro-expansion.md index da4615128..a7777e80c 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -2,12 +2,13 @@ Macro expansion happens during parsing. `rustc` has two parsers, in fact: the normal Rust parser, and the macro parser. During the parsing phase, the normal -Rust parser will call into the macro parser when it encounters a macro -definition or macro invocation (TODO: verify). The macro parser, in turn, may -call back out to the Rust parser when it needs to bind a metavariable (e.g. -`$my_expr`) while parsing the contents of a macro invocation. The code for macro -expansion is in [`src/libsyntax/ext/tt/`][code_dir]. This chapter aims to -explain how macro expansion works. +Rust parser will set aside the contents of macros and their invokations. Later, +before name resolution, macros are expanded using these portions of the code. +The macro parser, in turn, may call the normal Rust parser when it needs to +bind a metavariable (e.g. `$my_expr`) while parsing the contents of a macro +invocation. The code for macro expansion is in +[`src/libsyntax/ext/tt/`][code_dir]. This chapter aims to explain how macro +expansion works. ### Example From f47633dd334d53dd09a8fe66866eb9e7ae81f06e Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 29 Jan 2018 13:12:28 -0600 Subject: [PATCH 038/648] copy type inference readme --- src/type-inference.md | 226 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 226 insertions(+) diff --git a/src/type-inference.md b/src/type-inference.md index 0fc99252a..6e2032fee 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -1 +1,227 @@ # Type inference + +The type inference is based on standard HM-type inference, but +extended in various way to accommodate subtyping, region inference, +and higher-ranked types. + +## A note on terminology + +We use the notation `?T` to refer to inference variables, also called +existential variables. + +We use the term "region" and "lifetime" interchangeably. Both refer to +the `'a` in `&'a T`. + +The term "bound region" refers to regions bound in a function +signature, such as the `'a` in `for<'a> fn(&'a u32)`. A region is +"free" if it is not bound. + +## Creating an inference context + +You create and "enter" an inference context by doing something like +the following: + +```rust +tcx.infer_ctxt().enter(|infcx| { + // use the inference context `infcx` in here +}) +``` + +Each inference context creates a short-lived type arena to store the +fresh types and things that it will create, as described in +[the README in the ty module][ty-readme]. This arena is created by the `enter` +function and disposed after it returns. + +[ty-readme]: src/librustc/ty/README.md + +Within the closure, the infcx will have the type `InferCtxt<'cx, 'gcx, +'tcx>` for some fresh `'cx` and `'tcx` -- the latter corresponds to +the lifetime of this temporary arena, and the `'cx` is the lifetime of +the `InferCtxt` itself. (Again, see [that ty README][ty-readme] for +more details on this setup.) + +The `tcx.infer_ctxt` method actually returns a build, which means +there are some kinds of configuration you can do before the `infcx` is +created. See `InferCtxtBuilder` for more information. + +## Inference variables + +The main purpose of the inference context is to house a bunch of +**inference variables** -- these represent types or regions whose precise +value is not yet known, but will be uncovered as we perform type-checking. + +If you're familiar with the basic ideas of unification from H-M type +systems, or logic languages like Prolog, this is the same concept. If +you're not, you might want to read a tutorial on how H-M type +inference works, or perhaps this blog post on +[unification in the Chalk project]. + +[Unification in the Chalk project]: http://smallcultfollowing.com/babysteps/blog/2017/03/25/unification-in-chalk-part-1/ + +All told, the inference context stores four kinds of inference variables as of this +writing: + +- Type variables, which come in three varieties: + - General type variables (the most common). These can be unified with any type. + - Integral type variables, which can only be unified with an integral type, and + arise from an integer literal expression like `22`. + - Float type variables, which can only be unified with a float type, and + arise from a float literal expression like `22.0`. +- Region variables, which represent lifetimes, and arise all over the dang place. + +All the type variables work in much the same way: you can create a new +type variable, and what you get is `Ty<'tcx>` representing an +unresolved type `?T`. Then later you can apply the various operations +that the inferencer supports, such as equality or subtyping, and it +will possibly **instantiate** (or **bind**) that `?T` to a specific +value as a result. + +The region variables work somewhat differently, and are described +below in a separate section. + +## Enforcing equality / subtyping + +The most basic operations you can perform in the type inferencer is +**equality**, which forces two types `T` and `U` to be the same. The +recommended way to add an equality constraint is using the `at` +method, roughly like so: + +``` +infcx.at(...).eq(t, u); +``` + +The first `at()` call provides a bit of context, i.e., why you are +doing this unification, and in what environment, and the `eq` method +performs the actual equality constraint. + +When you equate things, you force them to be precisely equal. Equating +returns a `InferResult` -- if it returns `Err(err)`, then equating +failed, and the enclosing `TypeError` will tell you what went wrong. + +The success case is perhaps more interesting. The "primary" return +type of `eq` is `()` -- that is, when it succeeds, it doesn't return a +value of any particular interest. Rather, it is executed for its +side-effects of constraining type variables and so forth. However, the +actual return type is not `()`, but rather `InferOk<()>`. The +`InferOk` type is used to carry extra trait obligations -- your job is +to ensure that these are fulfilled (typically by enrolling them in a +fulfillment context). See the [trait README] for more background here. + +[trait README]: ../traits/README.md + +You can also enforce subtyping through `infcx.at(..).sub(..)`. The same +basic concepts apply as above. + +## "Trying" equality + +Sometimes you would like to know if it is *possible* to equate two +types without error. You can test that with `infcx.can_eq` (or +`infcx.can_sub` for subtyping). If this returns `Ok`, then equality +is possible -- but in all cases, any side-effects are reversed. + +Be aware though that the success or failure of these methods is always +**modulo regions**. That is, two types `&'a u32` and `&'b u32` will +return `Ok` for `can_eq`, even if `'a != 'b`. This falls out from the +"two-phase" nature of how we solve region constraints. + +## Snapshots + +As described in the previous section on `can_eq`, often it is useful +to be able to do a series of operations and then roll back their +side-effects. This is done for various reasons: one of them is to be +able to backtrack, trying out multiple possibilities before settling +on which path to take. Another is in order to ensure that a series of +smaller changes take place atomically or not at all. + +To allow for this, the inference context supports a `snapshot` method. +When you call it, it will start recording changes that occur from the +operations you perform. When you are done, you can either invoke +`rollback_to`, which will undo those changes, or else `confirm`, which +will make the permanent. Snapshots can be nested as long as you follow +a stack-like discipline. + +Rather than use snapshots directly, it is often helpful to use the +methods like `commit_if_ok` or `probe` that encapsulate higher-level +patterns. + +## Subtyping obligations + +One thing worth discussing are subtyping obligations. When you force +two types to be a subtype, like `?T <: i32`, we can often convert those +into equality constraints. This follows from Rust's rather limited notion +of subtyping: so, in the above case, `?T <: i32` is equivalent to `?T = i32`. + +However, in some cases we have to be more careful. For example, when +regions are involved. So if you have `?T <: &'a i32`, what we would do +is to first "generalize" `&'a i32` into a type with a region variable: +`&'?b i32`, and then unify `?T` with that (`?T = &'?b i32`). We then +relate this new variable with the original bound: + + &'?b i32 <: &'a i32 + +This will result in a region constraint (see below) of `'?b: 'a`. + +One final interesting case is relating two unbound type variables, +like `?T <: ?U`. In that case, we can't make progress, so we enqueue +an obligation `Subtype(?T, ?U)` and return it via the `InferOk` +mechanism. You'll have to try again when more details about `?T` or +`?U` are known. + +## Region constraints + +Regions are inferred somewhat differently from types. Rather than +eagerly unifying things, we simply collect constraints as we go, but +make (almost) no attempt to solve regions. These constraints have the +form of an outlives constraint: + + 'a: 'b + +Actually the code tends to view them as a subregion relation, but it's the same +idea: + + 'b <= 'a + +(There are various other kinds of constriants, such as "verifys"; see +the `region_constraints` module for details.) + +There is one case where we do some amount of eager unification. If you have an equality constraint +between two regions + + 'a = 'b + +we will record that fact in a unification table. You can then use +`opportunistic_resolve_var` to convert `'b` to `'a` (or vice +versa). This is sometimes needed to ensure termination of fixed-point +algorithms. + +## Extracting region constraints + +Ultimately, region constraints are only solved at the very end of +type-checking, once all other constraints are known. There are two +ways to solve region constraints right now: lexical and +non-lexical. Eventually there will only be one. + +To solve **lexical** region constraints, you invoke +`resolve_regions_and_report_errors`. This will "close" the region +constraint process and invoke the `lexical_region_resolve` code. Once +this is done, any further attempt to equate or create a subtyping +relationship will yield an ICE. + +Non-lexical region constraints are not handled within the inference +context. Instead, the NLL solver (actually, the MIR type-checker) +invokes `take_and_reset_region_constraints` periodically. This +extracts all of the outlives constraints from the region solver, but +leaves the set of variables intact. This is used to get *just* the +region constraints that resulted from some particular point in the +program, since the NLL solver needs to know not just *what* regions +were subregions but *where*. Finally, the NLL solver invokes +`take_region_var_origins`, which "closes" the region constraint +process in the same way as normal solving. + +## Lexical region resolution + +Lexical region resolution is done by initially assigning each region +variable to an empty value. We then process each outlives constraint +repeatedly, growing region variables until a fixed-point is reached. +Region variables can be grown using a least-upper-bound relation on +the region lattice in a fairly straight-forward fashion. From 066a32c961682259aef620ba73cc9274da98d39a Mon Sep 17 00:00:00 2001 From: Michal 'vorner' Vaner Date: Mon, 29 Jan 2018 22:18:35 +0100 Subject: [PATCH 039/648] The first approximation of name resolution (#22) * The first approximation of name resolution The first attempt to write something useful about the name resolution. As the TODO section says, his is not finished thing, but it might hopefully be useful to someone already. --- src/name-resolution.md | 106 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) diff --git a/src/name-resolution.md b/src/name-resolution.md index 487a31d2c..25ce3e29e 100644 --- a/src/name-resolution.md +++ b/src/name-resolution.md @@ -1 +1,107 @@ # Name resolution + +The name resolution is a separate pass in the compiler. Its input is the syntax +tree, produced by parsing input files. It produces links from all the names in +the source to relevant places where the name was introduced. It also generates +helpful error messages, like typo suggestions or traits to import. + +The name resolution lives in the `librustc_resolve` crate, with the meat in +`lib.rs` and some helpers or symbol-type specific logic in the other modules. + +## Namespaces + +Different kind of symbols live in different namespaces ‒ eg. types don't +clash with variables. This usually doesn't happen, because variables start with +lower-case letter while types with upper case one, but this is only a +convention. This is legal Rust code that'll compile (with warnings): + +```rust +type x = u32; +let x: x = 1; +let y: x = 2; // See? x is still a type here. +``` + +To cope with this, and with slightly different scoping rules for these +namespaces, the resolver keeps them separated and builds separate structures for +them. + +In other words, when the code talks about namespaces, it doesn't mean the module +hierarchy, it's types vs. values vs. macros. + +## Scopes and ribs + +A name is visible only in certain area in the source code. This forms a +hierarchical structure, but not necessarily a simple one ‒ if one scope is part +of another, it doesn't mean the name visible in the outer one is also visible in +the inner one, or that it refers to the same thing. + +To cope with that, the compiler introduces the concept of Ribs. This is +abstraction of a scope. Every time the set of visible names potentially changes, +a new rib is pushed onto a stack. The places where this can happen includes for +example: + +* The obvious places ‒ curly braces enclosing a block, function boundaries, + modules. +* Introducing a let binding ‒ this can shadow another binding with the same + name. +* Macro expansion border ‒ to cope with macro hygiene. + +When searching for a name, the stack of ribs is traversed from the innermost +outwards. This helps to find the closest meaning of the name (the one not +shadowed by anything else). The transition to outer rib may also change the +rules what names are usable ‒ if there are nested functions (not closures), the +inner one can't access parameters and local bindings of the outer one, even +though they should be visible by ordinary scoping rules. An example: + +```rust +fn do_something(val: T) { // <- New rib in both types and values (1) + // `val` is accessible, as is the helper function + // `T` is accessible + let helper = || { // New rib on `helper` (2) and another on the block (3) + // `val` is accessible here + }; // End of (3) + // `val` is accessible, `helper` variable shadows `helper` function + fn helper() { // <- New rib in both types and values (4) + // `val` is not accessible here, (4) is not transparent for locals) + // `T` is not accessible here + } // End of (4) + let val = T::default(); // New rib (5) + // `val` is the variable, not the parameter here +} // End of (5), (2) and (1) +``` + +Because the rules for different namespaces are a bit different, each namespace +has its own independent rib stack that is constructed in parallel to the others. + +## Overall strategy + +To perform the name resolution of the whole crate, the syntax tree is traversed +top-down and every encountered name is resolved. This works for most kinds of +names, because at the point of use of a name it is already introduced in the Rib +hierarchy. + +There are some exceptions to this. Items are bit tricky, because they can be +used even before encountered ‒ therefore every block needs to be first scanned +for items to fill in its Rib. + +Other, even more problematic ones, are imports which need recursive fixed-point +resolution and macros, that need to be resolved and expanded before the rest of +the code can be processed. + +Therefore, the resolution is performed in multiple stages. + +## TODO: + +This is a result of the first pass of learning the code. It is definitely +incomplete and not detailed enough. It also might be inaccurate in places. +Still, it probably provides useful first guidepost to what happens in there. + +* What exactly does it link to and how is that published and consumed by + following stages of compilation? +* Who calls it and how it is actually used. +* Is it a pass and then the result is only used, or can it be computed + incrementally (eg. for RLS)? +* The overall strategy description is a bit vague. +* Where does the name `Rib` come from? +* Does this thing have its own tests, or is it tested only as part of some e2e + testing? From 150db58b61a0d67c596af234e2e47ab635cc3a52 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Tue, 30 Jan 2018 11:38:37 -0600 Subject: [PATCH 040/648] Fix minor typos --- src/incremental-compilation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/incremental-compilation.md b/src/incremental-compilation.md index 23910c5b3..df88125e1 100644 --- a/src/incremental-compilation.md +++ b/src/incremental-compilation.md @@ -64,7 +64,7 @@ Try-mark-green works as follows: - If there is a saved result, then we load the `reads(Q)` vector from the query DAG. The "reads" is the set of queries that Q executed during its execution. - - For each query R that in `reads(Q)`, we recursively demand the color + - For each query R in `reads(Q)`, we recursively demand the color of R using try-mark-green. - Note: it is important that we visit each node in `reads(Q)` in same order as they occurred in the original compilation. See [the section on the query DAG below](#dag). @@ -84,7 +84,7 @@ The query DAG code is stored in by instrumenting the query execution. One key point is that the query DAG also tracks ordering; that is, for -each query Q, we noy only track the queries that Q reads, we track the +each query Q, we not only track the queries that Q reads, we track the **order** in which they were read. This allows try-mark-green to walk those queries back in the same order. This is important because once a subquery comes back as red, we can no longer be sure that Q will continue along the same path as before. From 4f983ebe1462ca0ce821647dc8476feb184c4ca4 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 30 Jan 2018 16:29:16 -0600 Subject: [PATCH 041/648] Add a para about bootstrapping --- src/how-to-build-and-run.md | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index f685f569c..24cfbfad7 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -1,7 +1,7 @@ # How to build the compiler and run what you built The compiler is built using a tool called `x.py`. You will need to -have Python installed to run it. But before we get to that, if you're going to +have Python installed to run it. But before we get to that, if you're going to be hacking on rustc, you'll want to tweak the configuration of the compiler. The default configuration is oriented towards running the compiler as a user, not a developer. @@ -48,6 +48,19 @@ use-jemalloc = false ### Running x.py and building a stage1 compiler +One thing to keep in mind is that `rustc` is a _bootstrapping_ compiler. That +is, since `rustc` is written in Rust, we need to use an older version of the +compiler to compile the newer version. In particular, the newer version of the +compiler, `libstd`, and other tooling may use some unstable features +internally. The result is the compiling `rustc` is done in stages. + +- Stage 0: the current _beta_ compiler is compiled using the current _stable_ compiler. +- Stage 1: the code in your clone is then compiled with the stage 0 compiler. +- Stage 2: the code in your clone is then compiled with the stage 1 compiler (i.e. it builds itself). + +For hacking, often building the stage 1 compiler is enough, but for testing and +release, the stage 2 compiler is used. + Once you've created a config.toml, you are now ready to run `x.py`. There are a lot of options here, but let's start with what is probably the best "go to" command for building a local rust: @@ -117,4 +130,4 @@ Here are a few other useful x.py commands. We'll cover some of them in detail in - `./x.py build` -- builds the stage2 compiler - Running tests (see the section [running tests](./running-tests.html) for more details): - `./x.py test --stage 1 src/libstd` -- runs the `#[test]` tests from libstd - - `./x.py test --stage 1 src/test/run-pass` -- runs the `run-pass` test suite + - `./x.py test --stage 1 src/test/run-pass` -- runs the `run-pass` test suite From c3eb273f8eb6a77bc808e6692017b2e20ab91391 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Wed, 31 Jan 2018 01:20:07 +0000 Subject: [PATCH 042/648] improved grammar use en-dashes instead of `--` --- src/about-this-guide.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/about-this-guide.md b/src/about-this-guide.md index ea840177b..79106736f 100644 --- a/src/about-this-guide.md +++ b/src/about-this-guide.md @@ -1,14 +1,14 @@ # About this guide -This guide is meant to help document how rustc -- the Rust compiler -- +This guide is meant to help document how rustc – the Rust compiler – works, as well as to help new contributors get involved in rustc -development. It is not meant to replace code documentation -- each -chapter gives only high-level details, the kinds of things that +development. It is not meant to replace code documentation – each +chapter gives only high-level details – the kinds of things that (ideally) don't change frequently. -The guide itself is of course open source as well, and the sources can -be found at [the GitHub repository]. If you find any mistakes in the -guide, please file an issue about it -- or, even better, open a PR +The guide itself is of course open-source as well, and the sources can +be found at the [GitHub repository]. If you find any mistakes in the +guide, please file an issue about it, or even better, open a PR with a correction! -[the GitHub repository]: https://github.com/rust-lang-nursery/rustc-guide/ +[GitHub repository]: https://github.com/rust-lang-nursery/rustc-guide/ From d15c82a6c5a76f4ce7458a24b05d32516848c204 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Wed, 31 Jan 2018 01:32:49 +0000 Subject: [PATCH 043/648] improved grammar and fixed small errors --- src/incremental-compilation.md | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-) diff --git a/src/incremental-compilation.md b/src/incremental-compilation.md index df88125e1..5cee70dbd 100644 --- a/src/incremental-compilation.md +++ b/src/incremental-compilation.md @@ -2,7 +2,7 @@ The incremental compilation scheme is, in essence, a surprisingly simple extension to the overall query system. We'll start by describing -a slightly simplified variant of the real thing, the "basic algorithm", and then describe +a slightly simplified variant of the real thing – the "basic algorithm" – and then describe some possible improvements. ## The basic algorithm @@ -11,8 +11,8 @@ The basic algorithm is called the **red-green** algorithm[^salsa]. The high-level idea is that, after each run of the compiler, we will save the results of all the queries that we do, as well as the **query DAG**. The -**query DAG** is a [DAG] that indices which queries executed which -other queries. So for example there would be an edge from a query Q1 +**query DAG** is a [DAG] that indexes which queries executed which +other queries. So, for example, there would be an edge from a query Q1 to another query Q2 if computing Q1 required computing Q2 (note that because queries cannot depend on themselves, this results in a DAG and not a general graph). @@ -43,24 +43,23 @@ There are two key insights here: ### The try-mark-green algorithm -The core of the incremental compilation is an algorithm called +At the core of incremental compilation is an algorithm called "try-mark-green". It has the job of determining the color of a given -query Q (which must not yet have been executed). In cases where Q has +query Q (which must not have yet been executed). In cases where Q has red inputs, determining Q's color may involve re-executing Q so that -we can compare its output; but if all of Q's inputs are green, then we -can determine that Q must be green without re-executing it or inspect -its value what-so-ever. In the compiler, this allows us to avoid -deserializing the result from disk when we don't need it, and -- in -fact -- enables us to sometimes skip *serializing* the result as well +we can compare its output, but if all of Q's inputs are green, then we +can conclude that Q must be green without re-executing it or inspecting +its value, regardless. In the compiler, this allows us to avoid +deserializing the result from disk when we don't need it, and in fact +enables us to sometimes skip *serializing* the result as well (see the refinements section below). Try-mark-green works as follows: -- First check if there is the query Q was executed during the previous - compilation. +- First check if the query Q was executed during the previous compilation. - If not, we can just re-execute the query as normal, and assign it the color of red. -- If yes, then load the 'dependent queries' that Q +- If yes, then load the 'dependent queries' of Q. - If there is a saved result, then we load the `reads(Q)` vector from the query DAG. The "reads" is the set of queries that Q executed during its execution. @@ -106,9 +105,9 @@ query `main_query` executes will be `subquery2`, and `subquery3` will not be executed at all. But now imagine that in the **next** compilation, the input has -changed such that `subquery` returns **false**. In this case, `subquery2` would never +changed such that `subquery1` returns **false**. In this case, `subquery2` would never execute. If try-mark-green were to visit `reads(main_query)` out of order, -however, it might have visited `subquery2` before `subquery1`, and hence executed it. +however, it visit `subquery2` before `subquery1`, and hence execute it. This can lead to ICEs and other problems in the compiler. [dep_graph]: https://github.com/rust-lang/rust/tree/master/src/librustc/dep_graph @@ -117,8 +116,8 @@ This can lead to ICEs and other problems in the compiler. In the description basic algorithm, we said that at the end of compilation we would save the results of all the queries that were -performed. In practice, this can be quite wasteful -- many of those -results are very cheap to recompute, and serializing + deserializing +performed. In practice, this can be quite wasteful – many of those +results are very cheap to recompute, and serializing and deserializing them is not a particular win. In practice, what we would do is to save **the hashes** of all the subqueries that we performed. Then, in select cases, we **also** save the results. From 2ee639a7aa9b698f4cf2988e2bc13ce070df272a Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Wed, 31 Jan 2018 02:08:14 +0000 Subject: [PATCH 044/648] fixed grammatical oversight --- src/incremental-compilation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/incremental-compilation.md b/src/incremental-compilation.md index 5cee70dbd..2ab053e17 100644 --- a/src/incremental-compilation.md +++ b/src/incremental-compilation.md @@ -107,7 +107,7 @@ not be executed at all. But now imagine that in the **next** compilation, the input has changed such that `subquery1` returns **false**. In this case, `subquery2` would never execute. If try-mark-green were to visit `reads(main_query)` out of order, -however, it visit `subquery2` before `subquery1`, and hence execute it. +however, it might visit `subquery2` before `subquery1`, and hence execute it. This can lead to ICEs and other problems in the compiler. [dep_graph]: https://github.com/rust-lang/rust/tree/master/src/librustc/dep_graph From 606fe051146540222f224418731ae78eab28e840 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Wed, 31 Jan 2018 02:09:24 +0000 Subject: [PATCH 045/648] minor clarification --- src/incremental-compilation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/incremental-compilation.md b/src/incremental-compilation.md index 2ab053e17..dc4d06c6c 100644 --- a/src/incremental-compilation.md +++ b/src/incremental-compilation.md @@ -49,7 +49,7 @@ query Q (which must not have yet been executed). In cases where Q has red inputs, determining Q's color may involve re-executing Q so that we can compare its output, but if all of Q's inputs are green, then we can conclude that Q must be green without re-executing it or inspecting -its value, regardless. In the compiler, this allows us to avoid +its value at all. In the compiler, this allows us to avoid deserializing the result from disk when we don't need it, and in fact enables us to sometimes skip *serializing* the result as well (see the refinements section below). From ed1e1f24dea07fef6f5a6138045cedaefbe8e32d Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Wed, 31 Jan 2018 02:09:59 +0000 Subject: [PATCH 046/648] changed hyphens to en-dashes --- src/the-parser.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/the-parser.md b/src/the-parser.md index 9a01d8a36..456f0a9ea 100644 --- a/src/the-parser.md +++ b/src/the-parser.md @@ -9,8 +9,8 @@ The bulk of the parser lives in the [libsyntax] crate. Like most parsers, the parsing process is composed of two main steps, -- lexical analysis - turn a stream of characters into a stream of token trees -- parsing - turn the token trees into an AST +- lexical analysis – turn a stream of characters into a stream of token trees +- parsing – turn the token trees into an AST The `syntax` crate contains several main players, @@ -39,4 +39,4 @@ all the information needed while parsing, as well as the `CodeMap` itself. [parser module]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/parse [`Parser`]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/parser.rs [`StringReader`]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/lexer/mod.rs -[visit module]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/visit.rs \ No newline at end of file +[visit module]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/visit.rs From 239e2c600aa0ac538bcbef209942b8c543ff564c Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 31 Jan 2018 12:22:42 -0600 Subject: [PATCH 047/648] copy MIR readme --- src/mir.md | 105 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 102 insertions(+), 3 deletions(-) diff --git a/src/mir.md b/src/mir.md index eeba68472..dbac8eb53 100644 --- a/src/mir.md +++ b/src/mir.md @@ -1,6 +1,105 @@ # The MIR (Mid-level IR) -TODO +MIR is Rust's _Mid-level Intermediate Representation_. It is constructed from +HIR (described in an earlier chapter). -Defined in the `src/librustc/mir/` module, but much of the code that -manipulates it is found in `src/librustc_mir`. +MIR is defined in the [`src/librustc/mir/`][mir] module, but much of the code +that manipulates it is found in [`src/librustc_mir`][mirmanip]. + + +_NOTE: copy/pasted from README... needs editing_ + +# MIR definition and pass system + +This file contains the definition of the MIR datatypes along with the +various types for the "MIR Pass" system, which lets you easily +register and define new MIR transformations and analyses. + +Most of the code that operates on MIR can be found in the +`librustc_mir` crate or other crates. The code found here in +`librustc` is just the datatype definitions, along with the functions +which operate on MIR to be placed everywhere else. + +## MIR Data Types and visitor + +The main MIR data type is `rustc::mir::Mir`, defined in `mod.rs`. +There is also the MIR visitor (in `visit.rs`) which allows you to walk +the MIR and override what actions will be taken at various points (you +can visit in either shared or mutable mode; the latter allows changing +the MIR in place). Finally `traverse.rs` contains various traversal +routines for visiting the MIR CFG in [different standard orders][traversal] +(e.g. pre-order, reverse post-order, and so forth). + +[traversal]: https://en.wikipedia.org/wiki/Tree_traversal + +## MIR pass suites and their integration into the query system + +As a MIR *consumer*, you are expected to use one of the queries that +returns a "final MIR". As of the time of this writing, there is only +one: `optimized_mir(def_id)`, but more are expected to come in the +future. For foreign def-ids, we simply read the MIR from the other +crate's metadata. But for local def-ids, the query will construct the +MIR and then iteratively optimize it by putting it through various +pipeline stages. This section describes those pipeline stages and how +you can extend them. + +To produce the `optimized_mir(D)` for a given def-id `D`, the MIR +passes through several suites of optimizations, each represented by a +query. Each suite consists of multiple optimizations and +transformations. These suites represent useful intermediate points +where we want to access the MIR for type checking or other purposes: + +- `mir_build(D)` -- not a query, but this constructs the initial MIR +- `mir_const(D)` -- applies some simple transformations to make MIR ready for constant evaluation; +- `mir_validated(D)` -- applies some more transformations, making MIR ready for borrow checking; +- `optimized_mir(D)` -- the final state, after all optimizations have been performed. + +### Stealing + +The intermediate queries `mir_const()` and `mir_validated()` yield up +a `&'tcx Steal>`, allocated using +`tcx.alloc_steal_mir()`. This indicates that the result may be +**stolen** by the next suite of optimizations -- this is an +optimization to avoid cloning the MIR. Attempting to use a stolen +result will cause a panic in the compiler. Therefore, it is important +that you do not read directly from these intermediate queries except as +part of the MIR processing pipeline. + +Because of this stealing mechanism, some care must also be taken to +ensure that, before the MIR at a particular phase in the processing +pipeline is stolen, anyone who may want to read from it has already +done so. Concretely, this means that if you have some query `foo(D)` +that wants to access the result of `mir_const(D)` or +`mir_validated(D)`, you need to have the successor pass "force" +`foo(D)` using `ty::queries::foo::force(...)`. This will force a query +to execute even though you don't directly require its result. + +As an example, consider MIR const qualification. It wants to read the +result produced by the `mir_const()` suite. However, that result will +be **stolen** by the `mir_validated()` suite. If nothing was done, +then `mir_const_qualif(D)` would succeed if it came before +`mir_validated(D)`, but fail otherwise. Therefore, `mir_validated(D)` +will **force** `mir_const_qualif` before it actually steals, thus +ensuring that the reads have already happened: + +``` +mir_const(D) --read-by--> mir_const_qualif(D) + | ^ + stolen-by | + | (forces) + v | +mir_validated(D) ------------+ +``` + +### Implementing and registering a pass + +To create a new MIR pass, you simply implement the `MirPass` trait for +some fresh singleton type `Foo`. Once you have implemented a trait for +your type `Foo`, you then have to insert `Foo` into one of the suites; +this is done in `librustc_driver/driver.rs` by invoking `push_pass(S, +Foo)` with the appropriate suite substituted for `S`. + + +[mir]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir +[mirmanip]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir +[mir]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir From d39ba39a24feecad5d2fba6a1bf49310a316f2a1 Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Sat, 27 Jan 2018 14:30:07 +0800 Subject: [PATCH 048/648] Added the mdbook-linkcheck backend --- book.toml | 3 +++ src/high-level-overview.md | 4 +++- src/ty.md | 2 +- 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/book.toml b/book.toml index 485a12ca3..3c4c537e8 100644 --- a/book.toml +++ b/book.toml @@ -3,3 +3,6 @@ title = "Guide to Rustc Development" author = "Rustc developers" description = "A guide to developing rustc " +[output.html] + +[output.linkcheck] diff --git a/src/high-level-overview.md b/src/high-level-overview.md index 55b596a2a..7da9b8ca1 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -43,7 +43,7 @@ The `rustc_driver` crate, at the top of this lattice, is effectively the "main" function for the rust compiler. It doesn't have much "real code", but instead ties together all of the code defined in the other crates and defines the overall flow of execution. (As we transition -more and more to the [query model](ty/maps/README.md), however, the +more and more to the [query model], however, the "flow" of compilation is becoming less centrally defined.) At the other extreme, the `rustc` crate defines the common and @@ -134,3 +134,5 @@ take: (one for each "codegen unit"). 6. **Linking** - Finally, those `.o` files are linked together. + +[query model]: https://github.com/rust-lang/rust/blob/master/src/librustc/ty/maps/README.md diff --git a/src/ty.md b/src/ty.md index 8debb71c7..906e99e71 100644 --- a/src/ty.md +++ b/src/ty.md @@ -78,7 +78,7 @@ is in fact a simple type alias for a reference with `'tcx` lifetime: pub type Ty<'tcx> = &'tcx TyS<'tcx>; ``` -[the HIR]: ../hir/README.md +[the HIR]: https://github.com/rust-lang/rust/blob/master/src/librustc/hir/README.md You can basically ignore the `TyS` struct -- you will basically never access it explicitly. We always pass it by reference using the From 8b47122474ab912abca3fcb5acc8bd4cf3266e33 Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Sat, 27 Jan 2018 14:36:23 +0800 Subject: [PATCH 049/648] Updated CI to install and use mdbook-linkcheck --- ci/install.sh | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/ci/install.sh b/ci/install.sh index 2d32caaa9..81cbd8fb7 100644 --- a/ci/install.sh +++ b/ci/install.sh @@ -1,12 +1,20 @@ #!/bin/bash set -ex -if command -v mdbook >/dev/null 2>&1; then - echo "mdbook already installed at $(command -v mdbook)" -else - echo "installing mdbook" - cargo install mdbook --vers "0.0.28" -fi +function cargo_install() { + local name=$1 + local version=$2 + + if command -v $name >/dev/null 2>&1; then + echo "$name is already installed at $(command -v $name)" + else + echo "Installing $name" + cargo install $name --version $version + fi +} + +cargo_install mdbook 0.1.1 +cargo_install mdbook-linkcheck 0.1.0 if command -v ghp-import >/dev/null 2>&1; then echo "ghp-import already installed at $(which ghp-import)" From acf36f823e2b7eb6605dffcde973f17c35662e6e Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Sat, 27 Jan 2018 14:41:40 +0800 Subject: [PATCH 050/648] Removed ghp-import and run mdbook build unconditionally --- .travis.yml | 5 +---- ci/github_pages.sh | 11 ----------- ci/install.sh | 7 ------- 3 files changed, 1 insertion(+), 22 deletions(-) delete mode 100644 ci/github_pages.sh diff --git a/.travis.yml b/.travis.yml index 2d107794f..0862ff9bb 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,14 +1,11 @@ language: rust cache: -- pip - cargo install: - source ~/.cargo/env || true - bash ci/install.sh script: -- true -after_success: -- bash ci/github_pages.sh +- RUST_LOG=debug mdbook build notifications: email: on_success: never diff --git a/ci/github_pages.sh b/ci/github_pages.sh deleted file mode 100644 index ffd89ad52..000000000 --- a/ci/github_pages.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash -set -ex - -BOOK_DIR=book - -# Only upload the built book to github pages if it's a commit to master -if [ "$TRAVIS_BRANCH" = master -a "$TRAVIS_PULL_REQUEST" = false ]; then - mdbook build -else - echo Skipping 'mdbook build' because this is not master or this is just a PR. -fi diff --git a/ci/install.sh b/ci/install.sh index 81cbd8fb7..99c85986c 100644 --- a/ci/install.sh +++ b/ci/install.sh @@ -15,10 +15,3 @@ function cargo_install() { cargo_install mdbook 0.1.1 cargo_install mdbook-linkcheck 0.1.0 - -if command -v ghp-import >/dev/null 2>&1; then - echo "ghp-import already installed at $(which ghp-import)" -else - echo "installing ghp-import" - pip install --user ghp-import -fi From 79052b4acaf14d45f00440e10fd2658b30c7bb55 Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Sat, 27 Jan 2018 14:53:46 +0800 Subject: [PATCH 051/648] Added a note about how to use mdbook-linkcheck --- README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/README.md b/README.md index 5ff51de2b..352c4ce0b 100644 --- a/README.md +++ b/README.md @@ -25,3 +25,11 @@ for you to talk with someone who **does** know the code, or who wants to pair with you and figure it out. Then you can work on writing up what you learned. +To help prevent accidentally introducing broken links, we use the +`mdbook-linkcheck`. If installed on your machine `mdbook` will automatically +invoke this link checker, otherwise it will emit a warning saying it couldn't +be found. + +``` +$ cargo install mdbook-linkcheck +``` From 66d2008ac082049f6214eb50af45a15b994762a0 Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Sat, 27 Jan 2018 14:54:56 +0800 Subject: [PATCH 052/648] Toned down the log verbosity --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 0862ff9bb..f17a947d4 100644 --- a/.travis.yml +++ b/.travis.yml @@ -5,7 +5,7 @@ install: - source ~/.cargo/env || true - bash ci/install.sh script: -- RUST_LOG=debug mdbook build +- mdbook build notifications: email: on_success: never From 046dc79b9de938b5007544b2d3fb61a6f3e7b7f6 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 31 Jan 2018 11:30:17 -0500 Subject: [PATCH 053/648] update to link within the book --- src/high-level-overview.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/high-level-overview.md b/src/high-level-overview.md index 7da9b8ca1..519d822e2 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -135,4 +135,5 @@ take: 6. **Linking** - Finally, those `.o` files are linked together. -[query model]: https://github.com/rust-lang/rust/blob/master/src/librustc/ty/maps/README.md + +[query model]: query.html From 0cedd25595e5fd7e531b6ed9035a6b60d55e240a Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 31 Jan 2018 11:31:43 -0500 Subject: [PATCH 054/648] Update link to hir --- src/ty.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ty.md b/src/ty.md index 906e99e71..e29ecb5e3 100644 --- a/src/ty.md +++ b/src/ty.md @@ -78,7 +78,7 @@ is in fact a simple type alias for a reference with `'tcx` lifetime: pub type Ty<'tcx> = &'tcx TyS<'tcx>; ``` -[the HIR]: https://github.com/rust-lang/rust/blob/master/src/librustc/hir/README.md +[the HIR]: ./hir.html You can basically ignore the `TyS` struct -- you will basically never access it explicitly. We always pass it by reference using the From 04cee3e1be2c2151524441bd13ea4009f54fb566 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 31 Jan 2018 14:13:49 -0500 Subject: [PATCH 055/648] fix some broken links --- src/macro-expansion.md | 2 +- src/type-inference.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/macro-expansion.md b/src/macro-expansion.md index a7777e80c..ee4bd322c 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -158,4 +158,4 @@ TODO [code_mp]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt/macro_parser.rs [code_mp]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt/macro_rules.rs [code_parse_int]: https://github.com/rust-lang/rust/blob/a97cd17f5d71fb4ec362f4fbd79373a6e7ed7b82/src/libsyntax/ext/tt/macro_parser.rs#L421 -[parsing]: ./the-parser.md +[parsing]: ./the-parser.html diff --git a/src/type-inference.md b/src/type-inference.md index 6e2032fee..feb694196 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -32,7 +32,7 @@ fresh types and things that it will create, as described in [the README in the ty module][ty-readme]. This arena is created by the `enter` function and disposed after it returns. -[ty-readme]: src/librustc/ty/README.md +[ty-readme]: ty.html Within the closure, the infcx will have the type `InferCtxt<'cx, 'gcx, 'tcx>` for some fresh `'cx` and `'tcx` -- the latter corresponds to @@ -107,7 +107,7 @@ actual return type is not `()`, but rather `InferOk<()>`. The to ensure that these are fulfilled (typically by enrolling them in a fulfillment context). See the [trait README] for more background here. -[trait README]: ../traits/README.md +[trait README]: trait-resolution.html You can also enforce subtyping through `infcx.at(..).sub(..)`. The same basic concepts apply as above. From 4066211f4714aa4d41ccb03fcec550dfba2a0ce1 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 31 Jan 2018 14:18:51 -0500 Subject: [PATCH 056/648] update wording to be a bit more clear --- src/how-to-build-and-run.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 24cfbfad7..7992dc987 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -54,12 +54,16 @@ compiler to compile the newer version. In particular, the newer version of the compiler, `libstd`, and other tooling may use some unstable features internally. The result is the compiling `rustc` is done in stages. -- Stage 0: the current _beta_ compiler is compiled using the current _stable_ compiler. -- Stage 1: the code in your clone is then compiled with the stage 0 compiler. -- Stage 2: the code in your clone is then compiled with the stage 1 compiler (i.e. it builds itself). - -For hacking, often building the stage 1 compiler is enough, but for testing and -release, the stage 2 compiler is used. +- **Stage 0:** the stage0 compiler is the current _beta_ compiler; we + download this binary from the internet. +- **Stage 1:** the code in your clone is then compiled with the stage + 0 compiler to produce the stage 1 compiler. +- **Stage 2:** the code in your clone is then compiled with the stage + 1 compiler *again* to produce the stage 2 compiler (i.e. it builds + itself). + +For hacking, often building the stage 1 compiler is enough, but for +final testing and release, the stage 2 compiler is used. Once you've created a config.toml, you are now ready to run `x.py`. There are a lot of options here, but let's start with what is From ddad2c3e95fa4e8d15867a6192ab6052527aba59 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Thu, 1 Feb 2018 03:47:39 +0000 Subject: [PATCH 057/648] fixed definition of code_mr hyperlink --- src/macro-expansion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/macro-expansion.md b/src/macro-expansion.md index ee4bd322c..8f94d6a0a 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -156,6 +156,6 @@ TODO [code_dir]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt [code_mp]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt/macro_parser.rs -[code_mp]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt/macro_rules.rs +[code_mr]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt/macro_rules.rs [code_parse_int]: https://github.com/rust-lang/rust/blob/a97cd17f5d71fb4ec362f4fbd79373a6e7ed7b82/src/libsyntax/ext/tt/macro_parser.rs#L421 [parsing]: ./the-parser.html From 9ad3a69332a24e4c98a1c978ba539f921d52d8bf Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Thu, 1 Feb 2018 03:50:28 +0000 Subject: [PATCH 058/648] replaced all instances of `--` (double hyphen) with `-` (en-dash) --- src/high-level-overview.md | 8 ++++---- src/hir.md | 6 +++--- src/how-to-build-and-run.md | 10 +++++----- src/macro-expansion.md | 2 +- src/mir.md | 10 +++++----- src/query.md | 4 ++-- src/trait-resolution.md | 18 +++++++++--------- src/ty.md | 12 ++++++------ src/type-inference.md | 12 ++++++------ 9 files changed, 41 insertions(+), 41 deletions(-) diff --git a/src/high-level-overview.md b/src/high-level-overview.md index 519d822e2..50ed072b8 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -13,7 +13,7 @@ many more. The source for each crate can be found in a directory like `src/libXXX`, where `XXX` is the crate name. (NB. The names and divisions of these crates are not set in -stone and may change over time -- for the time being, we tend towards +stone and may change over time – for the time being, we tend towards a finer-grained division to help with compilation time, though as incremental improves that may change.) @@ -53,7 +53,7 @@ also contains some amount of the compiler itself, although that is relatively limited. Finally, all the crates in the bulge in the middle define the bulk of -the compiler -- they all depend on `rustc`, so that they can make use +the compiler – they all depend on `rustc`, so that they can make use of the various types defined there, and they export public routines that `rustc_driver` will invoke as needed (more and more, what these crates export are "query definitions", but those are covered later @@ -117,9 +117,9 @@ take: - An important step in processing the HIR is to perform type checking. This process assigns types to every HIR expression, for example, and also is responsible for resolving some - "type-dependent" paths, such as field accesses (`x.f` -- we + "type-dependent" paths, such as field accesses (`x.f` – we can't know what field `f` is being accessed until we know the - type of `x`) and associated type references (`T::Item` -- we + type of `x`) and associated type references (`T::Item` – we can't know what type `Item` is until we know what `T` is). - Type checking creates "side-tables" (`TypeckTables`) that include the types of expressions, the way to resolve methods, and so forth. diff --git a/src/hir.md b/src/hir.md index 5d5e273c4..52fda69a6 100644 --- a/src/hir.md +++ b/src/hir.md @@ -1,6 +1,6 @@ # The HIR -The HIR -- "High-level IR" -- is the primary IR used in most of +The HIR – "High-level IR" – is the primary IR used in most of rustc. It is a desugared version of the "abstract syntax tree" (AST) that is generated after parsing, macro expansion, and name resolution have completed. Many parts of HIR resemble Rust surface syntax quite @@ -91,7 +91,7 @@ with a HIR node. For example, if you have a `DefId`, and you would like to convert it to a `NodeId`, you can use `tcx.hir.as_local_node_id(def_id)`. This -returns an `Option` -- this will be `None` if the def-id +returns an `Option` – this will be `None` if the def-id refers to something outside of the current crate (since then it has no HIR node), but otherwise returns `Some(n)` where `n` is the node-id of the definition. @@ -100,7 +100,7 @@ Similarly, you can use `tcx.hir.find(n)` to lookup the node for a `NodeId`. This returns a `Option>`, where `Node` is an enum defined in the map; by matching on this you can find out what sort of node the node-id referred to and also get a pointer to the data -itself. Often, you know what sort of node `n` is -- e.g., if you know +itself. Often, you know what sort of node `n` is – e.g., if you know that `n` must be some HIR expression, you can do `tcx.hir.expect_expr(n)`, which will extract and return the `&hir::Expr`, panicking if `n` is not in fact an expression. diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 7992dc987..2657da84c 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -129,9 +129,9 @@ LLVM version: 4.0 Here are a few other useful x.py commands. We'll cover some of them in detail in other sections: - Building things: - - `./x.py clean` -- clean up the build directory (`rm -rf build` works too, but then you have to rebuild LLVM) - - `./x.py build --stage 1` -- builds everything using the stage 1 compiler, not just up to libstd - - `./x.py build` -- builds the stage2 compiler + - `./x.py clean` – clean up the build directory (`rm -rf build` works too, but then you have to rebuild LLVM) + - `./x.py build --stage 1` – builds everything using the stage 1 compiler, not just up to libstd + - `./x.py build` – builds the stage2 compiler - Running tests (see the section [running tests](./running-tests.html) for more details): - - `./x.py test --stage 1 src/libstd` -- runs the `#[test]` tests from libstd - - `./x.py test --stage 1 src/test/run-pass` -- runs the `run-pass` test suite + - `./x.py test --stage 1 src/libstd` – runs the `#[test]` tests from libstd + - `./x.py test --stage 1 src/test/run-pass` – runs the `run-pass` test suite diff --git a/src/macro-expansion.md b/src/macro-expansion.md index 8f94d6a0a..55a550b56 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -33,7 +33,7 @@ a tree of _tokens_. A _token_ is a single "unit" of the grammar, such as an identifier (e.g., `foo`) or punctuation (e.g., `=>`). There are also other special tokens, such as `EOF`, which indicates that there are no more tokens. Token trees resulting from paired parentheses-like characters (`(`...`)`, -`[`...`]`, and `{`...`}`) -- they include the open and close and all the tokens +`[`...`]`, and `{`...`}`) – they include the open and close and all the tokens in between (we do require that parentheses-like characters be balanced). Having macro expansion operate on token streams rather than the raw bytes of a source file abstracts away a lot of complexity. The macro expander (and much of the diff --git a/src/mir.md b/src/mir.md index dbac8eb53..34f6bbb8f 100644 --- a/src/mir.md +++ b/src/mir.md @@ -49,17 +49,17 @@ query. Each suite consists of multiple optimizations and transformations. These suites represent useful intermediate points where we want to access the MIR for type checking or other purposes: -- `mir_build(D)` -- not a query, but this constructs the initial MIR -- `mir_const(D)` -- applies some simple transformations to make MIR ready for constant evaluation; -- `mir_validated(D)` -- applies some more transformations, making MIR ready for borrow checking; -- `optimized_mir(D)` -- the final state, after all optimizations have been performed. +- `mir_build(D)` – not a query, but this constructs the initial MIR +- `mir_const(D)` – applies some simple transformations to make MIR ready for constant evaluation; +- `mir_validated(D)` – applies some more transformations, making MIR ready for borrow checking; +- `optimized_mir(D)` – the final state, after all optimizations have been performed. ### Stealing The intermediate queries `mir_const()` and `mir_validated()` yield up a `&'tcx Steal>`, allocated using `tcx.alloc_steal_mir()`. This indicates that the result may be -**stolen** by the next suite of optimizations -- this is an +**stolen** by the next suite of optimizations – this is an optimization to avoid cloning the MIR. Attempting to use a stolen result will cause a panic in the compiler. Therefore, it is important that you do not read directly from these intermediate queries except as diff --git a/src/query.md b/src/query.md index 65d651307..fa17a3850 100644 --- a/src/query.md +++ b/src/query.md @@ -11,7 +11,7 @@ it to you. [hl]: high-level-overview.html -Query execution is **memoized** -- so the first time you invoke a +Query execution is **memoized** – so the first time you invoke a query, it will go do the computation, but the next time, the result is returned from a hashtable. Moreover, query execution fits nicely into **incremental computation**; the idea is roughly that, when you do a @@ -98,7 +98,7 @@ message"`. This is basically just a precaution in case you are wrong. So you may be wondering what happens when you invoke a query method. The answer is that, for each query, the compiler maintains a -cache -- if your query has already been executed, then, the answer is +cache – if your query has already been executed, then, the answer is simple: we clone the return value out of the cache and return it (therefore, you should try to ensure that the return types of queries are cheaply cloneable; insert a `Rc` if necessary). diff --git a/src/trait-resolution.md b/src/trait-resolution.md index 58efbd050..63f72b014 100644 --- a/src/trait-resolution.md +++ b/src/trait-resolution.md @@ -73,16 +73,16 @@ resolved and, if so, how it is to be resolved (via impl, where clause, etc). The main interface is the `select()` function, which takes an obligation and returns a `SelectionResult`. There are three possible outcomes: -- `Ok(Some(selection))` -- yes, the obligation can be resolved, and +- `Ok(Some(selection))` – yes, the obligation can be resolved, and `selection` indicates how. If the impl was resolved via an impl, then `selection` may also indicate nested obligations that are required by the impl. -- `Ok(None)` -- we are not yet sure whether the obligation can be +- `Ok(None)` – we are not yet sure whether the obligation can be resolved or not. This happens most commonly when the obligation contains unbound type variables. -- `Err(err)` -- the obligation definitely cannot be resolved due to a +- `Err(err)` – the obligation definitely cannot be resolved due to a type error, or because there are no impls that could possibly apply, etc. @@ -95,7 +95,7 @@ Searches for impls/where-clauses/etc that might possibly be used to satisfy the obligation. Each of those is called a candidate. To avoid ambiguity, we want to find exactly one candidate that is definitively applicable. In some cases, we may not -know whether an impl/where-clause applies or not -- this occurs when +know whether an impl/where-clause applies or not – this occurs when the obligation contains unbound inference variables. The basic idea for candidate assembly is to do a first pass in which @@ -172,11 +172,11 @@ impl Get for Box { ``` What happens when we invoke `get_it(&box 1_u16)`, for example? In this -case, the `Self` type is `Box` -- that unifies with both impls, +case, the `Self` type is `Box` – that unifies with both impls, because the first applies to all types, and the second to all boxes. In the olden days we'd have called this ambiguous. But what we do now is do a second *winnowing* pass that considers where clauses -and attempts to remove candidates -- in this case, the first impl only +and attempts to remove candidates – in this case, the first impl only applies if `Box : Copy`, which doesn't hold. After winnowing, then, we are left with just one candidate, so we can proceed. There is a test of this in `src/test/run-pass/traits-conditional-dispatch.rs`. @@ -326,7 +326,7 @@ to a `TraitRef`. We would then create the `TraitRef` from the impl, using fresh variables for it's bound regions (and thus getting `Foo<&'$a isize>`, where `'$a` is the inference variable for `'a`). Next we relate the two trait refs, yielding a graph with the constraint -that `'0 == '$a`. Finally, we check for skolemization "leaks" -- a +that `'0 == '$a`. Finally, we check for skolemization "leaks" – a leak is basically any attempt to relate a skolemized region to another skolemized region, or to any region that pre-existed the impl match. The leak check is done by searching from the skolemized region to find @@ -457,7 +457,7 @@ and the graph is consulted when propagating defaults down the specialization hierarchy. You might expect that the specialization graph would be used during -selection -- i.e., when actually performing specialization. This is +selection – i.e., when actually performing specialization. This is not done for two reasons: - It's merely an optimization: given a set of candidates that apply, @@ -476,7 +476,7 @@ not done for two reasons: Trait impl selection can succeed even when multiple impls can apply, as long as they are part of the same specialization family. In that -case, it returns a *single* impl on success -- this is the most +case, it returns a *single* impl on success – this is the most specialized impl *known* to apply. However, if there are any inference variables in play, the returned impl may not be the actual impl we will use at trans time. Thus, we take special care to avoid projecting diff --git a/src/ty.md b/src/ty.md index e29ecb5e3..0732e88bd 100644 --- a/src/ty.md +++ b/src/ty.md @@ -80,9 +80,9 @@ pub type Ty<'tcx> = &'tcx TyS<'tcx>; [the HIR]: ./hir.html -You can basically ignore the `TyS` struct -- you will basically never +You can basically ignore the `TyS` struct – you will basically never access it explicitly. We always pass it by reference using the -`Ty<'tcx>` alias -- the only exception I think is to define inherent +`Ty<'tcx>` alias – the only exception I think is to define inherent methods on types. Instances of `TyS` are only ever allocated in one of the rustc arenas (never e.g. on the stack). @@ -115,7 +115,7 @@ of type variants. For example: let array_ty = tcx.mk_array(elem_ty, len * 2); ``` -These methods all return a `Ty<'tcx>` -- note that the lifetime you +These methods all return a `Ty<'tcx>` – note that the lifetime you get back is the lifetime of the innermost arena that this `tcx` has access to. In fact, types are always canonicalized and interned (so we never allocate exactly the same type twice) and are always allocated @@ -125,7 +125,7 @@ allocated in the global arena). However, the lifetime `'tcx` is always a safe approximation, so that is what you get back. > NB. Because types are interned, it is possible to compare them for -> equality efficiently using `==` -- however, this is almost never what +> equality efficiently using `==` – however, this is almost never what > you want to do unless you happen to be hashing and looking for > duplicates. This is because often in Rust there are multiple ways to > represent the same type, particularly once inference is involved. If @@ -141,10 +141,10 @@ In addition to types, there are a number of other arena-allocated data structures that you can allocate, and which are found in this module. Here are a few examples: -- `Substs`, allocated with `mk_substs` -- this will intern a slice of types, often used to +- `Substs`, allocated with `mk_substs` – this will intern a slice of types, often used to specify the values to be substituted for generics (e.g., `HashMap` would be represented as a slice `&'tcx [tcx.types.i32, tcx.types.u32]`). -- `TraitRef`, typically passed by value -- a **trait reference** +- `TraitRef`, typically passed by value – a **trait reference** consists of a reference to a trait along with its various type parameters (including `Self`), like `i32: Display` (here, the def-id would reference the `Display` trait, and the substs would contain diff --git a/src/type-inference.md b/src/type-inference.md index feb694196..070b7bb4c 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -35,7 +35,7 @@ function and disposed after it returns. [ty-readme]: ty.html Within the closure, the infcx will have the type `InferCtxt<'cx, 'gcx, -'tcx>` for some fresh `'cx` and `'tcx` -- the latter corresponds to +'tcx>` for some fresh `'cx` and `'tcx` – the latter corresponds to the lifetime of this temporary arena, and the `'cx` is the lifetime of the `InferCtxt` itself. (Again, see [that ty README][ty-readme] for more details on this setup.) @@ -47,7 +47,7 @@ created. See `InferCtxtBuilder` for more information. ## Inference variables The main purpose of the inference context is to house a bunch of -**inference variables** -- these represent types or regions whose precise +**inference variables** – these represent types or regions whose precise value is not yet known, but will be uncovered as we perform type-checking. If you're familiar with the basic ideas of unification from H-M type @@ -95,15 +95,15 @@ doing this unification, and in what environment, and the `eq` method performs the actual equality constraint. When you equate things, you force them to be precisely equal. Equating -returns a `InferResult` -- if it returns `Err(err)`, then equating +returns a `InferResult` – if it returns `Err(err)`, then equating failed, and the enclosing `TypeError` will tell you what went wrong. The success case is perhaps more interesting. The "primary" return -type of `eq` is `()` -- that is, when it succeeds, it doesn't return a +type of `eq` is `()` – that is, when it succeeds, it doesn't return a value of any particular interest. Rather, it is executed for its side-effects of constraining type variables and so forth. However, the actual return type is not `()`, but rather `InferOk<()>`. The -`InferOk` type is used to carry extra trait obligations -- your job is +`InferOk` type is used to carry extra trait obligations – your job is to ensure that these are fulfilled (typically by enrolling them in a fulfillment context). See the [trait README] for more background here. @@ -117,7 +117,7 @@ basic concepts apply as above. Sometimes you would like to know if it is *possible* to equate two types without error. You can test that with `infcx.can_eq` (or `infcx.can_sub` for subtyping). If this returns `Ok`, then equality -is possible -- but in all cases, any side-effects are reversed. +is possible – but in all cases, any side-effects are reversed. Be aware though that the success or failure of these methods is always **modulo regions**. That is, two types `&'a u32` and `&'b u32` will From 96d3038965af3259d09576c8feb2dd3294b19d78 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Thu, 1 Feb 2018 03:57:51 +0000 Subject: [PATCH 059/648] minor grammatical fixes --- src/macro-expansion.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/macro-expansion.md b/src/macro-expansion.md index 55a550b56..a43c09394 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -134,8 +134,9 @@ it calls back to the normal Rust parser to get the contents of that non-terminal. In this case, the Rust parser would look for an `ident` token, which it finds (`foo`) and returns to the macro parser. Then, the macro parser proceeds in parsing as normal. Also, note that exactly one of the matchers from -the various arms should match the invocation (otherwise, the macro is -ambiguous). +the various arms should match the invocation; if there is more than one match, +the parse is ambiguous, while if there are no matches at all, there is a syntax +error. For more information about the macro parser's implementation, see the comments in [`src/libsyntax/ext/tt/macro_parser.rs`][code_mp]. From e022bcd2141ba47394c12980a0fef1f67091221c Mon Sep 17 00:00:00 2001 From: avborhanian Date: Sat, 3 Feb 2018 00:51:25 -0500 Subject: [PATCH 060/648] Adding ICH to the glossary. --- src/glossary.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/glossary.md b/src/glossary.md index 3202e5f4c..186e1a4c7 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -14,6 +14,7 @@ HirId | identifies a particular node in the HIR by combining 'gcx | the lifetime of the global arena ([see more](ty.html)) generics | the set of generic type parameters defined on a type or item ICE | internal compiler error. When the compiler crashes. +ICH | incremental compilation hash. infcx | the inference context (see `librustc/infer`) MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) From 4a72994f113b8fbd65343150e55ba33db3bd92aa Mon Sep 17 00:00:00 2001 From: avborhanian Date: Sun, 4 Feb 2018 13:46:29 -0500 Subject: [PATCH 061/648] Update glossary.md --- src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/glossary.md b/src/glossary.md index 186e1a4c7..ee27362af 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -14,7 +14,7 @@ HirId | identifies a particular node in the HIR by combining 'gcx | the lifetime of the global arena ([see more](ty.html)) generics | the set of generic type parameters defined on a type or item ICE | internal compiler error. When the compiler crashes. -ICH | incremental compilation hash. +ICH | incremental compilation hash. ICHs are used as fingerprints for things such as HIR and crate metadata, to check if changes have been made. infcx | the inference context (see `librustc/infer`) MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) From 1d9eeb079f46753ed451d174a554456f50c40d41 Mon Sep 17 00:00:00 2001 From: avborhanian Date: Sun, 4 Feb 2018 13:54:45 -0500 Subject: [PATCH 062/648] Update glossary.md --- src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/glossary.md b/src/glossary.md index ee27362af..10c2d4535 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -14,7 +14,7 @@ HirId | identifies a particular node in the HIR by combining 'gcx | the lifetime of the global arena ([see more](ty.html)) generics | the set of generic type parameters defined on a type or item ICE | internal compiler error. When the compiler crashes. -ICH | incremental compilation hash. ICHs are used as fingerprints for things such as HIR and crate metadata, to check if changes have been made. +ICH | incremental compilation hash. ICHs are used as fingerprints for things such as HIR and crate metadata, to check if changes have been made. This is useful in incremental compilation to see if part of a crate has changed and should be recompiled. infcx | the inference context (see `librustc/infer`) MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) From dc037e216bf4cb9daf588d131b5abc854e538d17 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Sun, 4 Feb 2018 16:11:19 -0600 Subject: [PATCH 063/648] Update README.md --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 352c4ce0b..7e468fbb0 100644 --- a/README.md +++ b/README.md @@ -33,3 +33,5 @@ be found. ``` $ cargo install mdbook-linkcheck ``` +You will need `mdbook` version `>= 0.1`. `linkcheck` will be run automatically +when you run `mdbook build`. From c32587aaed2086e49ca58b1ef9372b2c83e84f8e Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Sun, 4 Feb 2018 18:37:37 +0000 Subject: [PATCH 064/648] Renamed `trans: generating LLVM IR` section. --- src/SUMMARY.md | 2 +- src/trans.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 8e18969a1..8f128f648 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -19,5 +19,5 @@ - [MIR construction](./mir-construction.md) - [MIR borrowck](./mir-borrowck.md) - [MIR optimizations](./mir-optimizations.md) -- [trans: generating LLVM IR](./trans.md) +- [The `trans` crate: generating LLVM IR](./trans.md) - [Glossary](./glossary.md) diff --git a/src/trans.md b/src/trans.md index 7092b7a0b..efff52ef9 100644 --- a/src/trans.md +++ b/src/trans.md @@ -1 +1 @@ -# trans: generating LLVM IR +# The `trans` crate: generating LLVM IR From 1a399f5ea3c861b7ae3d282f621eabad59e34c19 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Sun, 4 Feb 2018 18:48:38 +0000 Subject: [PATCH 065/648] Improved grammar of HIR section. --- src/hir.md | 89 ++++++++++++++++++++++++++---------------------------- 1 file changed, 43 insertions(+), 46 deletions(-) diff --git a/src/hir.md b/src/hir.md index 52fda69a6..ac7690bd5 100644 --- a/src/hir.md +++ b/src/hir.md @@ -1,12 +1,11 @@ # The HIR -The HIR – "High-level IR" – is the primary IR used in most of -rustc. It is a desugared version of the "abstract syntax tree" (AST) -that is generated after parsing, macro expansion, and name resolution -have completed. Many parts of HIR resemble Rust surface syntax quite -closely, with the exception that some of Rust's expression forms have -been desugared away (as an example, `for` loops are converted into a -`loop` and do not appear in the HIR). +The HIR – "High-level IR" – is the primary IR used in most of rustc. +It is a desugared version of the "abstract syntax tree" (AST) that is generated +after parsing, macro expansion, and name resolution have completed. Many parts +of HIR resemble Rust surface syntax quite closely, with the exception that some +of Rust's expression forms have been desugared away (as an example, `for` loops +are converted into a `loop` and do not appear in the HIR). This chapter covers the main concepts of the HIR. @@ -21,8 +20,8 @@ serve to organize the content of the crate for easier access. For example, the contents of individual items (e.g., modules, functions, traits, impls, etc) in the HIR are not immediately -accessible in the parents. So, for example, if had a module item `foo` -containing a function `bar()`: +accessible in the parents. So, for example, if there is a module item +`foo` containing a function `bar()`: ``` mod foo { @@ -30,25 +29,25 @@ mod foo { } ``` -Then in the HIR the representation of module `foo` (the `Mod` -stuct) would have only the **`ItemId`** `I` of `bar()`. To get the +then in the HIR the representation of module `foo` (the `Mod` +stuct) would only have the **`ItemId`** `I` of `bar()`. To get the details of the function `bar()`, we would lookup `I` in the `items` map. One nice result from this representation is that one can iterate over all items in the crate by iterating over the key-value pairs -in these maps (without the need to trawl through the IR in total). +in these maps (without the need to trawl through the whole HIR). There are similar maps for things like trait items and impl items, as well as "bodies" (explained below). -The other reason to setup the representation this way is for better +The other reason to set up the representation this way is for better integration with incremental compilation. This way, if you gain access -to a `&hir::Item` (e.g. for the mod `foo`), you do not immediately +to an `&hir::Item` (e.g. for the mod `foo`), you do not immediately gain access to the contents of the function `bar()`. Instead, you only gain access to the **id** for `bar()`, and you must invoke some -function to lookup the contents of `bar()` given its id; this gives us -a chance to observe that you accessed the data for `bar()` and record -the dependency. +function to lookup the contents of `bar()` given its id; this gives the +compiler a chance to observe that you accessed the data for `bar()`, +and then record the dependency. ### Identifiers in the HIR @@ -57,37 +56,35 @@ carry around references into the HIR, but rather to carry around *identifier numbers* (or just "ids"). Right now, you will find four sorts of identifiers in active use: -- `DefId`, which primarily names "definitions" or top-level items. - - You can think of a `DefId` as being shorthand for a very explicit - and complete path, like `std::collections::HashMap`. However, - these paths are able to name things that are not nameable in - normal Rust (e.g., impls), and they also include extra information - about the crate (such as its version number, as two versions of - the same crate can co-exist). - - A `DefId` really consists of two parts, a `CrateNum` (which - identifies the crate) and a `DefIndex` (which indixes into a list - of items that is maintained per crate). -- `HirId`, which combines the index of a particular item with an - offset within that item. - - the key point of a `HirId` is that it is *relative* to some item (which is named - via a `DefId`). -- `BodyId`, this is an absolute identifier that refers to a specific - body (definition of a function or constant) in the crate. It is currently - effectively a "newtype'd" `NodeId`. -- `NodeId`, which is an absolute id that identifies a single node in the HIR tree. +- `DefId` – primarily names "definitions" or top-level items. + - You can think of a `DefId` as shorthand for a very explicit and complete + path, like `std::collections::HashMap`. However, these paths are able to + name things that are not nameable in normal Rust (e.g. impls), and they also + include extra information about the crate (such as its version number, since + two versions of the same crate can co-exist). + - A `DefId` really consists of two parts, a `CrateNum` (which identifies the + crate) and a `DefIndex` (which indexes into a list of items that is + maintained per crate). +- `HirId` – combines the index of a particular item with an offset within + that item. + - The key point of an `HirId` is that it is *relative* to some item (which is + named via a `DefId`). +- `BodyId` – an absolute identifier that refers to a specific body (definition + of a function or constant) in the crate. It is currently effectively a + "newtype'd" `NodeId`. +- `NodeId` – an absolute ID that identifies a single node in the HIR tree. - While these are still in common use, **they are being slowly phased out**. - - Since they are absolute within the crate, adding a new node - anywhere in the tree causes the node-ids of all subsequent code in - the crate to change. This is terrible for incremental compilation, - as you can perhaps imagine. + - Since they are absolute within the crate, adding a new node anywhere in the + tree causes the `NodeId`s of all subsequent code in the crate to change. + This is terrible for incremental compilation, as you can perhaps imagine. -### HIR Map +### The HIR Map Most of the time when you are working with the HIR, you will do so via the **HIR Map**, accessible in the tcx via `tcx.hir` (and defined in the `hir::map` module). The HIR map contains a number of methods to -convert between ids of various kinds and to lookup data associated -with a HIR node. +convert between IDs of various kinds and to lookup data associated +with an HIR node. For example, if you have a `DefId`, and you would like to convert it to a `NodeId`, you can use `tcx.hir.as_local_node_id(def_id)`. This @@ -100,7 +97,7 @@ Similarly, you can use `tcx.hir.find(n)` to lookup the node for a `NodeId`. This returns a `Option>`, where `Node` is an enum defined in the map; by matching on this you can find out what sort of node the node-id referred to and also get a pointer to the data -itself. Often, you know what sort of node `n` is – e.g., if you know +itself. Often, you know what sort of node `n` is – e.g. if you know that `n` must be some HIR expression, you can do `tcx.hir.expect_expr(n)`, which will extract and return the `&hir::Expr`, panicking if `n` is not in fact an expression. @@ -113,7 +110,7 @@ calls like `tcx.hir.get_parent_node(n)`. A **body** represents some kind of executable code, such as the body of a function/closure or the definition of a constant. Bodies are associated with an **owner**, which is typically some kind of item -(e.g., a `fn()` or `const`), but could also be a closure expression -(e.g., `|x, y| x + y`). You can use the HIR map to find the body -associated with a given def-id (`maybe_body_owned_by()`) or to find +(e.g. an `fn()` or `const`), but could also be a closure expression +(e.g. `|x, y| x + y`). You can use the HIR map to find the body +associated with a given `DefId` (`maybe_body_owned_by()`) or to find the owner of a body (`body_owner_def_id()`). From 7d7d9171b8be728783453c02b98a5b2a11fdfd7f Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Sun, 4 Feb 2018 18:49:41 +0000 Subject: [PATCH 066/648] Changed all instances of `e.g.,` to `e.g.`, and similar. --- CODE_OF_CONDUCT.md | 4 ++-- src/glossary.md | 2 +- src/high-level-overview.md | 6 +++--- src/hir.md | 2 +- src/macro-expansion.md | 2 +- src/query.md | 4 ++-- src/trait-resolution.md | 2 +- src/type-inference.md | 2 +- 8 files changed, 12 insertions(+), 12 deletions(-) diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index d42476bc4..89a9cdfcc 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -23,8 +23,8 @@ These are the policies for upholding our community's standards of conduct. If yo 1. Remarks that violate the Rust standards of conduct, including hateful, hurtful, oppressive, or exclusionary remarks, are not allowed. (Cursing is allowed, but never targeting another user, and never in a hateful manner.) 2. Remarks that moderators find inappropriate, whether listed in the code of conduct or not, are also not allowed. 3. Moderators will first respond to such remarks with a warning. -4. If the warning is unheeded, the user will be "kicked," i.e., kicked out of the communication channel to cool off. -5. If the user comes back and continues to make trouble, they will be banned, i.e., indefinitely excluded. +4. If the warning is unheeded, the user will be "kicked," i.e. kicked out of the communication channel to cool off. +5. If the user comes back and continues to make trouble, they will be banned, i.e. indefinitely excluded. 6. Moderators may choose at their discretion to un-ban the user if it was a first offense and they offer the offended party a genuine apology. 7. If a moderator bans someone and you think it was unjustified, please take it up with that moderator, or with a different moderator, **in private**. Complaints about bans in-channel are not allowed. 8. Moderators are held to a higher standard than other community members. If a moderator creates an inappropriate situation, they should expect less leeway than others. diff --git a/src/glossary.md b/src/glossary.md index 10c2d4535..91af4c7da 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -25,7 +25,7 @@ provider | the function that executes a query ([see more](query. sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. -substs | the substitutions for a given generic type or item (e.g., the `i32`, `u32` in `HashMap`) +substs | the substitutions for a given generic type or item (e.g. the `i32`, `u32` in `HashMap`) tcx | the "typing context", main data structure of the compiler ([see more](ty.html)) 'tcx | the lifetime of the currently active inference context ([see more](ty.html)) trans | the code to translate MIR into LLVM IR. diff --git a/src/high-level-overview.md b/src/high-level-overview.md index 50ed072b8..0d3ad8be5 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -48,7 +48,7 @@ more and more to the [query model], however, the At the other extreme, the `rustc` crate defines the common and pervasive data structures that all the rest of the compiler uses -(e.g., how to represent types, traits, and the program itself). It +(e.g. how to represent types, traits, and the program itself). It also contains some amount of the compiler itself, although that is relatively limited. @@ -77,8 +77,8 @@ purely "pass-based" compiler, where we ran a number of passes over the entire program, and each did a particular check of transformation. We are gradually replacing this pass-based code with an alternative setup based on on-demand **queries**. In the query-model, we work backwards, -executing a *query* that expresses our ultimate goal (e.g., "compile -this crate"). This query in turn may make other queries (e.g., "get me +executing a *query* that expresses our ultimate goal (e.g. "compile +this crate"). This query in turn may make other queries (e.g. "get me a list of all modules in the crate"). Those queries make other queries that ultimately bottom out in the base operations, like parsing the input, running the type-checker, and so forth. This on-demand model diff --git a/src/hir.md b/src/hir.md index ac7690bd5..d7eff5cd9 100644 --- a/src/hir.md +++ b/src/hir.md @@ -18,7 +18,7 @@ data structure basically just contains the root module, the HIR `Crate` structure contains a number of maps and other things that serve to organize the content of the crate for easier access. -For example, the contents of individual items (e.g., modules, +For example, the contents of individual items (e.g. modules, functions, traits, impls, etc) in the HIR are not immediately accessible in the parents. So, for example, if there is a module item `foo` containing a function `bar()`: diff --git a/src/macro-expansion.md b/src/macro-expansion.md index a43c09394..95ea64f19 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -30,7 +30,7 @@ macro_rules! printer { `$mvar` is called a _metavariable_. Unlike normal variables, rather than binding to a value in a computation, a metavariable binds _at compile time_ to a tree of _tokens_. A _token_ is a single "unit" of the grammar, such as an -identifier (e.g., `foo`) or punctuation (e.g., `=>`). There are also other +identifier (e.g. `foo`) or punctuation (e.g. `=>`). There are also other special tokens, such as `EOF`, which indicates that there are no more tokens. Token trees resulting from paired parentheses-like characters (`(`...`)`, `[`...`]`, and `{`...`}`) – they include the open and close and all the tokens diff --git a/src/query.md b/src/query.md index fa17a3850..d119c16f3 100644 --- a/src/query.md +++ b/src/query.md @@ -25,7 +25,7 @@ will in turn demand information about that crate, starting from the *end*. For example: - This "compile" query might demand to get a list of codegen-units - (i.e., modules that need to be compiled by LLVM). + (i.e. modules that need to be compiled by LLVM). - But computing the list of codegen-units would invoke some subquery that returns the list of all modules defined in the Rust source. - That query in turn would invoke something asking for the HIR. @@ -134,7 +134,7 @@ fn provider<'cx, 'tcx>(tcx: TyCtxt<'cx, 'tcx, 'tcx>, ``` Providers take two arguments: the `tcx` and the query key. Note also -that they take the *global* tcx (i.e., they use the `'tcx` lifetime +that they take the *global* tcx (i.e. they use the `'tcx` lifetime twice), rather than taking a tcx with some active inference context. They return the result of the query. diff --git a/src/trait-resolution.md b/src/trait-resolution.md index 63f72b014..1c7f09e9b 100644 --- a/src/trait-resolution.md +++ b/src/trait-resolution.md @@ -408,7 +408,7 @@ if we had a trait reference `usize : Foo<$1>`, where `$n` is an unbound inference variable, we might replace it with `usize : Foo<%0>`, where `%n` is a skolemized type. We would then look this up in the cache. If we found a hit, the hit would tell us the immediate next step to -take in the selection process: i.e., apply impl #22, or apply where +take in the selection process: i.e. apply impl #22, or apply where clause `X : Foo`. Let's say in this case there is no hit. Therefore, we search through impls and where clauses and so forth, and we come to the conclusion that the only possible impl is this one, diff --git a/src/type-inference.md b/src/type-inference.md index 070b7bb4c..8155829fe 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -90,7 +90,7 @@ method, roughly like so: infcx.at(...).eq(t, u); ``` -The first `at()` call provides a bit of context, i.e., why you are +The first `at()` call provides a bit of context, i.e. why you are doing this unification, and in what environment, and the `eq` method performs the actual equality constraint. From 8a0935c485cea82fbf48feb84ba635214ac8f30e Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Sun, 4 Feb 2018 21:15:46 +0000 Subject: [PATCH 067/648] Normalized instances of `N.B.`. --- src/query.md | 2 +- src/ty.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/query.md b/src/query.md index d119c16f3..481b4fed3 100644 --- a/src/query.md +++ b/src/query.md @@ -190,7 +190,7 @@ pub fn provide(providers: &mut Providers) { fn fubar<'cx, 'tcx>(tcx: TyCtxt<'cx, 'tcx>, key: DefId) -> Fubar<'tcx> { .. } ``` -NB. Most of the `rustc_*` crates only provide **local +N.B. Most of the `rustc_*` crates only provide **local providers**. Almost all **extern providers** wind up going through the [`rustc_metadata` crate][rustc_metadata], which loads the information from the crate metadata. But in some cases there are crates that provide queries for diff --git a/src/ty.md b/src/ty.md index 0732e88bd..8b72d440f 100644 --- a/src/ty.md +++ b/src/ty.md @@ -102,7 +102,7 @@ The `sty` field (the origin of this name is unclear to me; perhaps structural type?) is of type `TypeVariants<'tcx>`, which is an enum defining all of the different kinds of types in the compiler. -> NB: inspecting the `sty` field on types during type inference can be +> N.B. inspecting the `sty` field on types during type inference can be > risky, as there may be inference variables and other things to > consider, or sometimes types are not yet known that will become > known later.). From 6bacd33a0b3a27df64593f9d95a3ced8a4b57092 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Sun, 4 Feb 2018 21:24:50 +0000 Subject: [PATCH 068/648] Very minor grammatical fixes. --- src/ty.md | 2 +- src/type-inference.md | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/ty.md b/src/ty.md index 8b72d440f..1da70d916 100644 --- a/src/ty.md +++ b/src/ty.md @@ -135,7 +135,7 @@ a safe approximation, so that is what you get back. You can also find various common types in the `tcx` itself by accessing `tcx.types.bool`, `tcx.types.char`, etc (see `CommonTypes` for more). -### Beyond types: Other kinds of arena-allocated data structures +### Beyond types: other kinds of arena-allocated data structures In addition to types, there are a number of other arena-allocated data structures that you can allocate, and which are found in this diff --git a/src/type-inference.md b/src/type-inference.md index 8155829fe..ef97e59d8 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -9,10 +9,10 @@ and higher-ranked types. We use the notation `?T` to refer to inference variables, also called existential variables. -We use the term "region" and "lifetime" interchangeably. Both refer to +We use the terms "region" and "lifetime" interchangeably. Both refer to the `'a` in `&'a T`. -The term "bound region" refers to regions bound in a function +The term "bound region" refers to a region bound in a function signature, such as the `'a` in `for<'a> fn(&'a u32)`. A region is "free" if it is not bound. @@ -158,7 +158,7 @@ is to first "generalize" `&'a i32` into a type with a region variable: relate this new variable with the original bound: &'?b i32 <: &'a i32 - + This will result in a region constraint (see below) of `'?b: 'a`. One final interesting case is relating two unbound type variables, @@ -175,7 +175,7 @@ make (almost) no attempt to solve regions. These constraints have the form of an outlives constraint: 'a: 'b - + Actually the code tends to view them as a subregion relation, but it's the same idea: @@ -188,7 +188,7 @@ There is one case where we do some amount of eager unification. If you have an e between two regions 'a = 'b - + we will record that fact in a unification table. You can then use `opportunistic_resolve_var` to convert `'b` to `'a` (or vice versa). This is sometimes needed to ensure termination of fixed-point From 61d67c582ccd38f0fd2b2bf5fe952515c0eed8eb Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Sun, 4 Feb 2018 21:28:29 +0000 Subject: [PATCH 069/648] Explained what HM type inference is. --- src/type-inference.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/type-inference.md b/src/type-inference.md index ef97e59d8..2cc922c76 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -1,7 +1,7 @@ # Type inference -The type inference is based on standard HM-type inference, but -extended in various way to accommodate subtyping, region inference, +The type inference is based on the standard Hindley–Milner (HM) system, +but extended in various way to accommodate subtyping, region inference, and higher-ranked types. ## A note on terminology From fde99687ab455fafb522775985932a221d9ed908 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Sun, 4 Feb 2018 21:41:49 +0000 Subject: [PATCH 070/648] Cleaned up section on type inference. --- src/type-inference.md | 43 +++++++++++++++++++++---------------------- 1 file changed, 21 insertions(+), 22 deletions(-) diff --git a/src/type-inference.md b/src/type-inference.md index 2cc922c76..b867c99e2 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -23,22 +23,21 @@ the following: ```rust tcx.infer_ctxt().enter(|infcx| { - // use the inference context `infcx` in here + // Use the inference context `infcx` here. }) ``` Each inference context creates a short-lived type arena to store the -fresh types and things that it will create, as described in -[the README in the ty module][ty-readme]. This arena is created by the `enter` -function and disposed after it returns. +fresh types and things that it will create, as described in the +[README in the `ty` module][ty-readme]. This arena is created by the `enter` +function and disposed of after it returns. [ty-readme]: ty.html -Within the closure, the infcx will have the type `InferCtxt<'cx, 'gcx, -'tcx>` for some fresh `'cx` and `'tcx` – the latter corresponds to -the lifetime of this temporary arena, and the `'cx` is the lifetime of -the `InferCtxt` itself. (Again, see [that ty README][ty-readme] for -more details on this setup.) +Within the closure, `infcx` has the type `InferCtxt<'cx, 'gcx, 'tcx>` +for some fresh `'cx` and `'tcx` – the latter corresponds to the lifetime of +this temporary arena, and the `'cx` is the lifetime of the `InferCtxt` itself. +(Again, see the [`ty` README][ty-readme] for more details on this setup.) The `tcx.infer_ctxt` method actually returns a build, which means there are some kinds of configuration you can do before the `infcx` is @@ -58,7 +57,7 @@ inference works, or perhaps this blog post on [Unification in the Chalk project]: http://smallcultfollowing.com/babysteps/blog/2017/03/25/unification-in-chalk-part-1/ -All told, the inference context stores four kinds of inference variables as of this +All said, the inference context stores four kinds of inference variables as of writing: - Type variables, which come in three varieties: @@ -67,7 +66,7 @@ writing: arise from an integer literal expression like `22`. - Float type variables, which can only be unified with a float type, and arise from a float literal expression like `22.0`. -- Region variables, which represent lifetimes, and arise all over the dang place. +- Region variables, which represent lifetimes, and arise all over the place. All the type variables work in much the same way: you can create a new type variable, and what you get is `Ty<'tcx>` representing an @@ -86,7 +85,7 @@ The most basic operations you can perform in the type inferencer is recommended way to add an equality constraint is using the `at` method, roughly like so: -``` +```rust infcx.at(...).eq(t, u); ``` @@ -95,7 +94,7 @@ doing this unification, and in what environment, and the `eq` method performs the actual equality constraint. When you equate things, you force them to be precisely equal. Equating -returns a `InferResult` – if it returns `Err(err)`, then equating +returns an `InferResult` – if it returns `Err(err)`, then equating failed, and the enclosing `TypeError` will tell you what went wrong. The success case is perhaps more interesting. The "primary" return @@ -105,12 +104,12 @@ side-effects of constraining type variables and so forth. However, the actual return type is not `()`, but rather `InferOk<()>`. The `InferOk` type is used to carry extra trait obligations – your job is to ensure that these are fulfilled (typically by enrolling them in a -fulfillment context). See the [trait README] for more background here. +fulfillment context). See the [trait README] for more background on that. [trait README]: trait-resolution.html -You can also enforce subtyping through `infcx.at(..).sub(..)`. The same -basic concepts apply as above. +You can similarly enforce subtyping through `infcx.at(..).sub(..)`. The same +basic concepts as above apply. ## "Trying" equality @@ -119,7 +118,7 @@ types without error. You can test that with `infcx.can_eq` (or `infcx.can_sub` for subtyping). If this returns `Ok`, then equality is possible – but in all cases, any side-effects are reversed. -Be aware though that the success or failure of these methods is always +Be aware, though, that the success or failure of these methods is always **modulo regions**. That is, two types `&'a u32` and `&'b u32` will return `Ok` for `can_eq`, even if `'a != 'b`. This falls out from the "two-phase" nature of how we solve region constraints. @@ -146,7 +145,7 @@ patterns. ## Subtyping obligations -One thing worth discussing are subtyping obligations. When you force +One thing worth discussing is subtyping obligations. When you force two types to be a subtype, like `?T <: i32`, we can often convert those into equality constraints. This follows from Rust's rather limited notion of subtyping: so, in the above case, `?T <: i32` is equivalent to `?T = i32`. @@ -172,11 +171,11 @@ mechanism. You'll have to try again when more details about `?T` or Regions are inferred somewhat differently from types. Rather than eagerly unifying things, we simply collect constraints as we go, but make (almost) no attempt to solve regions. These constraints have the -form of an outlives constraint: +form of an "outlives" constraint: 'a: 'b -Actually the code tends to view them as a subregion relation, but it's the same +Actually, the code tends to view them as a subregion relation, but it's the same idea: 'b <= 'a @@ -202,7 +201,7 @@ ways to solve region constraints right now: lexical and non-lexical. Eventually there will only be one. To solve **lexical** region constraints, you invoke -`resolve_regions_and_report_errors`. This will "close" the region +`resolve_regions_and_report_errors`. This "closes" the region constraint process and invoke the `lexical_region_resolve` code. Once this is done, any further attempt to equate or create a subtyping relationship will yield an ICE. @@ -224,4 +223,4 @@ Lexical region resolution is done by initially assigning each region variable to an empty value. We then process each outlives constraint repeatedly, growing region variables until a fixed-point is reached. Region variables can be grown using a least-upper-bound relation on -the region lattice in a fairly straight-forward fashion. +the region lattice in a fairly straightforward fashion. From 606e39a58b1bc17cdfccb664f076b750b6d85291 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Mon, 5 Feb 2018 02:30:32 +0000 Subject: [PATCH 071/648] More minor fixes. --- src/hir.md | 4 ++-- src/trait-resolution.md | 2 +- src/ty.md | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src/hir.md b/src/hir.md index d7eff5cd9..504f04b24 100644 --- a/src/hir.md +++ b/src/hir.md @@ -23,9 +23,9 @@ functions, traits, impls, etc) in the HIR are not immediately accessible in the parents. So, for example, if there is a module item `foo` containing a function `bar()`: -``` +```rust mod foo { - fn bar() { } + fn bar() { } } ``` diff --git a/src/trait-resolution.md b/src/trait-resolution.md index 1c7f09e9b..f12ce3b4b 100644 --- a/src/trait-resolution.md +++ b/src/trait-resolution.md @@ -457,7 +457,7 @@ and the graph is consulted when propagating defaults down the specialization hierarchy. You might expect that the specialization graph would be used during -selection – i.e., when actually performing specialization. This is +selection – i.e. when actually performing specialization. This is not done for two reasons: - It's merely an optimization: given a set of candidates that apply, diff --git a/src/ty.md b/src/ty.md index 1da70d916..26b8c1ca8 100644 --- a/src/ty.md +++ b/src/ty.md @@ -142,7 +142,7 @@ structures that you can allocate, and which are found in this module. Here are a few examples: - `Substs`, allocated with `mk_substs` – this will intern a slice of types, often used to - specify the values to be substituted for generics (e.g., `HashMap` + specify the values to be substituted for generics (e.g. `HashMap` would be represented as a slice `&'tcx [tcx.types.i32, tcx.types.u32]`). - `TraitRef`, typically passed by value – a **trait reference** consists of a reference to a trait along with its various type @@ -161,5 +161,5 @@ use ty::{self, Ty, TyCtxt}; In particular, since they are so common, the `Ty` and `TyCtxt` types are imported directly. Other types are often referenced with an -explicit `ty::` prefix (e.g., `ty::TraitRef<'tcx>`). But some modules +explicit `ty::` prefix (e.g. `ty::TraitRef<'tcx>`). But some modules choose to import a larger or smaller set of names explicitly. From 733663967129a40a8e4c65a2b250b591f1dd94f7 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Mon, 5 Feb 2018 02:15:36 +0000 Subject: [PATCH 072/648] Fixed issues mentioned by @mark-i-m in review. --- src/SUMMARY.md | 2 +- src/high-level-overview.md | 12 ++++++------ src/hir.md | 6 +++--- src/trans.md | 2 +- src/type-inference.md | 8 ++++---- 5 files changed, 15 insertions(+), 15 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 8f128f648..a2f40ed35 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -19,5 +19,5 @@ - [MIR construction](./mir-construction.md) - [MIR borrowck](./mir-borrowck.md) - [MIR optimizations](./mir-optimizations.md) -- [The `trans` crate: generating LLVM IR](./trans.md) +- [Generating LLVM IR](./trans.md) - [Glossary](./glossary.md) diff --git a/src/high-level-overview.md b/src/high-level-overview.md index 0d3ad8be5..493656713 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -12,10 +12,10 @@ Rustc consists of a number of crates, including `syntax`, many more. The source for each crate can be found in a directory like `src/libXXX`, where `XXX` is the crate name. -(NB. The names and divisions of these crates are not set in -stone and may change over time – for the time being, we tend towards -a finer-grained division to help with compilation time, though as -incremental improves that may change.) +(N.B. The names and divisions of these crates are not set in +stone and may change over time. For the time being, we tend towards a +finer-grained division to help with compilation time, though as incremental +compilation improves, that may change.) The dependency structure of these crates is roughly a diamond: @@ -37,7 +37,7 @@ rustc_trans rustc_borrowck ... rustc_metadata / \ / \ syntax_pos syntax_ext -``` +``` The `rustc_driver` crate, at the top of this lattice, is effectively the "main" function for the rust compiler. It doesn't have much "real @@ -127,7 +127,7 @@ take: 4. **Lowering to MIR and post-processing** - Once type-checking is done, we can lower the HIR into MIR ("middle IR"), which is a **very** desugared version of Rust, well suited to the borrowck but also - certain high-level optimizations. + certain high-level optimizations. 5. **Translation to LLVM and LLVM optimizations** - From MIR, we can produce LLVM IR. - LLVM then runs its various optimizations, which produces a number of `.o` files diff --git a/src/hir.md b/src/hir.md index 504f04b24..f78d2add2 100644 --- a/src/hir.md +++ b/src/hir.md @@ -59,9 +59,9 @@ sorts of identifiers in active use: - `DefId` – primarily names "definitions" or top-level items. - You can think of a `DefId` as shorthand for a very explicit and complete path, like `std::collections::HashMap`. However, these paths are able to - name things that are not nameable in normal Rust (e.g. impls), and they also - include extra information about the crate (such as its version number, since - two versions of the same crate can co-exist). + name things that are not nameable in normal Rust (e.g. `impl`s), and they + also include extra information about the crate (such as its version number, + since two versions of the same crate can co-exist). - A `DefId` really consists of two parts, a `CrateNum` (which identifies the crate) and a `DefIndex` (which indexes into a list of items that is maintained per crate). diff --git a/src/trans.md b/src/trans.md index efff52ef9..f5894baa7 100644 --- a/src/trans.md +++ b/src/trans.md @@ -1 +1 @@ -# The `trans` crate: generating LLVM IR +# Generating LLVM IR diff --git a/src/type-inference.md b/src/type-inference.md index b867c99e2..5e51c7358 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -1,8 +1,8 @@ # Type inference -The type inference is based on the standard Hindley–Milner (HM) system, -but extended in various way to accommodate subtyping, region inference, -and higher-ranked types. +The type inference is based on the standard Hindley-Milner (HM) type inference +algorithm, but extended in various way to accommodate subtyping, region +inference, and higher-ranked types. ## A note on terminology @@ -12,7 +12,7 @@ existential variables. We use the terms "region" and "lifetime" interchangeably. Both refer to the `'a` in `&'a T`. -The term "bound region" refers to a region bound in a function +The term "bound region" refers to a region that is a bound in a function signature, such as the `'a` in `for<'a> fn(&'a u32)`. A region is "free" if it is not bound. From 2b9828a566573c2d54e9b12dbee26263292a10dc Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Mon, 5 Feb 2018 03:43:46 +0000 Subject: [PATCH 073/648] Removed spurious `a`. --- src/type-inference.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/type-inference.md b/src/type-inference.md index 5e51c7358..327bd530a 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -12,7 +12,7 @@ existential variables. We use the terms "region" and "lifetime" interchangeably. Both refer to the `'a` in `&'a T`. -The term "bound region" refers to a region that is a bound in a function +The term "bound region" refers to a region that is bound in a function signature, such as the `'a` in `for<'a> fn(&'a u32)`. A region is "free" if it is not bound. From 45db49d4de1773cc31113c49a6e23976809ae064 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sun, 4 Feb 2018 21:51:10 -0600 Subject: [PATCH 074/648] Update link text --- src/type-inference.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/type-inference.md b/src/type-inference.md index 327bd530a..c4307ce7f 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -29,15 +29,15 @@ tcx.infer_ctxt().enter(|infcx| { Each inference context creates a short-lived type arena to store the fresh types and things that it will create, as described in the -[README in the `ty` module][ty-readme]. This arena is created by the `enter` +[chapter on the `ty` module][ty-ch]. This arena is created by the `enter` function and disposed of after it returns. -[ty-readme]: ty.html +[ty-ch]: ty.html Within the closure, `infcx` has the type `InferCtxt<'cx, 'gcx, 'tcx>` for some fresh `'cx` and `'tcx` – the latter corresponds to the lifetime of this temporary arena, and the `'cx` is the lifetime of the `InferCtxt` itself. -(Again, see the [`ty` README][ty-readme] for more details on this setup.) +(Again, see the [`ty` chapter][ty-ch] for more details on this setup.) The `tcx.infer_ctxt` method actually returns a build, which means there are some kinds of configuration you can do before the `infcx` is @@ -104,9 +104,9 @@ side-effects of constraining type variables and so forth. However, the actual return type is not `()`, but rather `InferOk<()>`. The `InferOk` type is used to carry extra trait obligations – your job is to ensure that these are fulfilled (typically by enrolling them in a -fulfillment context). See the [trait README] for more background on that. +fulfillment context). See the [trait chapter] for more background on that. -[trait README]: trait-resolution.html +[trait chapter]: trait-resolution.html You can similarly enforce subtyping through `infcx.at(..).sub(..)`. The same basic concepts as above apply. From 7d8cdcbb531601f52348e26c5eea64d2bd2e39f4 Mon Sep 17 00:00:00 2001 From: Philipp Hansch Date: Mon, 5 Feb 2018 20:39:35 +0100 Subject: [PATCH 075/648] Add DAG to glossary --- src/glossary.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/glossary.md b/src/glossary.md index 91af4c7da..de1390d53 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -6,6 +6,7 @@ The compiler uses a number of...idiosyncratic abbreviations and things. This glo Term | Meaning ------------------------|-------- AST | the abstract syntax tree produced by the syntax crate; reflects user syntax very closely. +DAG | a directed acyclic graph is used during compilation to index which queries execute which other queries. ([see more](incremental-compilation.html)) codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. cx | we tend to use "cx" as an abbrevation for context. See also `tcx`, `infcx`, etc. DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. From 6ebe10d55aa0b3bf080150fa056e6a49888b3a3a Mon Sep 17 00:00:00 2001 From: Philipp Hansch Date: Tue, 6 Feb 2018 07:54:03 +0100 Subject: [PATCH 076/648] Order alphabetically, improve explanation of DAG --- src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/glossary.md b/src/glossary.md index de1390d53..119e73255 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -6,9 +6,9 @@ The compiler uses a number of...idiosyncratic abbreviations and things. This glo Term | Meaning ------------------------|-------- AST | the abstract syntax tree produced by the syntax crate; reflects user syntax very closely. -DAG | a directed acyclic graph is used during compilation to index which queries execute which other queries. ([see more](incremental-compilation.html)) codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. cx | we tend to use "cx" as an abbrevation for context. See also `tcx`, `infcx`, etc. +DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](incremental-compilation.html)) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. HIR | the High-level IR, created by lowering and desugaring the AST ([see more](hir.html)) HirId | identifies a particular node in the HIR by combining a def-id with an "intra-definition offset". From a2d2cfa89afb9aa2aa87452e0ee414e488731b4e Mon Sep 17 00:00:00 2001 From: Phil Ellison Date: Wed, 7 Feb 2018 07:08:58 +0000 Subject: [PATCH 077/648] Explain how to dump HIR --- src/hir.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/hir.md b/src/hir.md index 52fda69a6..129336874 100644 --- a/src/hir.md +++ b/src/hir.md @@ -10,6 +10,14 @@ been desugared away (as an example, `for` loops are converted into a This chapter covers the main concepts of the HIR. +You can view the HIR representation of your code by passing the +`-Zunpretty=hir-tree` flag to rustc, for example by setting the `RUSTFLAGS` +environment variable: + +``` +RUSTFLAGS=-Zunpretty=hir-tree cargo build +``` + ### Out-of-band storage and the `Crate` type The top-level data-structure in the HIR is the `Crate`, which stores From 5ba0d6bc04309dc01913e6b9590efb30641276ea Mon Sep 17 00:00:00 2001 From: Phil Ellison Date: Wed, 7 Feb 2018 19:14:58 +0000 Subject: [PATCH 078/648] Use cargo rustc instead of setting RUSTFLAGS --- src/hir.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/hir.md b/src/hir.md index 129336874..f3145925c 100644 --- a/src/hir.md +++ b/src/hir.md @@ -11,11 +11,10 @@ been desugared away (as an example, `for` loops are converted into a This chapter covers the main concepts of the HIR. You can view the HIR representation of your code by passing the -`-Zunpretty=hir-tree` flag to rustc, for example by setting the `RUSTFLAGS` -environment variable: +`-Zunpretty=hir-tree` flag to rustc: ``` -RUSTFLAGS=-Zunpretty=hir-tree cargo build +cargo rustc -- -Zunpretty=hir-tree ``` ### Out-of-band storage and the `Crate` type From 4b7bdec2e3111f8a55e541099b3341ab1e6c38a5 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sun, 11 Feb 2018 15:46:36 -0600 Subject: [PATCH 079/648] alphabetize glossary --- src/glossary.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/glossary.md b/src/glossary.md index 119e73255..b7282c3eb 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -17,12 +17,12 @@ generics | the set of generic type parameters defined on a type ICE | internal compiler error. When the compiler crashes. ICH | incremental compilation hash. ICHs are used as fingerprints for things such as HIR and crate metadata, to check if changes have been made. This is useful in incremental compilation to see if part of a crate has changed and should be recompiled. infcx | the inference context (see `librustc/infer`) -MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) -obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) local crate | the crate currently being compiled. +MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) node-id or NodeId | an index identifying a particular node in the AST or HIR; gradually being phased out and replaced with `HirId`. -query | perhaps some sub-computation during compilation ([see more](query.html)) +obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) provider | the function that executes a query ([see more](query.html)) +query | perhaps some sub-computation during compilation ([see more](query.html)) sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. From f79e0d0cb1c5db336c43438567a3ca16efd444ff Mon Sep 17 00:00:00 2001 From: Brad Gibson Date: Tue, 13 Feb 2018 05:53:35 -0800 Subject: [PATCH 080/648] created compiletest.md describe the steps required to add a test and a header command to compiletest --- src/compiletest.md | 183 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100644 src/compiletest.md diff --git a/src/compiletest.md b/src/compiletest.md new file mode 100644 index 000000000..a8942efba --- /dev/null +++ b/src/compiletest.md @@ -0,0 +1,183 @@ +# `compiletest` +## Introduction +`compiletest` is the main test harness of the Rust test suite. It allows test authors to organize large numbers of tests (the +Rust compiler has many thousands), efficient test execution (parallel execution is supported), and allows the test author to +configure behavior and expected results of both individual and groups of tests. + +`compiletest` tests may check test code for success, for failure or in some cases, even failure to compile. Tests are +typically organized as a Rust source file with annotations in comments before and/or within the test code, which serve to +direct `compiletest` on if or how to run the test, what behavior to expect, and more. + +The tests themselves are typically (but not always) organized into "suites"--for example, `run-pass`, a folder +representing tests that should succeed, `run-fail`, a folder holding tests that should compile successfully, but return +a failure (non-zero status), `compile-fail`, a folder holding tests that should fail to compile, and many more. The various +suites are defined in `src/tools/compiletest/src/common.rs` in the `pub struct Config` declaration. And a very good +introduction to the different suites of compiler tests along with details about them can be found +[here, at Brian Anderson's blog](https://brson.github.io/2017/07/10/how-rust-is-tested#s-ct). + +## Adding a new test file +Simply create your new test in the appropriate location under `/src/test`. No registration of test files is necessary as +`compiletest` will scan the `src/test` subfolder recursively, and will execute any Rust source files it finds as tests. + +## Header Commands +Source file annotations which appear in comments near the top of the source file *before* any test code are known as header +commands. These commands can instruct `compiletest` to ignore this test, set expectations on whether it is expected to +succeed at compiling, or what the test's return code is expected to be. Header commands (and their inline counterparts, +Error Info commands) are described more fully +[here](https://github.com/rust-lang/rust/blob/master/src/test/COMPILER_TESTS.md). + +### Adding a new header command +Header commands are defined in `src/tools/compiletest/src/header.rs`. At a high level, dozens of test properties are defined +in the `TestProps` struct. These are all set to default values in the struct's `impl` block and any test can override this +default value by specifying the property as header command as a comment (`//`) in the test source file, before any source +code. + +#### Using a header command +Here is an example, specifying the `must-compile-successfully` header command, which takes no arguments, followed by the +`failure-status` header command, which takes a single argument (which, in this case is a value of 1). `failure-status` is +instructing `compiletest` to expect a failure status of 1 (rather than the current Rust default of 101 at the time of this +writing). The header command and the argument list (if present) are typically separated by a colon: +``` +// Copyright 2018 The Rust Project Developers. See the COPYRIGHT +// file at the top-level directory of this distribution and at +// http://rust-lang.org/COPYRIGHT. +// +// Licensed under the Apache License, Version 2.0 or the MIT license +// , at your +// option. This file may not be copied, modified, or distributed +// except according to those terms. + +// must-compile-successfully +// failure-status: 1 + +#![feature(termination_trait)] + +use std::io::{Error, ErrorKind}; + +fn main() -> Result<(), Box> { + Err(Box::new(Error::new(ErrorKind::Other, "returned Box from main()"))) +} +``` + +#### Adding a new header command property +One would add a new header command if there is a need to define some test property or behavior on an individual, test-by-test +basis. A header command property serves as the header command's backing store (holds the command's current value) at +runtime. + +To add a new header command property: + 1. Look for the `pub struct TestProps` declaration in `src/tools/compiletest/src/header.rs` and add +the new public property to the end of the declaration. + 2. Look for the `impl TestProps` implementation block immediately following the struct declaration and initialize the new +property to its default value. + +#### Adding a new header command parser +When `compiletest` encounters a test file, it parses the file a line at a time by calling every parsers defined in the +`Config` struct's implementation block, also in `src/tools/compiletest/src/header.rs` (note the `Config` struct's declaration +block is found in `src/tools/compiletest/src/common.rs`). `TestProps`'s `load_from()` method will try passing the current +line of text to each parser, which, in turn typically checks to see if the line begins with a particular commented (`//`) +header command such as `// must-compile-successfully` or `// failure-status`. Whitespace after the comment marker is +optional. + +Parsers will override a given header command property's default value merely by being specified in the test file as a header +command or by having a parameter value specified in the test file, depending on the header command. + +Parsers defined in `impl Config` are typically named `parse_` (note kebab-case `` transformed +to snake-case ``). `impl Config` also defines several 'low-level' parsers which make it simple to parse +common patterns like simple presence or not (`parse_name_directive()`), header-command:parameter(s) +(`parse_name_value_directive()`), optional parsing only if a particular `cfg` attribute is defined (`has_cfg_prefix()`) and +many more. The low-level parsers are found near the end of the `impl Config` block--be sure to look through them and their +associated parsers immediately above to see how they are used to avoid writing additional parsing code unneccessarily. + +As a concrete example, here is the implementation for the `parse_failure_status()` parser, in +`src/tools/compiletest/src/header.rs`: +```diff +@@ -232,6 +232,7 @@ pub struct TestProps { + // customized normalization rules + pub normalize_stdout: Vec<(String, String)>, + pub normalize_stderr: Vec<(String, String)>, ++ pub failure_status: i32, + } + + impl TestProps { +@@ -260,6 +261,7 @@ impl TestProps { + run_pass: false, + normalize_stdout: vec![], + normalize_stderr: vec![], ++ failure_status: 101, + } + } + +@@ -383,6 +385,10 @@ impl TestProps { + if let Some(rule) = config.parse_custom_normalization(ln, "normalize-stderr") { + self.normalize_stderr.push(rule); + } ++ ++ if let Some(code) = config.parse_failure_status(ln) { ++ self.failure_status = code; ++ } + }); + + for key in &["RUST_TEST_NOCAPTURE", "RUST_TEST_THREADS"] { +@@ -488,6 +494,13 @@ impl Config { + self.parse_name_directive(line, "pretty-compare-only") + } + ++ fn parse_failure_status(&self, line: &str) -> Option { ++ match self.parse_name_value_directive(line, "failure-status") { ++ Some(code) => code.trim().parse::().ok(), ++ _ => None, ++ } ++ } +``` + +## Implementing the behavior change +When a test invokes a particular header command, it is expected that some behavior will change as a result. What behavior, +obviously, will depend on the purpose of the header command. In the case of `failure-status`, the behavior that changes +is that `compiletest` expects the failure code defined by the header command invoked in the test, rather than the default +value. + +Although specific to `failure-status` (as every header ommand will have a different implementation in order to invoke +behavior change) perhaps it is helpful to see the behavior hange implementation of one case, simply as an example. To +implement `failure-status`, the `check_correct_failure_status()` function found in the `TestCx` implementation block, located +in `src/tools/compiletest/src/runtest.rs`, was modified as per below: +```diff +@@ -295,11 +295,14 @@ impl<'test> TestCx<'test> { + } + + fn check_correct_failure_status(&self, proc_res: &ProcRes) { +- // The value the rust runtime returns on failure +- const RUST_ERR: i32 = 101; +- if proc_res.status.code() != Some(RUST_ERR) { ++ let expected_status = Some(self.props.failure_status); ++ let received_status = proc_res.status.code(); ++ ++ if expected_status != received_status { + self.fatal_proc_rec( +- &format!("failure produced the wrong error: {}", proc_res.status), ++ &format!("Error: expected failure status ({:?}) but received status {:?}.", ++ expected_status, ++ received_status), + proc_res, + ); + } +@@ -320,7 +323,6 @@ impl<'test> TestCx<'test> { + ); + + let proc_res = self.exec_compiled_test(); +- + if !proc_res.status.success() { + self.fatal_proc_rec("test run failed!", &proc_res); + } +@@ -499,7 +501,6 @@ impl<'test> TestCx<'test> { + expected, + actual + ); +- panic!(); + } + } +``` +Note the use of `self.props.failure_status` to access the header command property. In tests which do not specify the failure +status header command, `self.props.failure_status` will evaluate to the default value of 101 at the time of this writing. +But for a test which specifies a header command of, for example, `// failure-status: 1`, `self.props.failure_status` will +evaluate to 1, as `parse_failure_status()` will have overridden the `TestProps` default value, for that test specifically. From 59485d219af70485f6e9ec406fc68c963e9831b8 Mon Sep 17 00:00:00 2001 From: Brad Gibson Date: Tue, 13 Feb 2018 09:22:33 -0800 Subject: [PATCH 081/648] Update compiletest.md changed file references to GitHub links directly to files --- src/compiletest.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/src/compiletest.md b/src/compiletest.md index a8942efba..aa4558514 100644 --- a/src/compiletest.md +++ b/src/compiletest.md @@ -11,13 +11,13 @@ direct `compiletest` on if or how to run the test, what behavior to expect, and The tests themselves are typically (but not always) organized into "suites"--for example, `run-pass`, a folder representing tests that should succeed, `run-fail`, a folder holding tests that should compile successfully, but return a failure (non-zero status), `compile-fail`, a folder holding tests that should fail to compile, and many more. The various -suites are defined in `src/tools/compiletest/src/common.rs` in the `pub struct Config` declaration. And a very good +suites are defined in [src/tools/compiletest/src/common.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs) in the `pub struct Config` declaration. And a very good introduction to the different suites of compiler tests along with details about them can be found [here, at Brian Anderson's blog](https://brson.github.io/2017/07/10/how-rust-is-tested#s-ct). ## Adding a new test file -Simply create your new test in the appropriate location under `/src/test`. No registration of test files is necessary as -`compiletest` will scan the `src/test` subfolder recursively, and will execute any Rust source files it finds as tests. +Simply create your new test in the appropriate location under [src/test](https://github.com/rust-lang/rust/tree/master/src/test). No registration of test files is necessary as +`compiletest` will scan the [src/test](https://github.com/rust-lang/rust/tree/master/src/test) subfolder recursively, and will execute any Rust source files it finds as tests. ## Header Commands Source file annotations which appear in comments near the top of the source file *before* any test code are known as header @@ -27,7 +27,7 @@ Error Info commands) are described more fully [here](https://github.com/rust-lang/rust/blob/master/src/test/COMPILER_TESTS.md). ### Adding a new header command -Header commands are defined in `src/tools/compiletest/src/header.rs`. At a high level, dozens of test properties are defined +Header commands are defined in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs). At a high level, dozens of test properties are defined in the `TestProps` struct. These are all set to default values in the struct's `impl` block and any test can override this default value by specifying the property as header command as a comment (`//`) in the test source file, before any source code. @@ -66,15 +66,15 @@ basis. A header command property serves as the header command's backing store ( runtime. To add a new header command property: - 1. Look for the `pub struct TestProps` declaration in `src/tools/compiletest/src/header.rs` and add + 1. Look for the `pub struct TestProps` declaration in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs) and add the new public property to the end of the declaration. 2. Look for the `impl TestProps` implementation block immediately following the struct declaration and initialize the new property to its default value. #### Adding a new header command parser When `compiletest` encounters a test file, it parses the file a line at a time by calling every parsers defined in the -`Config` struct's implementation block, also in `src/tools/compiletest/src/header.rs` (note the `Config` struct's declaration -block is found in `src/tools/compiletest/src/common.rs`). `TestProps`'s `load_from()` method will try passing the current +`Config` struct's implementation block, also in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs) (note the `Config` struct's declaration +block is found in [src/tools/compiletest/src/common.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs). `TestProps`'s `load_from()` method will try passing the current line of text to each parser, which, in turn typically checks to see if the line begins with a particular commented (`//`) header command such as `// must-compile-successfully` or `// failure-status`. Whitespace after the comment marker is optional. @@ -90,7 +90,7 @@ many more. The low-level parsers are found near the end of the `impl Config` bl associated parsers immediately above to see how they are used to avoid writing additional parsing code unneccessarily. As a concrete example, here is the implementation for the `parse_failure_status()` parser, in -`src/tools/compiletest/src/header.rs`: +[src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs): ```diff @@ -232,6 +232,7 @@ pub struct TestProps { // customized normalization rules @@ -140,7 +140,7 @@ value. Although specific to `failure-status` (as every header ommand will have a different implementation in order to invoke behavior change) perhaps it is helpful to see the behavior hange implementation of one case, simply as an example. To implement `failure-status`, the `check_correct_failure_status()` function found in the `TestCx` implementation block, located -in `src/tools/compiletest/src/runtest.rs`, was modified as per below: +in [src/tools/compiletest/src/runtest.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs), was modified as per below: ```diff @@ -295,11 +295,14 @@ impl<'test> TestCx<'test> { } From 5723bce6f1e531a922878ad3b847a5c478b7110b Mon Sep 17 00:00:00 2001 From: Brad Gibson Date: Tue, 13 Feb 2018 09:45:26 -0800 Subject: [PATCH 082/648] fixed typos --- src/compiletest.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/src/compiletest.md b/src/compiletest.md index aa4558514..0b10cfbb7 100644 --- a/src/compiletest.md +++ b/src/compiletest.md @@ -72,7 +72,7 @@ the new public property to the end of the declaration. property to its default value. #### Adding a new header command parser -When `compiletest` encounters a test file, it parses the file a line at a time by calling every parsers defined in the +When `compiletest` encounters a test file, it parses the file a line at a time by calling every parser defined in the `Config` struct's implementation block, also in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs) (note the `Config` struct's declaration block is found in [src/tools/compiletest/src/common.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs). `TestProps`'s `load_from()` method will try passing the current line of text to each parser, which, in turn typically checks to see if the line begins with a particular commented (`//`) @@ -86,7 +86,7 @@ Parsers defined in `impl Config` are typically named `parse_` (n to snake-case ``). `impl Config` also defines several 'low-level' parsers which make it simple to parse common patterns like simple presence or not (`parse_name_directive()`), header-command:parameter(s) (`parse_name_value_directive()`), optional parsing only if a particular `cfg` attribute is defined (`has_cfg_prefix()`) and -many more. The low-level parsers are found near the end of the `impl Config` block--be sure to look through them and their +many more. The low-level parsers are found near the end of the `impl Config` block; be sure to look through them and their associated parsers immediately above to see how they are used to avoid writing additional parsing code unneccessarily. As a concrete example, here is the implementation for the `parse_failure_status()` parser, in @@ -137,10 +137,9 @@ obviously, will depend on the purpose of the header command. In the case of `fa is that `compiletest` expects the failure code defined by the header command invoked in the test, rather than the default value. -Although specific to `failure-status` (as every header ommand will have a different implementation in order to invoke -behavior change) perhaps it is helpful to see the behavior hange implementation of one case, simply as an example. To -implement `failure-status`, the `check_correct_failure_status()` function found in the `TestCx` implementation block, located -in [src/tools/compiletest/src/runtest.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs), was modified as per below: +Although specific to `failure-status` (as every header command will have a different implementation in order to invoke +behavior change) perhaps it is helpful to see the behavior hange implementation of one case, simply as an example. To implement `failure-status`, the `check_correct_failure_status()` function found in the `TestCx` implementation block, +located in [src/tools/compiletest/src/runtest.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs), was modified as per below: ```diff @@ -295,11 +295,14 @@ impl<'test> TestCx<'test> { } From fc83b6fe198525d7680d4b416afb555501c9edd9 Mon Sep 17 00:00:00 2001 From: Brad Gibson Date: Tue, 13 Feb 2018 09:53:45 -0800 Subject: [PATCH 083/648] Added 'How to add header commands to `compiletest`' --- src/SUMMARY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index a2f40ed35..6a3907551 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -3,6 +3,7 @@ - [About this guide](./about-this-guide.md) - [How to build the compiler and run what you built](./how-to-build-and-run.md) - [Using the compiler testing framework](./running-tests.md) +- [How to add new header commands to `compiletest`](./compiletest.md) - [Walkthrough: a typical contribution](./walkthrough.md) - [High-level overview of the compiler source](./high-level-overview.md) - [Queries: demand-driven compilation](./query.md) From 6ae355c18ddebde72f139607363cec968c4e33fb Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 13 Feb 2018 11:00:42 -0600 Subject: [PATCH 084/648] add soundness and completeness to glossary --- src/glossary.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/glossary.md b/src/glossary.md index b7282c3eb..cb3437f1d 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -7,6 +7,7 @@ Term | Meaning ------------------------|-------- AST | the abstract syntax tree produced by the syntax crate; reflects user syntax very closely. codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. +completeness | soundness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). cx | we tend to use "cx" as an abbrevation for context. See also `tcx`, `infcx`, etc. DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](incremental-compilation.html)) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. @@ -25,6 +26,7 @@ provider | the function that executes a query ([see more](query. query | perhaps some sub-computation during compilation ([see more](query.html)) sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. +soundness | soundness is a technical term in type theory. Roughly, if a type system is sound, then if a program type-checks, it is type-safe; i.e. I can never (in safe rust) force a value into a variable of the wrong type. (see "completeness"). span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. substs | the substitutions for a given generic type or item (e.g. the `i32`, `u32` in `HashMap`) tcx | the "typing context", main data structure of the compiler ([see more](ty.html)) From 4357cf5c8ee1aa3bea59a57ef6819dc1ca4fbfe6 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Tue, 13 Feb 2018 11:20:19 -0600 Subject: [PATCH 085/648] Update glossary.md --- src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/glossary.md b/src/glossary.md index cb3437f1d..a22c49632 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -7,7 +7,7 @@ Term | Meaning ------------------------|-------- AST | the abstract syntax tree produced by the syntax crate; reflects user syntax very closely. codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. -completeness | soundness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). +completeness | completeness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). cx | we tend to use "cx" as an abbrevation for context. See also `tcx`, `infcx`, etc. DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](incremental-compilation.html)) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. From 169cdeb78ffc5a22f0dc22454dc036493e93bce7 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Thu, 15 Feb 2018 12:53:49 -0600 Subject: [PATCH 086/648] Add a few more terms to the glossary --- src/glossary.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/glossary.md b/src/glossary.md index a22c49632..f252e6956 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -24,13 +24,16 @@ node-id or NodeId | an index identifying a particular node in the AST or obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) provider | the function that executes a query ([see more](query.html)) query | perhaps some sub-computation during compilation ([see more](query.html)) +region | another term for "lifetime" often used in the literature and in the borrow checker. sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. +sigil | like a keyword but composed entirely of non-alphanumeric tokens. For example, `&` is a sigil for references. soundness | soundness is a technical term in type theory. Roughly, if a type system is sound, then if a program type-checks, it is type-safe; i.e. I can never (in safe rust) force a value into a variable of the wrong type. (see "completeness"). span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. substs | the substitutions for a given generic type or item (e.g. the `i32`, `u32` in `HashMap`) tcx | the "typing context", main data structure of the compiler ([see more](ty.html)) 'tcx | the lifetime of the currently active inference context ([see more](ty.html)) +token | the smallest unit of parsing. Tokens are produced after lexing ([see more](the-parser.html)). trans | the code to translate MIR into LLVM IR. trait reference | a trait and values for its type parameters ([see more](ty.html)). ty | the internal representation of a type ([see more](ty.html)). From 58b67c21d6139034477ad6f00ba5f533d88316ee Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 7 Feb 2018 09:47:02 -0500 Subject: [PATCH 087/648] document the test infrastructure --- src/SUMMARY.md | 4 +- src/running-tests.md | 1 - src/tests/adding.md | 273 +++++++++++++++++++++++++++++++++++++++++++ src/tests/intro.md | 48 ++++++++ src/tests/running.md | 65 +++++++++++ 5 files changed, 389 insertions(+), 2 deletions(-) delete mode 100644 src/running-tests.md create mode 100644 src/tests/adding.md create mode 100644 src/tests/intro.md create mode 100644 src/tests/running.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index a2f40ed35..21976fbf7 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -2,7 +2,9 @@ - [About this guide](./about-this-guide.md) - [How to build the compiler and run what you built](./how-to-build-and-run.md) -- [Using the compiler testing framework](./running-tests.md) +- [The compiler testing framework](./tests/intro.md) + - [Running tests](./tests/running.md) + - [Adding new tests](./tests/adding.md) - [Walkthrough: a typical contribution](./walkthrough.md) - [High-level overview of the compiler source](./high-level-overview.md) - [Queries: demand-driven compilation](./query.md) diff --git a/src/running-tests.md b/src/running-tests.md deleted file mode 100644 index 02c9de840..000000000 --- a/src/running-tests.md +++ /dev/null @@ -1 +0,0 @@ -# Using the compiler testing framework diff --git a/src/tests/adding.md b/src/tests/adding.md new file mode 100644 index 000000000..9ce8318db --- /dev/null +++ b/src/tests/adding.md @@ -0,0 +1,273 @@ +# Adding new tests + +**In general, we expect every PR that fixes a bug in rustc to come +accompanied with a regression test of some kind.** This test should +fail in master but pass after the PR. These tests are really useful +for preventing us from repeating the mistakes of the past. + +To add a new test, the first thing you generally do is to create a +file, typically a Rust source file. Test files have a particular +structure: + +- They always begin with the copyright notice; +- then they should have some kind of + [comment explaining what the test is about](#explanatory_comment); +- next, they can have one or more [header commands](#header_commands), which are special + comments that the test interpreter knows how to interpret. +- finally, they have the Rust source. This may have various [error + annotations](#error_annotations) which indicate expected compilation errors or + warnings. + +Depending on the test suite, there may be some other details to be aware of: + - For [the `ui` test suite](#ui), you need to generate reference output files. + +## Naming your test + +We have not traditionally had a lot of structure in the names of +tests. Moreover, for a long time, the rustc test runner did not +support subdirectories (it now does), so test suites like +`src/test/run-pass` have a huge mess of files in them. This is not +considered an ideal setup. + +For regression tests -- basically, some random snippet of code that +came in from the internet -- we often just name the test after the +issue. For example, `src/test/run-pass/issue-1234.rs`. If possible, +though, it is better if you can put the test into a directory that +helps identify what piece of code is being tested here (e.g., +`borrowck/issue-12345.rs` is much better), or perhaps give it a more +meaningful name. Still, **do include the issue number somewhere**. + +When writing a new feature, **create a subdirectory to store your +tests**. For example, if you are implementing RFC 1234 ("Widgets"), +then it might make sense to put the tests in directories like: + +- `src/test/ui/rfc1234-widgets/` +- `src/test/run-pass/rfc1234-widgets/` +- etc + +In other cases, there may already be a suitable directory. (The proper +directory structure to use is actually an area of active debate.) + + + +## Comment explaining what the test is about + +When you create a test file, **include a comment summarizing the point +of the test immediately after the copyright notice**. This should +highlight which parts of the test are more important, and what the bug +was that the test is fixing. Citing an issue number is often very +helpful. + +This comment doesn't have to be super extensive. Just something like +"Regression test for #18060: match arms were matching in the wrong +order." might already be enough. + +These comments are very useful to others later on when your test +breaks, since they often can highlight what the problem is. They are +also useful if for some reason the tests need to be refactored, since +they let others know which parts of the test were important (often a +test must be rewritten because it no longer tests what is was meant to +test, and then it's useful to know what it *was* meant to test +exactly). + + + +## Header commands: configuring rustc + +Header commands are special comments that the test runner knows how to +interpret. They must appear before the Rust source in the test. They +are normally put after the short comment that explains the point of +this test. For example, this test uses the `// compile-flags` command +to specify a custom flag to give to rustc when the test is compiled: + +```rust +// Copyright 2017 The Rust Project Developers. blah blah blah. +// ... +// except according to those terms. + +// Test the behavior of `0 - 1` when overflow checks are disabled. + +// compile-flags: -Coverflow-checks=off + +fn main() { + let x = 0 - 1; + ... +} +``` + +### Ignoring tests + +These are used to ignore the test in some situations, which means the test won't +be compiled or run. + +* `ignore-X` where `X` is a target detail or stage will ignore the test accordingly (see below) +* `ignore-pretty` will not compile the pretty-printed test (this is done to test the pretty-printer, but might not always work) +* `ignore-test` always ignores the test +* `ignore-lldb` and `ignore-gdb` will skip a debuginfo test on that debugger. + +Some examples of `X` in `ignore-X`: + +* Architecture: `aarch64`, `arm`, `asmjs`, `mips`, `wasm32`, `x86_64`, `x86`, ... +* OS: `android`, `emscripten`, `freebsd`, `ios`, `linux`, `macos`, `windows`, ... +* Environment (fourth word of the target triple): `gnu`, `msvc`, `musl`. +* Pointer width: `32bit`, `64bit`. +* Stage: `stage0`, `stage1`, `stage2`. + +### Other Header Commands + +Here is a list of other header commands. This list is not +exhaustive. Header commands can generally be found by browsing the +`TestProps` structure found in [`header.rs`] from the compiletest +source. + +* `min-{gdb,lldb}-version` +* `min-llvm-version` +* `must-compile-successfully` for UI tests, indicates that the test is supposed + to compile, as opposed to the default where the test is supposed to error out. +* `compile-flags` passes extra command-line args to the compiler, + e.g. `compile-flags -g` which forces debuginfo to be enabled. +* `should-fail` indicates that the test should fail; used for "meta testing", + where we test the compiletest program itself to check that it will generate + errors in appropriate scenarios. This header is ignored for pretty-printer tests. +* `gate-test-X` where `X` is a feature marks the test as "gate test" for feature X. + Such tests are supposed to ensure that the compiler errors when usage of a gated + feature is attempted without the proper `#![feature(X)]` tag. + Each unstable lang feature is required to have a gate test. + +[`header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs + + + +## Error annotations + +Error annotations specify the errors that the compiler is expected to +emit. They are "attached" to the line in source where the error is +located. + +* `~`: Associates the following error level and message with the + current line +* `~|`: Associates the following error level and message with the same + line as the previous comment +* `~^`: Associates the following error level and message with the + previous line. Each caret (`^`) that you add adds a line to this, so + `~^^^^^^^` is seven lines up. + +The error levels that you can have are: + +1. `ERROR` +2. `WARNING` +3. `NOTE` +4. `HELP` and `SUGGESTION`* + +\* **Note**: `SUGGESTION` must follow immediately after `HELP`. + +## Revisions + +Certain classes of tests support "revisions" (as of the time of this +writing, this includes run-pass, compile-fail, run-fail, and +incremental, though incremental tests are somewhat +different). Revisions allow a single test file to be used for multiple +tests. This is done by adding a special header at the top of the file: + +``` +// revisions: foo bar baz +``` + +This will result in the test being compiled (and tested) three times, +once with `--cfg foo`, once with `--cfg bar`, and once with `--cfg +baz`. You can therefore use `#[cfg(foo)]` etc within the test to tweak +each of these results. + +You can also customize headers and expected error messages to a particular +revision. To do this, add `[foo]` (or `bar`, `baz`, etc) after the `//` +comment, like so: + +``` +// A flag to pass in only for cfg `foo`: +//[foo]compile-flags: -Z verbose + +#[cfg(foo)] +fn test_foo() { + let x: usize = 32_u32; //[foo]~ ERROR mismatched types +} +``` + +Note that not all headers have meaning when customized to a revision. +For example, the `ignore-test` header (and all "ignore" headers) +currently only apply to the test as a whole, not to particular +revisions. The only headers that are intended to really work when +customized to a revision are error patterns and compiler flags. + + + +## Guide to the UI tests + +The UI tests are intended to capture the compiler's complete output, +so that we can test all aspects of the presentation. They work by +compiling a file (e.g., `ui/hello_world/main.rs`), capturing the output, +and then applying some normalization (see below). This normalized +result is then compared against reference files named +`ui/hello_world/main.stderr` and `ui/hello_world/main.stdout`. If either of +those files doesn't exist, the output must be empty. If the test run +fails, we will print out the current output, but it is also saved in +`build//test/ui/hello_world/main.stdout` (this path is +printed as part of the test failure message), so you can run `diff` and +so forth. + +Normally, the test-runner checks that UI tests fail compilation. If you want +to do a UI test for code that *compiles* (e.g. to test warnings, or if you +have a collection of tests, only some of which error out), you can use the +`// must-compile-successfully` header command to have the test runner instead +check that the test compiles successfully. + +### Editing and updating the reference files + +If you have changed the compiler's output intentionally, or you are +making a new test, you can use the script `ui/update-references.sh` to +update the references. When you run the test framework, it will report +various errors: in those errors is a command you can use to run the +`ui/update-references.sh` script, which will then copy over the files +from the build directory and use them as the new reference. You can +also just run `ui/update-all-references.sh`. In both cases, you can run +the script with `--help` to get a help message. + +### Normalization + +The normalization applied is aimed at eliminating output difference +between platforms, mainly about filenames: + +- the test directory is replaced with `$DIR` +- all backslashes (`\`) are converted to forward slashes (`/`) (for Windows) +- all CR LF newlines are converted to LF + +Sometimes these built-in normalizations are not enough. In such cases, you +may provide custom normalization rules using the header commands, e.g. + +``` +// normalize-stdout-test: "foo" -> "bar" +// normalize-stderr-32bit: "fn\(\) \(32 bits\)" -> "fn\(\) \($$PTR bits\)" +// normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)" +``` + +This tells the test, on 32-bit platforms, whenever the compiler writes +`fn() (32 bits)` to stderr, it should be normalized to read `fn() ($PTR bits)` +instead. Similar for 64-bit. The replacement is performed by regexes using +default regex flavor provided by `regex` crate. + +The corresponding reference file will use the normalized output to test both +32-bit and 64-bit platforms: + +``` +... + | + = note: source type: fn() ($PTR bits) + = note: target type: u16 (16 bits) +... +``` + +Please see `ui/transmute/main.rs` and `.stderr` for a concrete usage example. + +Besides `normalize-stderr-32bit` and `-64bit`, one may use any target +information or stage supported by `ignore-X` here as well (e.g. +`normalize-stderr-windows` or simply `normalize-stderr-test` for unconditional +replacement). diff --git a/src/tests/intro.md b/src/tests/intro.md new file mode 100644 index 000000000..c22916dca --- /dev/null +++ b/src/tests/intro.md @@ -0,0 +1,48 @@ +# Using the compiler testing framework + +The compiler has an extensive testing framework, masterminded by the +compiletest tool (sources in the [`src/tools/compiletest`]). This +section gives a brief overview of how the testing framework is setup, +and then gets into some of the details on +[how to run tests](running.html) as well as +[how to add new tests](adding.html). + +[`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest + +## Test suites + +The tests are located in the tree in the [`src/test`] +directory. Immediately within you will see a series of subdirectories +(e.g. `ui`, `run-make`, and so forth). Each of those directories is +called a **test suite** -- they house a group of tests that are run in +a distinct mode. + +[`src/test`]: https://github.com/rust-lang/rust/tree/master/src/test + +Here is a brief summary of the test suites as of this writing and what +they mean. In some cases, the test suites are linked to parts of the manual +that give more details. + +- [`ui`](ui.html) -- tests that check the exact stdout/stderr from compilation + and/or running the test +- `run-pass` -- tests that are expected to compile and execute successfully (no panics) + - `run-pass-valgrind` -- tests that ought to run with valrind +- `run-fail` -- tests that are expected to compile but then panic during execution +- `compile-fail` -- tests that are expected to fail compilation. +- `parse-fail` -- tests that are expected to fail to parse +- `pretty` -- tests targeting the Rust "pretty printer", which + generates valid Rust code from the AST +- `debuginfo` -- tests that run in gdb or lldb and query the debug info +- `codegen` -- tests that compile and then test the generated LLVM + code to make sure that optimizing we want are kicking in etc +- `mir-opt` -- tests that check parts of the generated MIR to make sure we are optimizing + etc. +- `incremental` -- tests for incremental compilation, checking that + when certain modifications are performed, we are able to reuse the + results from previous compilations. +- `run-make` -- tests that basically just execute a `Makefile`; the ultimate in flexibility + but annoying as all get out to write. +- `rustdoc` -- tests for rustdoc, making sure that the generated files contain + documentation for various entities etc +- `*-fulldeps` -- same as above, but indicates that the test depends on things other + than `libstd` (and hence those things must be built) diff --git a/src/tests/running.md b/src/tests/running.md new file mode 100644 index 000000000..8813862d8 --- /dev/null +++ b/src/tests/running.md @@ -0,0 +1,65 @@ +# Running tests + +You can run the tests using `x.py`. The most basic command -- which +you will almost never want to use! -- is as follows: + +``` +./x.py test +``` + +This will build the full stage 2 compiler and then run the whole test +suite. You probably don't want to do this very often, because it takes +a very long time, and anyway bors / travis will do it for you. (Often, +I will run this command in the background after opening a PR that I +think is done, but rarely otherwise. -nmatsakis) + +## Running a subset of the test suites + +When working on a specific PR, you will usually want to run a smaller +set of tests, and with a stage 1 build. For example, a good "smoke +test" that can be used after modifying rustc to see if things are +generally working correctly would be the following: + +```bash +./x.py test --stage 1 src/test/{ui,compile-fail,run-pass} +``` + +This will run the `ui`, `compile-fail`, and `run-pass` test suites, and +only with the stage 1 build. Of course, the choice of test suites is somewhat +arbitrary, and may not suit the task you are doing. For example, if you are hacking +on debuginfo, you may be better off with the debuginfo test suite: + +```bash +./x.py test --stage 1 src/test/debuginfo +``` + +**Warning:** Note that bors only runs the tests with the full stage 2 +build; therefore, while the tests **usually** work fine with stage 1, +there are some limitations. In particular, `./x.py test --stage 1` +(ie., + +## Running an individual test + +Another common thing that people want to do is to run an **individual +test**, often the test they are trying to fix. One way to do this is +to invoke `x.py` with the `--test-args` option: + +``` +./x.py test --stage 1 src/test/ui --test-args issue-1234 +``` + +Under the hood, the test runner invokes the standard rust test runner +(the same one you get with `#[test]`), so this command would wind up +filtering for tests that include "issue-1234" in the name. + +Often, though, it's easier to just run the test by hand. More tests are +just `rs` files, so you can do something like + +``` +rustc +stage1 src/test/ui/issue-1234.rs +``` + +This is much faster, but doesn't always work. For example, some tests +include directives that specify specific compiler flags, or which rely +on other crates, and they may not run the same without those options. + From 8c657b1f88ffbeacdc537debb3ef01186df74049 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 17 Feb 2018 09:41:13 -0500 Subject: [PATCH 088/648] add some notes on coding conventions and link to copyright notice --- src/SUMMARY.md | 1 + src/conventions.md | 63 +++++++++++++++++++++++++++++++++++++++++++++ src/tests/adding.md | 2 +- 3 files changed, 65 insertions(+), 1 deletion(-) create mode 100644 src/conventions.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 21976fbf7..981eb22c9 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -2,6 +2,7 @@ - [About this guide](./about-this-guide.md) - [How to build the compiler and run what you built](./how-to-build-and-run.md) +- [Coding conventions](./conventions.md) - [The compiler testing framework](./tests/intro.md) - [Running tests](./tests/running.md) - [Adding new tests](./tests/adding.md) diff --git a/src/conventions.md b/src/conventions.md new file mode 100644 index 000000000..14ef7119a --- /dev/null +++ b/src/conventions.md @@ -0,0 +1,63 @@ +rustc is slowly moving towards the [Rust standard coding style][fmt]; +at the moment, however, it follows a rather more *chaotic* +style. There are a few things that are always true. + +[fmt]: https://github.com/rust-lang-nursery/fmt-rfcs + +# The tidy script + +Running `./x.py test` begins with a "tidy" step. This tidy step checks +that your source files meet some basic conventions. + + + +## Copyright notice + +All files must begin with the following copyright notice: + +``` +// Copyright 2012-2013 The Rust Project Developers. See the COPYRIGHT +// file at the top-level directory of this distribution and at +// http://rust-lang.org/COPYRIGHT. +// +// Licensed under the Apache License, Version 2.0 or the MIT license +// , at your +// option. This file may not be copied, modified, or distributed +// except according to those terms. +``` + +The year at the top is not meaningful: copyright protections are in +fact automatic from the moment of authorship. We do not typically edit +the years on existing files. When creating a new file, you can use the +current year if you like, but you don't have to. + +## Line length + +Lines should be at most 100 characters. It's even better if you can +keep things to 80. + +**Ignoring the line length limit.** Sometimes -- in particular for +tests -- it can be necessary to exempt yourself from this limit. In +that case, you can add a comment towards the top of the file (after +the copyright notice) like so: + +``` +// ignore-tidy-linelength +``` + +## Tabs vs spaces + +Prefer 4-space indent. + +# Warnings and lints + +In general, Rust crates + +# Policy on using crates from crates.io + +It is allowed to use crates from crates.io, though external +dependencies should not be added gratuitously. All such crates must +have a suitably permissive license. There is an automatic check which +inspects the Cargo metadata to ensure this. + diff --git a/src/tests/adding.md b/src/tests/adding.md index 9ce8318db..ae247010d 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -9,7 +9,7 @@ To add a new test, the first thing you generally do is to create a file, typically a Rust source file. Test files have a particular structure: -- They always begin with the copyright notice; +- They always begin with the [copyright notice](../conventions.html#copyright); - then they should have some kind of [comment explaining what the test is about](#explanatory_comment); - next, they can have one or more [header commands](#header_commands), which are special From 14f74dff9154b6b64b0581283b0b4e0ac94860fc Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 17 Feb 2018 09:59:50 -0500 Subject: [PATCH 089/648] add some more conventions --- src/conventions.md | 106 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 95 insertions(+), 11 deletions(-) diff --git a/src/conventions.md b/src/conventions.md index 14ef7119a..93a07ad43 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -1,17 +1,24 @@ -rustc is slowly moving towards the [Rust standard coding style][fmt]; -at the moment, however, it follows a rather more *chaotic* -style. There are a few things that are always true. +This file offers some tips on the coding conventions for rustc. This +chapter covers [formatting](#f), [coding for correctness](#cc), +[using crates from crates.io](#cio), and some tips on +[structuring your PR for easy review](#er). -[fmt]: https://github.com/rust-lang-nursery/fmt-rfcs + -# The tidy script +# Formatting and the tidy script -Running `./x.py test` begins with a "tidy" step. This tidy step checks -that your source files meet some basic conventions. +rustc is slowly moving towards the [Rust standard coding style][fmt]; +at the moment, however, it follows a rather more *chaotic* style. We +do have some mandatory formatting conventions, which are automatically +enforced by a script we affectionately[^not_true] call the "tidy" +script. The tidy script runs automatically when you do `./x.py test`. + +[^not_true]: Secretly, I hate tidy. -nmatsakis +[fmt]: https://github.com/rust-lang-nursery/fmt-rfcs -## Copyright notice +### Copyright notice All files must begin with the following copyright notice: @@ -50,14 +57,91 @@ the copyright notice) like so: Prefer 4-space indent. -# Warnings and lints + + +# Coding for correctness + +Beyond formatting, there are a few other tips that are worth +following. + +## Prefer exhaustive matches + +Using `_` in a match is convenient, but it means that when new +variants are added to the enum, they may not get handled correctly. +Ask yourself: if a new variant were added to this enum, what's the +chance that it would want to use the `_` code, versus having some +other treatment? Unless the answer is "low", then prefer an +exhaustive match. (The same advice applies to `if let` and `while +let`, which are effectively tests for a single variant.) -In general, Rust crates +## Use "TODO" comments for things you don't want to forget -# Policy on using crates from crates.io +As a useful tool to yourself, you can insert a `// TODO` comment +for something that you want to get back to before you land your PR: + +```rust,ignore +fn do_something() { + if something_else { + unimplemented!(): // TODO write this + } +} +``` + +The tidy script will report an error for a `// TODO` comment, so this +code would not be able to land until the TODO is fixed (or removed). + +This can also be useful in a PR as a way to signal from one commit that you are +leaving a bug that a later commit will fix: + +```rust,ignore +if foo { + return true; // TODO wrong, but will be fixed in a later commit +} +``` + + + +# Using crates from crates.io It is allowed to use crates from crates.io, though external dependencies should not be added gratuitously. All such crates must have a suitably permissive license. There is an automatic check which inspects the Cargo metadata to ensure this. + + +# How to structure your PR + +How you prepare the commits in your PR can make a big difference for the reviewer. +Here are some tips. + +**Isolate "pure refactorings" into their own commit.** For example, if +you rename a method, then put that rename into its own commit, along +with the renames of all the uses. + +**More commits is usually better.** If you are doing a large change, +it's almost always better to break it up into smaller steps that can +be independently understood. The one thing to be aware of is that if +you introduce some code following one strategy, then change it +dramatically (versus adding to it) in a later commit, that +'back-and-forth' can be confusing. + +**If you run rustfmt and the file was not already formatted, isolate +that into its own commit.** This is really the same as the previous +rule, but it's worth highlighting. It's ok to rustfmt files, but since +we do not currently run rustfmt all the time, that can introduce a lot +of noise into your commit. Please isolate that into its own +commit. This also makes rebases a lot less painful, since rustfmt +tends to cause a lot of merge conflicts, and having those isolated +into their own commit makes htem easier to resolve. + +**No merges.** We do not allow merge commits into our history, other +than those by bors. If you get a merge conflict, rebase instead via a +command like `git rebase -i rust-lang/master` (presuming you use the +name `rust-lang` for your remote). + +**Individual commits do not have to build (but it's nice).** We do not +require that every intermediate commit successfully builds -- we only +expect to be able to bisect at a PR level. However, if you *can* make +individual commits build, that is always helpful. + From 346cbab485283a23f3ce3004e2f15d14fd302eaf Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 17 Feb 2018 10:17:23 -0500 Subject: [PATCH 090/648] give some advice about which test suite to use --- src/tests/adding.md | 71 ++++++++++++++++++++++++++++++++------------- src/tests/intro.md | 15 +++++----- 2 files changed, 59 insertions(+), 27 deletions(-) diff --git a/src/tests/adding.md b/src/tests/adding.md index ae247010d..c2a5ef43f 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -1,9 +1,9 @@ # Adding new tests **In general, we expect every PR that fixes a bug in rustc to come -accompanied with a regression test of some kind.** This test should -fail in master but pass after the PR. These tests are really useful -for preventing us from repeating the mistakes of the past. +accompanied by a regression test of some kind.** This test should fail +in master but pass after the PR. These tests are really useful for +preventing us from repeating the mistakes of the past. To add a new test, the first thing you generally do is to create a file, typically a Rust source file. Test files have a particular @@ -20,18 +20,36 @@ structure: Depending on the test suite, there may be some other details to be aware of: - For [the `ui` test suite](#ui), you need to generate reference output files. - + +## What kind of test should I add? + +It can be difficult to know what kind of test to use. Here are some +rough heuristics: + +- Some tests have specialized needs: + - need to run gdb or lldb? use the `debuginfo` test suite + - need to inspect LLVM IR or MIR IR? use the `codegen` or `mir-opt` test suites + - need to run rustdoc? Prefer a `rustdoc` test + - need to inspect the resulting binary in some way? Then use `run-make` +- For most other things, [a `ui` (or `ui-fulldeps`) test](#ui) is to be preferred: + - `ui` tests subsume both run-pass, compile-fail, and parse-fail tests + - in the case of warnings or errors, `ui` tests capture the full output, + which makes it easier to review but also helps prevent "hidden" regressions + in the output + ## Naming your test We have not traditionally had a lot of structure in the names of tests. Moreover, for a long time, the rustc test runner did not support subdirectories (it now does), so test suites like -`src/test/run-pass` have a huge mess of files in them. This is not +[`src/test/run-pass`] have a huge mess of files in them. This is not considered an ideal setup. +[`src/test/run-pass`]: https://github.com/rust-lang/rust/tree/master/src/test/run-pass/ + For regression tests -- basically, some random snippet of code that came in from the internet -- we often just name the test after the -issue. For example, `src/test/run-pass/issue-1234.rs`. If possible, +issue. For example, `src/test/run-pass/issue-12345.rs`. If possible, though, it is better if you can put the test into a directory that helps identify what piece of code is being tested here (e.g., `borrowck/issue-12345.rs` is much better), or perhaps give it a more @@ -204,21 +222,31 @@ customized to a revision are error patterns and compiler flags. The UI tests are intended to capture the compiler's complete output, so that we can test all aspects of the presentation. They work by -compiling a file (e.g., `ui/hello_world/main.rs`), capturing the output, -and then applying some normalization (see below). This normalized -result is then compared against reference files named -`ui/hello_world/main.stderr` and `ui/hello_world/main.stdout`. If either of -those files doesn't exist, the output must be empty. If the test run -fails, we will print out the current output, but it is also saved in +compiling a file (e.g., [`ui/hello_world/main.rs`][hw-main]), +capturing the output, and then applying some normalization (see +below). This normalized result is then compared against reference +files named `ui/hello_world/main.stderr` and +`ui/hello_world/main.stdout`. If either of those files doesn't exist, +the output must be empty (that is actually the case for +[this particular test][hw]). If the test run fails, we will print out +the current output, but it is also saved in `build//test/ui/hello_world/main.stdout` (this path is -printed as part of the test failure message), so you can run `diff` and -so forth. +printed as part of the test failure message), so you can run `diff` +and so forth. -Normally, the test-runner checks that UI tests fail compilation. If you want -to do a UI test for code that *compiles* (e.g. to test warnings, or if you -have a collection of tests, only some of which error out), you can use the -`// must-compile-successfully` header command to have the test runner instead -check that the test compiles successfully. +[hw-main]: https://github.com/rust-lang/rust/blob/master/src/test/ui/hello_world/main.rs +[hw]: https://github.com/rust-lang/rust/blob/master/src/test/ui/hello_world/ + +### Tests that do not result in compile errors + +By default, a UI test is expected **not to compile** (in which case, +it should contain at least one `//~ ERROR` annotation). However, you +can also make UI tests where compilation is expected to succeed, and +you can even run the resulting program. Just add one of the following +[header commands](#header_commands): + +- `// must-compile-successfully` -- compilation should succeed but do not run the resulting binary +- `// run-pass` -- compilation should succeed and we should run the resulting binary ### Editing and updating the reference files @@ -265,7 +293,10 @@ The corresponding reference file will use the normalized output to test both ... ``` -Please see `ui/transmute/main.rs` and `.stderr` for a concrete usage example. +Please see [`ui/transmute/main.rs`][mrs] and [`main.stderr`][] for a concrete usage example. + +[mrs]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.rs +[`main.stderr`]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.stderr Besides `normalize-stderr-32bit` and `-64bit`, one may use any target information or stage supported by `ignore-X` here as well (e.g. diff --git a/src/tests/intro.md b/src/tests/intro.md index c22916dca..334042a41 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -23,7 +23,7 @@ Here is a brief summary of the test suites as of this writing and what they mean. In some cases, the test suites are linked to parts of the manual that give more details. -- [`ui`](ui.html) -- tests that check the exact stdout/stderr from compilation +- [`ui`](adding.html#) -- tests that check the exact stdout/stderr from compilation and/or running the test - `run-pass` -- tests that are expected to compile and execute successfully (no panics) - `run-pass-valgrind` -- tests that ought to run with valrind @@ -34,15 +34,16 @@ that give more details. generates valid Rust code from the AST - `debuginfo` -- tests that run in gdb or lldb and query the debug info - `codegen` -- tests that compile and then test the generated LLVM - code to make sure that optimizing we want are kicking in etc -- `mir-opt` -- tests that check parts of the generated MIR to make sure we are optimizing - etc. + code to make sure that the optimizations we want are taking effect. +- `mir-opt` -- tests that check parts of the generated MIR to make + sure we are building things correctly or doing the optimizations we + expect. - `incremental` -- tests for incremental compilation, checking that when certain modifications are performed, we are able to reuse the results from previous compilations. -- `run-make` -- tests that basically just execute a `Makefile`; the ultimate in flexibility - but annoying as all get out to write. +- `run-make` -- tests that basically just execute a `Makefile`; the + ultimate in flexibility but quite annoying to write. - `rustdoc` -- tests for rustdoc, making sure that the generated files contain - documentation for various entities etc + the expected documentation. - `*-fulldeps` -- same as above, but indicates that the test depends on things other than `libstd` (and hence those things must be built) From a8bdd52e56e18d95cb9da083571fa9c4f2bc7ad8 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 17 Feb 2018 10:18:27 -0500 Subject: [PATCH 091/648] complete sentence --- src/tests/running.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/tests/running.md b/src/tests/running.md index 8813862d8..59a615dab 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -35,8 +35,8 @@ on debuginfo, you may be better off with the debuginfo test suite: **Warning:** Note that bors only runs the tests with the full stage 2 build; therefore, while the tests **usually** work fine with stage 1, -there are some limitations. In particular, `./x.py test --stage 1` -(ie., +there are some limitations. In particular, the stage1 compiler doesn't +work well with procedural macros or custom derive tests. ## Running an individual test From 2e04d6444a6606b46437adeb6dc66cf6f3b61608 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 17 Feb 2018 10:21:27 -0500 Subject: [PATCH 092/648] link to brson's blog post --- src/tests/intro.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/tests/intro.md b/src/tests/intro.md index 334042a41..1e4a2bed3 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -47,3 +47,11 @@ that give more details. the expected documentation. - `*-fulldeps` -- same as above, but indicates that the test depends on things other than `libstd` (and hence those things must be built) + +## Further reading + +The following blog posts may also be of interest: + +- brson's classic ["How Rust is tested"][howtest] + +[howtest]: https://brson.github.io/2017/07/10/how-rust-is-tested From b502a6a3fca9dc8161e858a42d7a2f7fa3af2ef4 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 17 Feb 2018 10:21:32 -0500 Subject: [PATCH 093/648] fix typo --- src/tests/running.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/tests/running.md b/src/tests/running.md index 59a615dab..98d590671 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -52,7 +52,7 @@ Under the hood, the test runner invokes the standard rust test runner (the same one you get with `#[test]`), so this command would wind up filtering for tests that include "issue-1234" in the name. -Often, though, it's easier to just run the test by hand. More tests are +Often, though, it's easier to just run the test by hand. Most tests are just `rs` files, so you can do something like ``` From caea7ae5adfa6edf9292505fc0c912265441698d Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 17 Feb 2018 10:27:57 -0500 Subject: [PATCH 094/648] mention tidy --- src/conventions.md | 4 ++-- src/tests/running.md | 7 +++++++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/src/conventions.md b/src/conventions.md index 93a07ad43..3ed58bf49 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -1,9 +1,9 @@ This file offers some tips on the coding conventions for rustc. This -chapter covers [formatting](#f), [coding for correctness](#cc), +chapter covers [formatting](#formatting), [coding for correctness](#cc), [using crates from crates.io](#cio), and some tips on [structuring your PR for easy review](#er). - + # Formatting and the tidy script diff --git a/src/tests/running.md b/src/tests/running.md index 98d590671..dc9de0019 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -13,6 +13,13 @@ a very long time, and anyway bors / travis will do it for you. (Often, I will run this command in the background after opening a PR that I think is done, but rarely otherwise. -nmatsakis) +## Tidy + +When you run the full suite of tests via `./x.py test`, the first +thing that executes is a "tidy suite" that checks for long lines and +other formatting conventions. There is more information in the +[section on coding conventions](../conventions.html#formatting). + ## Running a subset of the test suites When working on a specific PR, you will usually want to run a smaller From 7f8364d6f92aacd657867599cab4724c3ef385aa Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 17 Feb 2018 10:31:48 -0500 Subject: [PATCH 095/648] correct links (mdbook seems to want them relative to the book root) --- src/how-to-build-and-run.md | 2 +- src/tests/adding.md | 2 +- src/tests/intro.md | 6 +++--- src/tests/running.md | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 2657da84c..7b6088c7c 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -132,6 +132,6 @@ Here are a few other useful x.py commands. We'll cover some of them in detail in - `./x.py clean` – clean up the build directory (`rm -rf build` works too, but then you have to rebuild LLVM) - `./x.py build --stage 1` – builds everything using the stage 1 compiler, not just up to libstd - `./x.py build` – builds the stage2 compiler -- Running tests (see the section [running tests](./running-tests.html) for more details): +- Running tests (see the [section on running tests](./tests/running.html) for more details): - `./x.py test --stage 1 src/libstd` – runs the `#[test]` tests from libstd - `./x.py test --stage 1 src/test/run-pass` – runs the `run-pass` test suite diff --git a/src/tests/adding.md b/src/tests/adding.md index c2a5ef43f..276189f59 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -9,7 +9,7 @@ To add a new test, the first thing you generally do is to create a file, typically a Rust source file. Test files have a particular structure: -- They always begin with the [copyright notice](../conventions.html#copyright); +- They always begin with the [copyright notice](./conventions.html#copyright); - then they should have some kind of [comment explaining what the test is about](#explanatory_comment); - next, they can have one or more [header commands](#header_commands), which are special diff --git a/src/tests/intro.md b/src/tests/intro.md index 1e4a2bed3..615673206 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -4,8 +4,8 @@ The compiler has an extensive testing framework, masterminded by the compiletest tool (sources in the [`src/tools/compiletest`]). This section gives a brief overview of how the testing framework is setup, and then gets into some of the details on -[how to run tests](running.html) as well as -[how to add new tests](adding.html). +[how to run tests](./tests/running.html#ui) as well as +[how to add new tests](./tests/adding.html). [`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest @@ -23,7 +23,7 @@ Here is a brief summary of the test suites as of this writing and what they mean. In some cases, the test suites are linked to parts of the manual that give more details. -- [`ui`](adding.html#) -- tests that check the exact stdout/stderr from compilation +- [`ui`](./tests/adding.html#ui) -- tests that check the exact stdout/stderr from compilation and/or running the test - `run-pass` -- tests that are expected to compile and execute successfully (no panics) - `run-pass-valgrind` -- tests that ought to run with valrind diff --git a/src/tests/running.md b/src/tests/running.md index dc9de0019..602aa51d9 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -18,7 +18,7 @@ think is done, but rarely otherwise. -nmatsakis) When you run the full suite of tests via `./x.py test`, the first thing that executes is a "tidy suite" that checks for long lines and other formatting conventions. There is more information in the -[section on coding conventions](../conventions.html#formatting). +[section on coding conventions](./conventions.html#formatting). ## Running a subset of the test suites From c7c80abb0be383c449ab2304613ba1c5f2d50e50 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 17 Feb 2018 10:32:35 -0500 Subject: [PATCH 096/648] remove humorous footnote which is not supported --- src/conventions.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/conventions.md b/src/conventions.md index 3ed58bf49..f7ef86f11 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -10,10 +10,9 @@ chapter covers [formatting](#formatting), [coding for correctness](#cc), rustc is slowly moving towards the [Rust standard coding style][fmt]; at the moment, however, it follows a rather more *chaotic* style. We do have some mandatory formatting conventions, which are automatically -enforced by a script we affectionately[^not_true] call the "tidy" -script. The tidy script runs automatically when you do `./x.py test`. +enforced by a script we affectionately call the "tidy" script. The +tidy script runs automatically when you do `./x.py test`. -[^not_true]: Secretly, I hate tidy. -nmatsakis [fmt]: https://github.com/rust-lang-nursery/fmt-rfcs From 4af21aad8960a7126fc5bbc03084475d2eb90d0b Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Sat, 17 Feb 2018 20:24:48 -0800 Subject: [PATCH 097/648] Force mdbook update on CI if there is a version mismatch. Fixes #58. --- ci/install.sh | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/ci/install.sh b/ci/install.sh index 99c85986c..a5d87a6bc 100644 --- a/ci/install.sh +++ b/ci/install.sh @@ -6,9 +6,15 @@ function cargo_install() { local version=$2 if command -v $name >/dev/null 2>&1; then - echo "$name is already installed at $(command -v $name)" + local installed_version=`$name --version | sed -E 's/[a-zA-Z_-]+ v?//g'` + if [ "$installed_version" == "$version" ]; then + echo "$name $version is already installed at $(command -v $name)" + else + echo "Forcing install $name $version" + cargo install $name --version $version --force + fi else - echo "Installing $name" + echo "Installing $name $version" cargo install $name --version $version fi } From b9bc44dc5c891df210923fc6133478be8bc0b3e7 Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Sat, 17 Feb 2018 20:35:57 -0800 Subject: [PATCH 098/648] Fix Github Pages path. The newer version of mdBook uses a different path for the html output. --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index f17a947d4..5ca97ba79 100644 --- a/.travis.yml +++ b/.travis.yml @@ -16,6 +16,6 @@ deploy: provider: pages skip-cleanup: true github-token: $GITHUB_TOKEN - local-dir: book + local-dir: book/html on: branch: master From 84c2cd5f605caa55792101a85e00537d48077468 Mon Sep 17 00:00:00 2001 From: Brad Gibson Date: Tue, 20 Feb 2018 15:32:17 -0800 Subject: [PATCH 099/648] added compiletest to summary.md; linked to existing resources for additional information --- src/SUMMARY.md | 1 + src/compiletest.md | 6 ++++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 6a3907551..1926a58cf 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -2,6 +2,7 @@ - [About this guide](./about-this-guide.md) - [How to build the compiler and run what you built](./how-to-build-and-run.md) + - [Using `compiletest` + commands to control test execution](./compiletest.md) - [Using the compiler testing framework](./running-tests.md) - [How to add new header commands to `compiletest`](./compiletest.md) - [Walkthrough: a typical contribution](./walkthrough.md) diff --git a/src/compiletest.md b/src/compiletest.md index 0b10cfbb7..e080e6ce7 100644 --- a/src/compiletest.md +++ b/src/compiletest.md @@ -6,7 +6,8 @@ configure behavior and expected results of both individual and groups of tests. `compiletest` tests may check test code for success, for failure or in some cases, even failure to compile. Tests are typically organized as a Rust source file with annotations in comments before and/or within the test code, which serve to -direct `compiletest` on if or how to run the test, what behavior to expect, and more. +direct `compiletest` on if or how to run the test, what behavior to expect, and more. If you are unfamiliar with the compiler +testing framework, see [`this chapter`](https://github.com/rust-lang-nursery/rustc-guide/blob/master/src/tests/intro.md) for additional background. The tests themselves are typically (but not always) organized into "suites"--for example, `run-pass`, a folder representing tests that should succeed, `run-fail`, a folder holding tests that should compile successfully, but return @@ -16,8 +17,9 @@ introduction to the different suites of compiler tests along with details about [here, at Brian Anderson's blog](https://brson.github.io/2017/07/10/how-rust-is-tested#s-ct). ## Adding a new test file -Simply create your new test in the appropriate location under [src/test](https://github.com/rust-lang/rust/tree/master/src/test). No registration of test files is necessary as +Briefly, simply create your new test in the appropriate location under [src/test](https://github.com/rust-lang/rust/tree/master/src/test). No registration of test files is necessary as `compiletest` will scan the [src/test](https://github.com/rust-lang/rust/tree/master/src/test) subfolder recursively, and will execute any Rust source files it finds as tests. +See [`Adding new tests`](https://github.com/rust-lang-nursery/rustc-guide/blob/master/src/tests/adding.md) for a complete guide on how to adding new tests. ## Header Commands Source file annotations which appear in comments near the top of the source file *before* any test code are known as header From 572eaa6e5dffabe6712787676f57563b0f4aa29e Mon Sep 17 00:00:00 2001 From: Brad Gibson Date: Wed, 21 Feb 2018 07:44:07 -0800 Subject: [PATCH 100/648] cleaned up some stragglers --- src/compiletest.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/src/compiletest.md b/src/compiletest.md index e080e6ce7..854796adb 100644 --- a/src/compiletest.md +++ b/src/compiletest.md @@ -14,7 +14,7 @@ representing tests that should succeed, `run-fail`, a folder holding tests that a failure (non-zero status), `compile-fail`, a folder holding tests that should fail to compile, and many more. The various suites are defined in [src/tools/compiletest/src/common.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs) in the `pub struct Config` declaration. And a very good introduction to the different suites of compiler tests along with details about them can be found -[here, at Brian Anderson's blog](https://brson.github.io/2017/07/10/how-rust-is-tested#s-ct). +in [`Adding new tests`](https://github.com/rust-lang-nursery/rustc-guide/blob/master/src/tests/adding.md). ## Adding a new test file Briefly, simply create your new test in the appropriate location under [src/test](https://github.com/rust-lang/rust/tree/master/src/test). No registration of test files is necessary as @@ -25,14 +25,11 @@ See [`Adding new tests`](https://github.com/rust-lang-nursery/rustc-guide/blob/m Source file annotations which appear in comments near the top of the source file *before* any test code are known as header commands. These commands can instruct `compiletest` to ignore this test, set expectations on whether it is expected to succeed at compiling, or what the test's return code is expected to be. Header commands (and their inline counterparts, -Error Info commands) are described more fully -[here](https://github.com/rust-lang/rust/blob/master/src/test/COMPILER_TESTS.md). +Error Info commands) are described more fully [here](https://github.com/rust-lang-nursery/rustc-guide/blob/master/src/tests/adding.md#header-commands-configuring-rustc). ### Adding a new header command -Header commands are defined in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs). At a high level, dozens of test properties are defined -in the `TestProps` struct. These are all set to default values in the struct's `impl` block and any test can override this -default value by specifying the property as header command as a comment (`//`) in the test source file, before any source -code. +Header commands are defined in the `TestProps` struct in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs). At a high level, there are dozens of test properties are defined here, all set to default values in the `TestProp` struct's `impl` block. Any test can override this +default value by specifying the property in question as header command as a comment (`//`) in the test source file, before any source code. #### Using a header command Here is an example, specifying the `must-compile-successfully` header command, which takes no arguments, followed by the @@ -140,7 +137,7 @@ is that `compiletest` expects the failure code defined by the header command inv value. Although specific to `failure-status` (as every header command will have a different implementation in order to invoke -behavior change) perhaps it is helpful to see the behavior hange implementation of one case, simply as an example. To implement `failure-status`, the `check_correct_failure_status()` function found in the `TestCx` implementation block, +behavior change) perhaps it is helpful to see the behavior change implementation of one case, simply as an example. To implement `failure-status`, the `check_correct_failure_status()` function found in the `TestCx` implementation block, located in [src/tools/compiletest/src/runtest.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs), was modified as per below: ```diff @@ -295,11 +295,14 @@ impl<'test> TestCx<'test> { From b1c4662b011da780b9028e78016296b183ddc8fd Mon Sep 17 00:00:00 2001 From: Brad Gibson Date: Wed, 21 Feb 2018 10:44:23 -0800 Subject: [PATCH 101/648] changed to relative links, where possible --- src/compiletest.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/src/compiletest.md b/src/compiletest.md index 854796adb..f07d3065a 100644 --- a/src/compiletest.md +++ b/src/compiletest.md @@ -7,28 +7,27 @@ configure behavior and expected results of both individual and groups of tests. `compiletest` tests may check test code for success, for failure or in some cases, even failure to compile. Tests are typically organized as a Rust source file with annotations in comments before and/or within the test code, which serve to direct `compiletest` on if or how to run the test, what behavior to expect, and more. If you are unfamiliar with the compiler -testing framework, see [`this chapter`](https://github.com/rust-lang-nursery/rustc-guide/blob/master/src/tests/intro.md) for additional background. +testing framework, see [`this chapter`](./tests/intro.html) for additional background. The tests themselves are typically (but not always) organized into "suites"--for example, `run-pass`, a folder representing tests that should succeed, `run-fail`, a folder holding tests that should compile successfully, but return a failure (non-zero status), `compile-fail`, a folder holding tests that should fail to compile, and many more. The various suites are defined in [src/tools/compiletest/src/common.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs) in the `pub struct Config` declaration. And a very good -introduction to the different suites of compiler tests along with details about them can be found -in [`Adding new tests`](https://github.com/rust-lang-nursery/rustc-guide/blob/master/src/tests/adding.md). +introduction to the different suites of compiler tests along with details about them can be found in [`Adding new tests`](./tests/adding.html). ## Adding a new test file Briefly, simply create your new test in the appropriate location under [src/test](https://github.com/rust-lang/rust/tree/master/src/test). No registration of test files is necessary as `compiletest` will scan the [src/test](https://github.com/rust-lang/rust/tree/master/src/test) subfolder recursively, and will execute any Rust source files it finds as tests. -See [`Adding new tests`](https://github.com/rust-lang-nursery/rustc-guide/blob/master/src/tests/adding.md) for a complete guide on how to adding new tests. +See [`Adding new tests`](./tests/adding.html) for a complete guide on how to adding new tests. ## Header Commands Source file annotations which appear in comments near the top of the source file *before* any test code are known as header commands. These commands can instruct `compiletest` to ignore this test, set expectations on whether it is expected to succeed at compiling, or what the test's return code is expected to be. Header commands (and their inline counterparts, -Error Info commands) are described more fully [here](https://github.com/rust-lang-nursery/rustc-guide/blob/master/src/tests/adding.md#header-commands-configuring-rustc). +Error Info commands) are described more fully [here](./tests/adding.html#header-commands-configuring-rustc). ### Adding a new header command -Header commands are defined in the `TestProps` struct in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs). At a high level, there are dozens of test properties are defined here, all set to default values in the `TestProp` struct's `impl` block. Any test can override this +Header commands are defined in the `TestProps` struct in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs). At a high level, there are dozens of test properties defined here, all set to default values in the `TestProp` struct's `impl` block. Any test can override this default value by specifying the property in question as header command as a comment (`//`) in the test source file, before any source code. #### Using a header command From 6e1eccd30c7d28ae07d264c9ed2aa1139b40c0fe Mon Sep 17 00:00:00 2001 From: Oliver Schneider Date: Tue, 6 Feb 2018 19:10:04 +0100 Subject: [PATCH 102/648] Add some documentation for const eval and related topics --- src/SUMMARY.md | 3 + src/const-eval.md | 37 +++++++++++ src/glossary.md | 3 + src/miri.md | 142 ++++++++++++++++++++++++++++++++++++++++ src/param_env.md | 30 +++++++++ src/trait-resolution.md | 2 +- 6 files changed, 216 insertions(+), 1 deletion(-) create mode 100644 src/const-eval.md create mode 100644 src/miri.md create mode 100644 src/param_env.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index cb15c2c71..ea2a6cd98 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -23,5 +23,8 @@ - [MIR construction](./mir-construction.md) - [MIR borrowck](./mir-borrowck.md) - [MIR optimizations](./mir-optimizations.md) +- [Constant evaluation](./const-eval.md) + - [miri const evaluator](./miri.md) +- [Parameter Environments](./param_env.md) - [Generating LLVM IR](./trans.md) - [Glossary](./glossary.md) diff --git a/src/const-eval.md b/src/const-eval.md new file mode 100644 index 000000000..3fb65216a --- /dev/null +++ b/src/const-eval.md @@ -0,0 +1,37 @@ +# Constant Evaluation + +Constant evaluation is the process of computing values at compile time. For a +specific item (constant/static/array length) this happens after the MIR for the +item is borrow-checked and optimized. In many cases trying to const evaluate an +item will trigger the computation of its MIR for the first time. + +Prominent examples are + +* The initializer of a `static` +* Array length + * needs to be known to reserve stack or heap space +* Enum variant discriminants + * needs to be known to prevent two variants from having the same discriminant +* Patterns + * need to be known to check for overlapping patterns + +Additionally constant evaluation can be used to reduce the workload or binary +size at runtime by precomputing complex operations at compiletime and only +storing the result. + +Constant evaluation can be done by calling the `const_eval` query of `TyCtxt`. + +The `const_eval` query takes a [`ParamEnv`](./param_env.html) of environment in +which the constant is evaluated (e.g. the function within which the constant is +used) and a `GlobalId`. The `GlobalId` is made up of an +`Instance` referring to a constant or static or of an +`Instance` of a function and an index into the function's `Promoted` table. + +Constant evaluation returns a `Result` with either the error, or the simplest +representation of the constant. "simplest" meaning if it is representable as an +integer or fat pointer, it will directly yield the value (via `Value::ByVal` or +`Value::ByValPair`), instead of referring to the [`miri`](./miri.html) virtual +memory allocation (via `Value::ByRef`). This means that the `const_eval` +function cannot be used to create miri-pointers to the evaluated constant or +static. If you need that, you need to directly work with the functions in +[src/librustc_mir/interpret/const_eval.rs](https://github.com/rust-lang/rust/blob/master/src/librustc_mir/interpret/const_eval.rs). diff --git a/src/glossary.md b/src/glossary.md index a22c49632..34b4bd446 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -18,6 +18,9 @@ generics | the set of generic type parameters defined on a type ICE | internal compiler error. When the compiler crashes. ICH | incremental compilation hash. ICHs are used as fingerprints for things such as HIR and crate metadata, to check if changes have been made. This is useful in incremental compilation to see if part of a crate has changed and should be recompiled. infcx | the inference context (see `librustc/infer`) +MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) +miri | an interpreter for MIR used for constant evaluation ([see more](./miri.html)) +obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) local crate | the crate currently being compiled. MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) node-id or NodeId | an index identifying a particular node in the AST or HIR; gradually being phased out and replaced with `HirId`. diff --git a/src/miri.md b/src/miri.md new file mode 100644 index 000000000..4b0a600be --- /dev/null +++ b/src/miri.md @@ -0,0 +1,142 @@ +# Miri + +Miri (**MIR** **I**nterpreter) is a virtual machine for executing MIR without +compiling to machine code. It is usually invoked via `tcx.const_eval`. + +If you start out with a constant + +```rust +const FOO: usize = 1 << 12; +``` + +rustc doesn't actually invoke anything until the constant is either used or +placed into metadata. + +Once you have a use-site like + +```rust +type Foo = [u8; FOO - 42]; +``` + +The compiler needs to figure out the length of the array before being able to +create items that use the type (locals, constants, function arguments, ...). + +To obtain the (in this case empty) parameter environment, one can call +`let param_env = tcx.param_env(length_def_id);`. The `GlobalId` needed is + +```rust +let gid = GlobalId { + promoted: None, + instance: Instance::mono(length_def_id), +}; +``` + +Invoking `tcx.const_eval(param_env.and(gid))` will now trigger the creation of +the MIR of the array length expression. The MIR will look something like this: + +```mir +const Foo::{{initializer}}: usize = { + let mut _0: usize; // return pointer + let mut _1: (usize, bool); + + bb0: { + _1 = CheckedSub(const Unevaluated(FOO, Slice([])), const 42usize); + assert(!(_1.1: bool), "attempt to subtract with overflow") -> bb1; + } + + bb1: { + _0 = (_1.0: usize); + return; + } +} +``` + +Before the evaluation, a virtual memory location (in this case essentially a +`vec![u8; 4]` or `vec![u8; 8]`) is created for storing the evaluation result. + +At the start of the evaluation, `_0` and `_1` are +`Value::ByVal(PrimVal::Undef)`. When the initialization of `_1` is invoked, the +value of the `FOO` constant is required, and triggers another call to +`tcx.const_eval`, which will not be shown here. If the evaluation of FOO is +successful, 42 will be subtracted by its value `4096` and the result stored in +`_1` as `Value::ByValPair(PrimVal::Bytes(4054), PrimVal::Bytes(0))`. The first +part of the pair is the computed value, the second part is a bool that's true if +an overflow happened. + +The next statement asserts that said boolean is `0`. In case the assertion +fails, its error message is used for reporting a compile-time error. + +Since it does not fail, `Value::ByVal(PrimVal::Bytes(4054))` is stored in the +virtual memory was allocated before the evaluation. `_0` always refers to that +location directly. + +After the evaluation is done, the virtual memory allocation is interned into the +`TyCtxt`. Future evaluations of the same constants will not actually invoke +miri, but just extract the value from the interned allocation. + +The `tcx.const_eval` function has one additional feature: it will not return a +`ByRef(interned_allocation_id)`, but a `ByVal(computed_value)` if possible. This +makes using the result much more convenient, as no further queries need to be +executed in order to get at something as simple as a `usize`. + +## Datastructures + +Miri's core datastructures can be found in +[librustc/mir/interpret](https://github.com/rust-lang/rust/blob/master/src/librustc/mir/interpret). +This is mainly the error enum and the `Value` and `PrimVal` types. A `Value` can +be either `ByVal` (a single `PrimVal`), `ByValPair` (two `PrimVal`s, usually fat +pointers or two element tuples) or `ByRef`, which is used for anything else and +refers to a virtual allocation. These allocations can be accessed via the +methods on `tcx.interpret_interner`. + +If you are expecting a numeric result, you can use `unwrap_u64` (panics on +anything that can't be representad as a `u64`) or `to_raw_bits` which results +in an `Option` yielding the `ByVal` if possible. + +## Allocations + +A miri allocation is either a byte sequence of the memory or an `Instance` in +the case of function pointers. Byte sequences can additionally contain +relocations that mark a group of bytes as a pointer to another allocation. The +actual bytes at the relocation refer to the offset inside the other allocation. + +These allocations exist so that references and raw pointers have something to +point to. There is no global linear heap in which things are allocated, but each +allocation (be it for a local variable, a static or a (future) heap allocation) +gets its own little memory with exactly the required size. So if you have a +pointer to an allocation for a local variable `a`, there is no possible (no +matter how unsafe) operation that you can do that would ever change said pointer +to a pointer to `b`. + +## Interpretation + +Although the main entry point to constant evaluation is the `tcx.const_eval` +query, there are additional functions in +[librustc_mir/interpret/const_eval.rs](https://github.com/rust-lang/rust/blob/master/src/librustc_mir/interpret/const_eval.rs) +that allow accessing the fields of a `Value` (`ByRef` or otherwise). You should +never have to access an `Allocation` directly except for translating it to the +compilation target (at the moment just LLVM). + +Miri starts by creating a virtual stack frame for the current constant that is +being evaluated. There's essentially no difference between a constant and a +function with no arguments, except that constants do not allow local (named) +variables at the time of writing this guide. + +A stack frame is defined by the `Frame` type in +[librustc_mir/interpret/eval_context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc_mir/interpret/eval_context.rs) +and contains all the local +variables memory (`None` at the start of evaluation). Each frame refers to the +evaluation of either the root constant or subsequent calls to `const fn`. The +evaluation of another constant simply calls `tcx.const_eval`, which produces an +entirely new and independent stack frame. + +The frames are just a `Vec`, there's no way to actually refer to a +`Frame`'s memory even if horrible shenigans are done via unsafe code. The only +memory that can be referred to are `Allocation`s. + +Miri now calls the `step` method (in +[librustc_mir/interpret/step.rs](https://github.com/rust-lang/rust/blob/master/src/librustc_mir/interpret/step.rs) +) until it either returns an error or has no further statements to execute. Each +statement will now initialize or modify the locals or the virtual memory +referred to by a local. This might require evaluating other constants or +statics, which just recursively invokes `tcx.const_eval`. diff --git a/src/param_env.md b/src/param_env.md new file mode 100644 index 000000000..2474886ef --- /dev/null +++ b/src/param_env.md @@ -0,0 +1,30 @@ +# Parameter Environment + +When working with associated and/or or generic items (types, constants, +functions/methods) it is often relevant to have more information about the +`Self` or generic parameters. Trait bounds and similar information is encoded in +the `ParamEnv`. Often this is not enough information to obtain things like the +type's `Layout`, but you can do all kinds of other checks on it (e.g. whether a +type implements `Copy`) or you can evaluate an associated constant whose value +does not depend on anything from the parameter environment. + +For example if you have a function + +```rust +fn foo(t: T) { +} +``` + +the parameter environment for that function is `[T: Copy]`. This means any +evaluation within this function will, when accessing the type `T`, know about +its `Copy` bound via the parameter environment. + +Although you can obtain a valid `ParamEnv` for any item via +`tcx.param_env(def_id)`, this `ParamEnv` can be too generic for your use case. +Using the `ParamEnv` from the surrounding context can allow you to evaluate more +things. + +Another great thing about `ParamEnv` is that you can use it to bundle the thing +depending on generic parameters (e.g. a `Ty`) by calling `param_env.and(ty)`. +This will produce a `ParamEnvAnd`, making clear that you should probably not +be using the inner value without taking care to also use the `ParamEnv`. diff --git a/src/trait-resolution.md b/src/trait-resolution.md index f12ce3b4b..10a5539e4 100644 --- a/src/trait-resolution.md +++ b/src/trait-resolution.md @@ -433,7 +433,7 @@ before, and hence the cache lookup would succeed, yielding One subtle interaction is that the results of trait lookup will vary depending on what where clauses are in scope. Therefore, we actually have *two* caches, a local and a global cache. The local cache is -attached to the `ParamEnv` and the global cache attached to the +attached to the [`ParamEnv`](./param_env.html) and the global cache attached to the `tcx`. We use the local cache whenever the result might depend on the where clauses that are in scope. The determination of which cache to use is done by the method `pick_candidate_cache` in `select.rs`. At From 2f04df39d1f6010d015c02dfbb1801d3aeb9113c Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 12 Feb 2018 15:22:38 -0600 Subject: [PATCH 103/648] Clean up and reorganize traits chapter --- src/SUMMARY.md | 3 + src/trait-caching.md | 54 +++++ src/trait-hrtb.md | 116 +++++++++++ src/trait-resolution.md | 402 ++++++++++-------------------------- src/trait-specialization.md | 39 ++++ 5 files changed, 322 insertions(+), 292 deletions(-) create mode 100644 src/trait-caching.md create mode 100644 src/trait-hrtb.md create mode 100644 src/trait-specialization.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index ea2a6cd98..af7dfbd66 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -18,6 +18,9 @@ - [The `ty` module: representing types](./ty.md) - [Type inference](./type-inference.md) - [Trait resolution](./trait-resolution.md) + - [Higher-ranked trait bounds](./trait-hrtb.md) + - [Caching subtleties](./trait-caching.md) + - [Speciailization](./trait-specialization.md) - [Type checking](./type-checking.md) - [The MIR (Mid-level IR)](./mir.md) - [MIR construction](./mir-construction.md) diff --git a/src/trait-caching.md b/src/trait-caching.md new file mode 100644 index 000000000..6a0e55b27 --- /dev/null +++ b/src/trait-caching.md @@ -0,0 +1,54 @@ +# Caching and subtle considerations therewith + +In general we attempt to cache the results of trait selection. This +is a somewhat complex process. Part of the reason for this is that we +want to be able to cache results even when all the types in the trait +reference are not fully known. In that case, it may happen that the +trait selection process is also influencing type variables, so we have +to be able to not only cache the *result* of the selection process, +but *replay* its effects on the type variables. + +## An example + +The high-level idea of how the cache works is that we first replace +all unbound inference variables with skolemized versions. Therefore, +if we had a trait reference `usize : Foo<$1>`, where `$n` is an unbound +inference variable, we might replace it with `usize : Foo<%0>`, where +`%n` is a skolemized type. We would then look this up in the cache. +If we found a hit, the hit would tell us the immediate next step to +take in the selection process: i.e. apply impl #22, or apply where +clause `X : Foo`. Let's say in this case there is no hit. +Therefore, we search through impls and where clauses and so forth, and +we come to the conclusion that the only possible impl is this one, +with def-id 22: + +```rust +impl Foo for usize { ... } // Impl #22 +``` + +We would then record in the cache `usize : Foo<%0> ==> +ImplCandidate(22)`. Next we would confirm `ImplCandidate(22)`, which +would (as a side-effect) unify `$1` with `isize`. + +Now, at some later time, we might come along and see a `usize : +Foo<$3>`. When skolemized, this would yield `usize : Foo<%0>`, just as +before, and hence the cache lookup would succeed, yielding +`ImplCandidate(22)`. We would confirm `ImplCandidate(22)` which would +(as a side-effect) unify `$3` with `isize`. + +## Where clauses and the local vs global cache + +One subtle interaction is that the results of trait lookup will vary +depending on what where clauses are in scope. Therefore, we actually +have *two* caches, a local and a global cache. The local cache is +attached to the `ParamEnv` and the global cache attached to the +`tcx`. We use the local cache whenever the result might depend on the +where clauses that are in scope. The determination of which cache to +use is done by the method `pick_candidate_cache` in `select.rs`. At +the moment, we use a very simple, conservative rule: if there are any +where-clauses in scope, then we use the local cache. We used to try +and draw finer-grained distinctions, but that led to a serious of +annoying and weird bugs like #22019 and #18290. This simple rule seems +to be pretty clearly safe and also still retains a very high hit rate +(~95% when compiling rustc). + diff --git a/src/trait-hrtb.md b/src/trait-hrtb.md new file mode 100644 index 000000000..f0c47d942 --- /dev/null +++ b/src/trait-hrtb.md @@ -0,0 +1,116 @@ +# Higher-ranked trait bounds + +One of the more subtle concepts at work are *higher-ranked trait +bounds*. An example of such a bound is `for<'a> MyTrait<&'a isize>`. +Let's walk through how selection on higher-ranked trait references +works. + +## Basic matching and skolemization leaks + +Let's walk through the test `compile-fail/hrtb-just-for-static.rs` to see +how it works. The test starts with the trait `Foo`: + +```rust +trait Foo { + fn foo(&self, x: X) { } +} +``` + +Let's say we have a function `want_hrtb` that wants a type which +implements `Foo<&'a isize>` for any `'a`: + +```rust +fn want_hrtb() where T : for<'a> Foo<&'a isize> { ... } +``` + +Now we have a struct `AnyInt` that implements `Foo<&'a isize>` for any +`'a`: + +```rust +struct AnyInt; +impl<'a> Foo<&'a isize> for AnyInt { } +``` + +And the question is, does `AnyInt : for<'a> Foo<&'a isize>`? We want the +answer to be yes. The algorithm for figuring it out is closely related +to the subtyping for higher-ranked types (which is described in +`middle::infer::higher_ranked::doc`, but also in a [paper by SPJ] that +I recommend you read). + +1. Skolemize the obligation. +2. Match the impl against the skolemized obligation. +3. Check for skolemization leaks. + +[paper by SPJ]: http://research.microsoft.com/en-us/um/people/simonpj/papers/higher-rank/ + +So let's work through our example. The first thing we would do is to +skolemize the obligation, yielding `AnyInt : Foo<&'0 isize>` (here `'0` +represents skolemized region #0). Note that now have no quantifiers; +in terms of the compiler type, this changes from a `ty::PolyTraitRef` +to a `TraitRef`. We would then create the `TraitRef` from the impl, +using fresh variables for it's bound regions (and thus getting +`Foo<&'$a isize>`, where `'$a` is the inference variable for `'a`). Next +we relate the two trait refs, yielding a graph with the constraint +that `'0 == '$a`. Finally, we check for skolemization "leaks" – a +leak is basically any attempt to relate a skolemized region to another +skolemized region, or to any region that pre-existed the impl match. +The leak check is done by searching from the skolemized region to find +the set of regions that it is related to in any way. This is called +the "taint" set. To pass the check, that set must consist *solely* of +itself and region variables from the impl. If the taint set includes +any other region, then the match is a failure. In this case, the taint +set for `'0` is `{'0, '$a}`, and hence the check will succeed. + +Let's consider a failure case. Imagine we also have a struct + +```rust +struct StaticInt; +impl Foo<&'static isize> for StaticInt; +``` + +We want the obligation `StaticInt : for<'a> Foo<&'a isize>` to be +considered unsatisfied. The check begins just as before. `'a` is +skolemized to `'0` and the impl trait reference is instantiated to +`Foo<&'static isize>`. When we relate those two, we get a constraint +like `'static == '0`. This means that the taint set for `'0` is `{'0, +'static}`, which fails the leak check. + +## Higher-ranked trait obligations + +Once the basic matching is done, we get to another interesting topic: +how to deal with impl obligations. I'll work through a simple example +here. Imagine we have the traits `Foo` and `Bar` and an associated impl: + +```rust +trait Foo { + fn foo(&self, x: X) { } +} + +trait Bar { + fn bar(&self, x: X) { } +} + +impl Foo for F + where F : Bar +{ +} +``` + +Now let's say we have a obligation `for<'a> Foo<&'a isize>` and we match +this impl. What obligation is generated as a result? We want to get +`for<'a> Bar<&'a isize>`, but how does that happen? + +After the matching, we are in a position where we have a skolemized +substitution like `X => &'0 isize`. If we apply this substitution to the +impl obligations, we get `F : Bar<&'0 isize>`. Obviously this is not +directly usable because the skolemized region `'0` cannot leak out of +our computation. + +What we do is to create an inverse mapping from the taint set of `'0` +back to the original bound region (`'a`, here) that `'0` resulted +from. (This is done in `higher_ranked::plug_leaks`). We know that the +leak check passed, so this taint set consists solely of the skolemized +region itself plus various intermediate region variables. We then walk +the trait-reference and convert every region in that taint set back to +a late-bound region, so in this case we'd wind up with `for<'a> F : +Bar<&'a isize>`. diff --git a/src/trait-resolution.md b/src/trait-resolution.md index 10a5539e4..1cad488ac 100644 --- a/src/trait-resolution.md +++ b/src/trait-resolution.md @@ -1,10 +1,7 @@ # Trait resolution -This document describes the general process and points out some non-obvious -things. - -**WARNING:** This material was moved verbatim from a rustc README, so -it may not "fit" the style of the guide until it is adapted. +This chapter describes the general process of _trait resolution_ and points out +some non-obvious things. ## Major concepts @@ -21,12 +18,12 @@ and then a call to that function: let v: Vec = clone_slice(&[1, 2, 3]) ``` -it is the job of trait resolution to figure out (in which case) -whether there exists an impl of `isize : Clone` +it is the job of trait resolution to figure out whether there exists an impl of +(in this case) `isize : Clone`. Note that in some cases, like generic functions, we may not be able to find a specific impl, but we can figure out that the caller must -provide an impl. To see what I mean, consider the body of `clone_slice`: +provide an impl. For example, consider the body of `clone_slice`: ```rust fn clone_slice(x: &[T]) -> Vec { @@ -43,26 +40,33 @@ is, we can't find the specific impl; but based on the bound `T:Clone`, we can say that there exists an impl which the caller must provide. We use the term *obligation* to refer to a trait reference in need of -an impl. +an impl. Basically, the trait resolution system resolves an obligation +by proving that an appropriate impl does exist. + +During type checking, we do not store the results of trait selection. +We simply wish to verify that trait selection will succeed. Then +later, at trans time, when we have all concrete types available, we +can repeat the trait selection to choose an actual implementation, which +will then be generated in the output binary. ## Overview Trait resolution consists of three major parts: -- SELECTION: Deciding how to resolve a specific obligation. For +- **Selection** is deciding how to resolve a specific obligation. For example, selection might decide that a specific obligation can be - resolved by employing an impl which matches the self type, or by - using a parameter bound. In the case of an impl, Selecting one + resolved by employing an impl which matches the `Self` type, or by + using a parameter bound (e.g. `T: Trait`). In the case of an impl, selecting one obligation can create *nested obligations* because of where clauses on the impl itself. It may also require evaluating those nested obligations to resolve ambiguities. -- FULFILLMENT: The fulfillment code is what tracks that obligations - are completely fulfilled. Basically it is a worklist of obligations +- **Fulfillment** is keeping track of which obligations + are completely fulfilled. Basically, it is a worklist of obligations to be selected: once selection is successful, the obligation is removed from the worklist and any nested obligations are enqueued. -- COHERENCE: The coherence checks are intended to ensure that there +- **Coherence** checks are intended to ensure that there are never overlapping impls, where two impls could be used with equal precedence. @@ -83,12 +87,21 @@ and returns a `SelectionResult`. There are three possible outcomes: contains unbound type variables. - `Err(err)` – the obligation definitely cannot be resolved due to a - type error, or because there are no impls that could possibly apply, - etc. + type error or because there are no impls that could possibly apply. The basic algorithm for selection is broken into two big phases: candidate assembly and confirmation. +Note that because of how lifetime inference works, it is not possible to +give back immediate feedback as to whether a unification or subtype +relationship between lifetimes holds or not. Therefore, lifetime +matching is *not* considered during selection. This is reflected in +the fact that subregion assignment is infallible. This may yield +lifetime constraints that will later be found to be in error (in +contrast, the non-lifetime-constraints have already been checked +during selection and can never cause an error, though naturally they +may lead to other errors downstream). + ### Candidate assembly Searches for impls/where-clauses/etc that might @@ -98,6 +111,15 @@ candidate that is definitively applicable. In some cases, we may not know whether an impl/where-clause applies or not – this occurs when the obligation contains unbound inference variables. +The subroutines that decide whether a particular impl/where-clause/etc +applies to a particular obligation are collectively refered to as the +process of _matching_. At the moment, this amounts to +unifying the `Self` types, but in the future we may also recursively +consider some of the nested obligations, in the case of an impl. + +**TODO**: what does "unifying the `Self` types" mean? The `Self` of the +obligation with that of an impl? + The basic idea for candidate assembly is to do a first pass in which we identify all possible candidates. During this pass, all that we do is try and unify the type parameters. (In particular, we ignore any @@ -141,16 +163,14 @@ let y = x.convert(); The call to convert will generate a trait reference `Convert<$Y> for isize`, where `$Y` is the type variable representing the type of -`y`. When we match this against the two impls we can see, we will find -that only one remains: `Convert for isize`. Therefore, we can +`y`. Of the two impls we can see, the only one that matches is +`Convert for isize`. Therefore, we can select this impl, which will cause the type of `$Y` to be unified to `usize`. (Note that while assembling candidates, we do the initial unifications in a transaction, so that they don't affect one another.) -There are tests to this effect in src/test/run-pass: - - traits-multidispatch-infer-convert-source-and-target.rs - traits-multidispatch-infer-convert-target.rs +**TODO**: The example says we can "select" the impl, but this section is talking specifically about candidate assembly. Does this mean we can sometimes skip confirmation? Or is this poor wording? +**TODO**: Is the unification of `$Y` part of trait resolution or type inference? Or is this not the same type of "inference variable" as in type inference? #### Winnowing: Resolving ambiguities @@ -167,94 +187,103 @@ impl Get for T { } impl Get for Box { - fn get(&self) -> Box { box get_it(&**self) } + fn get(&self) -> Box { Box::new(get_it(&**self)) } } ``` -What happens when we invoke `get_it(&box 1_u16)`, for example? In this +What happens when we invoke `get_it(&Box::new(1_u16))`, for example? In this case, the `Self` type is `Box` – that unifies with both impls, because the first applies to all types, and the second to all -boxes. In the olden days we'd have called this ambiguous. But what we -do now is do a second *winnowing* pass that considers where clauses -and attempts to remove candidates – in this case, the first impl only +boxes. In order for this to be unambiguous, the compiler does a *winnowing* +pass that considers `where` clauses +and attempts to remove candidates. In this case, the first impl only applies if `Box : Copy`, which doesn't hold. After winnowing, -then, we are left with just one candidate, so we can proceed. There is -a test of this in `src/test/run-pass/traits-conditional-dispatch.rs`. - -#### Matching - -The subroutines that decide whether a particular impl/where-clause/etc -applies to a particular obligation. At the moment, this amounts to -unifying the self types, but in the future we may also recursively -consider some of the nested obligations, in the case of an impl. +then, we are left with just one candidate, so we can proceed. -#### Lifetimes and selection - -Because of how that lifetime inference works, it is not possible to -give back immediate feedback as to whether a unification or subtype -relationship between lifetimes holds or not. Therefore, lifetime -matching is *not* considered during selection. This is reflected in -the fact that subregion assignment is infallible. This may yield -lifetime constraints that will later be found to be in error (in -contrast, the non-lifetime-constraints have already been checked -during selection and can never cause an error, though naturally they -may lead to other errors downstream). - -#### Where clauses +#### `where` clauses Besides an impl, the other major way to resolve an obligation is via a -where clause. The selection process is always given a *parameter -environment* which contains a list of where clauses, which are -basically obligations that can assume are satisfiable. We will iterate +where clause. The selection process is always given a [parameter +environment] which contains a list of where clauses, which are +basically obligations that we can assume are satisfiable. We will iterate over that list and check whether our current obligation can be found -in that list, and if so it is considered satisfied. More precisely, we +in that list. If so, it is considered satisfied. More precisely, we want to check whether there is a where-clause obligation that is for -the same trait (or some subtrait) and for which the self types match, -using the definition of *matching* given above. +the same trait (or some subtrait) and which can match against the obligation. + +[parameter environment]: ./param_env.html Consider this simple example: ```rust -trait A1 { /*...*/ } +trait A1 { + fn do_a1(&self); +} trait A2 : A1 { /*...*/ } -trait B { /*...*/ } +trait B { + fn do_b(&self); +} -fn foo { /*...*/ } +fn foo(x: X) { + x.do_a1(); // (*) + x.do_b(); // (#) +} ``` -Clearly we can use methods offered by `A1`, `A2`, or `B` within the -body of `foo`. In each case, that will incur an obligation like `X : -A1` or `X : A2`. The parameter environment will contain two -where-clauses, `X : A2` and `X : B`. For each obligation, then, we -search this list of where-clauses. To resolve an obligation `X:A1`, -we would note that `X:A2` implies that `X:A1`. +In the body of `foo`, clearly we can use methods of `A1`, `A2`, or `B` +on variable `x`. The line marked `(*)` will incur an obligation `X: A1`, +which the line marked `(#)` will incur an obligation `X: B`. Meanwhile, +the parameter environment will contain two where-clauses: `X : A2` and `X : B`. +For each obligation, then, we search this list of where-clauses. The +obligation `X: B` trivially matches against the where-clause `X: B`. +To resolve an obligation `X:A1`, we would note that `X:A2` implies that `X:A1`. ### Confirmation -Confirmation unifies the output type parameters of the trait with the -values found in the obligation, possibly yielding a type error. If we -return to our example of the `Convert` trait from the previous -section, confirmation is where an error would be reported, because the -impl specified that `T` would be `usize`, but the obligation reported -`char`. Hence the result of selection would be an error. +_Confirmation_ unifies the output type parameters of the trait with the +values found in the obligation, possibly yielding a type error. + +Suppose we have the following variation of the `Convert` example in the +previous section: + +```rust +trait Convert { + fn convert(&self) -> Target; +} + +impl Convert for isize { /*...*/ } // isize -> usize +impl Convert for usize { /*...*/ } // usize -> isize + +let x: isize = ...; +let y: char = x.convert(); // NOTE: `y: char` now! +``` + +Confirmation is where an error would be reported because the impl specified +that `Target` would be `usize`, but the obligation reported `char`. Hence the +result of selection would be an error. + +Note that the candidate impl is chosen base on the `Self` type, but +confirmation is done based on (in this case) the `Target` type parameter. ### Selection during translation -During type checking, we do not store the results of trait selection. -We simply wish to verify that trait selection will succeed. Then -later, at trans time, when we have all concrete types available, we -can repeat the trait selection. In this case, we do not consider any -where-clauses to be in scope. We know that therefore each resolution -will resolve to a particular impl. +As mentioned above, during type checking, we do not store the results of trait +selection. At trans time, repeat the trait selection to choose a particular +impl for each method call. In this second selection, we do not consider any +where-clauses to be in scope because we know that each resolution will resolve +to a particular impl. One interesting twist has to do with nested obligations. In general, in trans, we only need to do a "shallow" selection for an obligation. That is, we wish to identify which impl applies, but we do not (yet) need to decide how to select -any nested obligations. Nonetheless, we *do* currently do a complete resolution, -and that is because it can sometimes inform the results of type inference. That is, -we do not have the full substitutions in terms of the type variables of the impl available -to us, so we must run trait selection to figure everything out. +any nested obligations. Nonetheless, we *do* currently do a complete +resolution, and that is because it can sometimes inform the results of type +inference. That is, we do not have the full substitutions for the type +variables of the impl available to us, so we must run trait selection to figure +everything out. + +**TODO**: is this still talking about trans? Here is an example: @@ -272,214 +301,3 @@ nested obligation `isize : Bar` to find out that `U=usize`. It would be good to only do *just as much* nested resolution as necessary. Currently, though, we just do a full resolution. - -# Higher-ranked trait bounds - -One of the more subtle concepts at work are *higher-ranked trait -bounds*. An example of such a bound is `for<'a> MyTrait<&'a isize>`. -Let's walk through how selection on higher-ranked trait references -works. - -## Basic matching and skolemization leaks - -Let's walk through the test `compile-fail/hrtb-just-for-static.rs` to see -how it works. The test starts with the trait `Foo`: - -```rust -trait Foo { - fn foo(&self, x: X) { } -} -``` - -Let's say we have a function `want_hrtb` that wants a type which -implements `Foo<&'a isize>` for any `'a`: - -```rust -fn want_hrtb() where T : for<'a> Foo<&'a isize> { ... } -``` - -Now we have a struct `AnyInt` that implements `Foo<&'a isize>` for any -`'a`: - -```rust -struct AnyInt; -impl<'a> Foo<&'a isize> for AnyInt { } -``` - -And the question is, does `AnyInt : for<'a> Foo<&'a isize>`? We want the -answer to be yes. The algorithm for figuring it out is closely related -to the subtyping for higher-ranked types (which is described in -`middle::infer::higher_ranked::doc`, but also in a [paper by SPJ] that -I recommend you read). - -1. Skolemize the obligation. -2. Match the impl against the skolemized obligation. -3. Check for skolemization leaks. - -[paper by SPJ]: http://research.microsoft.com/en-us/um/people/simonpj/papers/higher-rank/ - -So let's work through our example. The first thing we would do is to -skolemize the obligation, yielding `AnyInt : Foo<&'0 isize>` (here `'0` -represents skolemized region #0). Note that now have no quantifiers; -in terms of the compiler type, this changes from a `ty::PolyTraitRef` -to a `TraitRef`. We would then create the `TraitRef` from the impl, -using fresh variables for it's bound regions (and thus getting -`Foo<&'$a isize>`, where `'$a` is the inference variable for `'a`). Next -we relate the two trait refs, yielding a graph with the constraint -that `'0 == '$a`. Finally, we check for skolemization "leaks" – a -leak is basically any attempt to relate a skolemized region to another -skolemized region, or to any region that pre-existed the impl match. -The leak check is done by searching from the skolemized region to find -the set of regions that it is related to in any way. This is called -the "taint" set. To pass the check, that set must consist *solely* of -itself and region variables from the impl. If the taint set includes -any other region, then the match is a failure. In this case, the taint -set for `'0` is `{'0, '$a}`, and hence the check will succeed. - -Let's consider a failure case. Imagine we also have a struct - -```rust -struct StaticInt; -impl Foo<&'static isize> for StaticInt; -``` - -We want the obligation `StaticInt : for<'a> Foo<&'a isize>` to be -considered unsatisfied. The check begins just as before. `'a` is -skolemized to `'0` and the impl trait reference is instantiated to -`Foo<&'static isize>`. When we relate those two, we get a constraint -like `'static == '0`. This means that the taint set for `'0` is `{'0, -'static}`, which fails the leak check. - -## Higher-ranked trait obligations - -Once the basic matching is done, we get to another interesting topic: -how to deal with impl obligations. I'll work through a simple example -here. Imagine we have the traits `Foo` and `Bar` and an associated impl: - -```rust -trait Foo { - fn foo(&self, x: X) { } -} - -trait Bar { - fn bar(&self, x: X) { } -} - -impl Foo for F - where F : Bar -{ -} -``` - -Now let's say we have a obligation `for<'a> Foo<&'a isize>` and we match -this impl. What obligation is generated as a result? We want to get -`for<'a> Bar<&'a isize>`, but how does that happen? - -After the matching, we are in a position where we have a skolemized -substitution like `X => &'0 isize`. If we apply this substitution to the -impl obligations, we get `F : Bar<&'0 isize>`. Obviously this is not -directly usable because the skolemized region `'0` cannot leak out of -our computation. - -What we do is to create an inverse mapping from the taint set of `'0` -back to the original bound region (`'a`, here) that `'0` resulted -from. (This is done in `higher_ranked::plug_leaks`). We know that the -leak check passed, so this taint set consists solely of the skolemized -region itself plus various intermediate region variables. We then walk -the trait-reference and convert every region in that taint set back to -a late-bound region, so in this case we'd wind up with `for<'a> F : -Bar<&'a isize>`. - -# Caching and subtle considerations therewith - -In general we attempt to cache the results of trait selection. This -is a somewhat complex process. Part of the reason for this is that we -want to be able to cache results even when all the types in the trait -reference are not fully known. In that case, it may happen that the -trait selection process is also influencing type variables, so we have -to be able to not only cache the *result* of the selection process, -but *replay* its effects on the type variables. - -## An example - -The high-level idea of how the cache works is that we first replace -all unbound inference variables with skolemized versions. Therefore, -if we had a trait reference `usize : Foo<$1>`, where `$n` is an unbound -inference variable, we might replace it with `usize : Foo<%0>`, where -`%n` is a skolemized type. We would then look this up in the cache. -If we found a hit, the hit would tell us the immediate next step to -take in the selection process: i.e. apply impl #22, or apply where -clause `X : Foo`. Let's say in this case there is no hit. -Therefore, we search through impls and where clauses and so forth, and -we come to the conclusion that the only possible impl is this one, -with def-id 22: - -```rust -impl Foo for usize { ... } // Impl #22 -``` - -We would then record in the cache `usize : Foo<%0> ==> -ImplCandidate(22)`. Next we would confirm `ImplCandidate(22)`, which -would (as a side-effect) unify `$1` with `isize`. - -Now, at some later time, we might come along and see a `usize : -Foo<$3>`. When skolemized, this would yield `usize : Foo<%0>`, just as -before, and hence the cache lookup would succeed, yielding -`ImplCandidate(22)`. We would confirm `ImplCandidate(22)` which would -(as a side-effect) unify `$3` with `isize`. - -## Where clauses and the local vs global cache - -One subtle interaction is that the results of trait lookup will vary -depending on what where clauses are in scope. Therefore, we actually -have *two* caches, a local and a global cache. The local cache is -attached to the [`ParamEnv`](./param_env.html) and the global cache attached to the -`tcx`. We use the local cache whenever the result might depend on the -where clauses that are in scope. The determination of which cache to -use is done by the method `pick_candidate_cache` in `select.rs`. At -the moment, we use a very simple, conservative rule: if there are any -where-clauses in scope, then we use the local cache. We used to try -and draw finer-grained distinctions, but that led to a serious of -annoying and weird bugs like #22019 and #18290. This simple rule seems -to be pretty clearly safe and also still retains a very high hit rate -(~95% when compiling rustc). - -# Specialization - -Defined in the `specialize` module. - -The basic strategy is to build up a *specialization graph* during -coherence checking. Insertion into the graph locates the right place -to put an impl in the specialization hierarchy; if there is no right -place (due to partial overlap but no containment), you get an overlap -error. Specialization is consulted when selecting an impl (of course), -and the graph is consulted when propagating defaults down the -specialization hierarchy. - -You might expect that the specialization graph would be used during -selection – i.e. when actually performing specialization. This is -not done for two reasons: - -- It's merely an optimization: given a set of candidates that apply, - we can determine the most specialized one by comparing them directly - for specialization, rather than consulting the graph. Given that we - also cache the results of selection, the benefit of this - optimization is questionable. - -- To build the specialization graph in the first place, we need to use - selection (because we need to determine whether one impl specializes - another). Dealing with this reentrancy would require some additional - mode switch for selection. Given that there seems to be no strong - reason to use the graph anyway, we stick with a simpler approach in - selection, and use the graph only for propagating default - implementations. - -Trait impl selection can succeed even when multiple impls can apply, -as long as they are part of the same specialization family. In that -case, it returns a *single* impl on success – this is the most -specialized impl *known* to apply. However, if there are any inference -variables in play, the returned impl may not be the actual impl we -will use at trans time. Thus, we take special care to avoid projecting -associated types unless either (1) the associated type does not use -`default` and thus cannot be overridden or (2) all input types are -known concretely. diff --git a/src/trait-specialization.md b/src/trait-specialization.md new file mode 100644 index 000000000..6b8bc64dd --- /dev/null +++ b/src/trait-specialization.md @@ -0,0 +1,39 @@ +# Specialization + +Defined in the `specialize` module. + +The basic strategy is to build up a *specialization graph* during +coherence checking. Insertion into the graph locates the right place +to put an impl in the specialization hierarchy; if there is no right +place (due to partial overlap but no containment), you get an overlap +error. Specialization is consulted when selecting an impl (of course), +and the graph is consulted when propagating defaults down the +specialization hierarchy. + +You might expect that the specialization graph would be used during +selection – i.e. when actually performing specialization. This is +not done for two reasons: + +- It's merely an optimization: given a set of candidates that apply, + we can determine the most specialized one by comparing them directly + for specialization, rather than consulting the graph. Given that we + also cache the results of selection, the benefit of this + optimization is questionable. + +- To build the specialization graph in the first place, we need to use + selection (because we need to determine whether one impl specializes + another). Dealing with this reentrancy would require some additional + mode switch for selection. Given that there seems to be no strong + reason to use the graph anyway, we stick with a simpler approach in + selection, and use the graph only for propagating default + implementations. + +Trait impl selection can succeed even when multiple impls can apply, +as long as they are part of the same specialization family. In that +case, it returns a *single* impl on success – this is the most +specialized impl *known* to apply. However, if there are any inference +variables in play, the returned impl may not be the actual impl we +will use at trans time. Thus, we take special care to avoid projecting +associated types unless either (1) the associated type does not use +`default` and thus cannot be overridden or (2) all input types are +known concretely. From 93958df2e771476fbf999460a9273a23d54dcd2d Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 12 Feb 2018 17:22:23 -0600 Subject: [PATCH 104/648] Clarifications and edits to hrtb chapter --- src/trait-caching.md | 1 - src/trait-hrtb.md | 41 +++++++++++++++++++++++++---------------- 2 files changed, 25 insertions(+), 17 deletions(-) diff --git a/src/trait-caching.md b/src/trait-caching.md index 6a0e55b27..31f398ff5 100644 --- a/src/trait-caching.md +++ b/src/trait-caching.md @@ -51,4 +51,3 @@ and draw finer-grained distinctions, but that led to a serious of annoying and weird bugs like #22019 and #18290. This simple rule seems to be pretty clearly safe and also still retains a very high hit rate (~95% when compiling rustc). - diff --git a/src/trait-hrtb.md b/src/trait-hrtb.md index f0c47d942..8c3a6f4e9 100644 --- a/src/trait-hrtb.md +++ b/src/trait-hrtb.md @@ -1,14 +1,13 @@ # Higher-ranked trait bounds -One of the more subtle concepts at work are *higher-ranked trait +One of the more subtle concepts in trait resolution is *higher-ranked trait bounds*. An example of such a bound is `for<'a> MyTrait<&'a isize>`. Let's walk through how selection on higher-ranked trait references works. ## Basic matching and skolemization leaks -Let's walk through the test `compile-fail/hrtb-just-for-static.rs` to see -how it works. The test starts with the trait `Foo`: +Suppose we have a trait `Foo`: ```rust trait Foo { @@ -33,25 +32,34 @@ impl<'a> Foo<&'a isize> for AnyInt { } And the question is, does `AnyInt : for<'a> Foo<&'a isize>`? We want the answer to be yes. The algorithm for figuring it out is closely related -to the subtyping for higher-ranked types (which is described in -`middle::infer::higher_ranked::doc`, but also in a [paper by SPJ] that -I recommend you read). +to the subtyping for higher-ranked types (which is described in [here][hrsubtype] +and also in a [paper by SPJ]. If you wish to understand higher-ranked +subtyping, we recommend you read the paper). There are a few parts: -1. Skolemize the obligation. +**TODO**: We should define _skolemize_. + +1. _Skolemize_ the obligation. 2. Match the impl against the skolemized obligation. -3. Check for skolemization leaks. +3. Check for _skolemization leaks_. +[hrsubtype]: https://github.com/rust-lang/rust/tree/master/src/librustc/infer/higher_ranked/README.md [paper by SPJ]: http://research.microsoft.com/en-us/um/people/simonpj/papers/higher-rank/ -So let's work through our example. The first thing we would do is to +So let's work through our example. + +1. The first thing we would do is to skolemize the obligation, yielding `AnyInt : Foo<&'0 isize>` (here `'0` -represents skolemized region #0). Note that now have no quantifiers; +represents skolemized region #0). Note that we now have no quantifiers; in terms of the compiler type, this changes from a `ty::PolyTraitRef` to a `TraitRef`. We would then create the `TraitRef` from the impl, using fresh variables for it's bound regions (and thus getting -`Foo<&'$a isize>`, where `'$a` is the inference variable for `'a`). Next +`Foo<&'$a isize>`, where `'$a` is the inference variable for `'a`). + +2. Next we relate the two trait refs, yielding a graph with the constraint -that `'0 == '$a`. Finally, we check for skolemization "leaks" – a +that `'0 == '$a`. + +3. Finally, we check for skolemization "leaks" – a leak is basically any attempt to relate a skolemized region to another skolemized region, or to any region that pre-existed the impl match. The leak check is done by searching from the skolemized region to find @@ -75,6 +83,8 @@ skolemized to `'0` and the impl trait reference is instantiated to like `'static == '0`. This means that the taint set for `'0` is `{'0, 'static}`, which fails the leak check. +**TODO**: This is because `'static` is not a region variable but is in the taint set, right? + ## Higher-ranked trait obligations Once the basic matching is done, we get to another interesting topic: @@ -96,9 +106,9 @@ impl Foo for F } ``` -Now let's say we have a obligation `for<'a> Foo<&'a isize>` and we match +Now let's say we have a obligation `Baz: for<'a> Foo<&'a isize>` and we match this impl. What obligation is generated as a result? We want to get -`for<'a> Bar<&'a isize>`, but how does that happen? +`Baz: for<'a> Bar<&'a isize>`, but how does that happen? After the matching, we are in a position where we have a skolemized substitution like `X => &'0 isize`. If we apply this substitution to the @@ -112,5 +122,4 @@ from. (This is done in `higher_ranked::plug_leaks`). We know that the leak check passed, so this taint set consists solely of the skolemized region itself plus various intermediate region variables. We then walk the trait-reference and convert every region in that taint set back to -a late-bound region, so in this case we'd wind up with `for<'a> F : -Bar<&'a isize>`. +a late-bound region, so in this case we'd wind up with `Baz: for<'a> Bar<&'a isize>`. From 0236316fb29925fd51d8666a487e74333b9f03c2 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 12 Feb 2018 17:49:56 -0600 Subject: [PATCH 105/648] Edits and cleanup in trait-caching subchapter --- src/trait-caching.md | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/src/trait-caching.md b/src/trait-caching.md index 31f398ff5..ee92814cd 100644 --- a/src/trait-caching.md +++ b/src/trait-caching.md @@ -1,6 +1,6 @@ # Caching and subtle considerations therewith -In general we attempt to cache the results of trait selection. This +In general, we attempt to cache the results of trait selection. This is a somewhat complex process. Part of the reason for this is that we want to be able to cache results even when all the types in the trait reference are not fully known. In that case, it may happen that the @@ -12,37 +12,43 @@ but *replay* its effects on the type variables. The high-level idea of how the cache works is that we first replace all unbound inference variables with skolemized versions. Therefore, -if we had a trait reference `usize : Foo<$1>`, where `$n` is an unbound -inference variable, we might replace it with `usize : Foo<%0>`, where -`%n` is a skolemized type. We would then look this up in the cache. +if we had a trait reference `usize : Foo<$t>`, where `$t` is an unbound +inference variable, we might replace it with `usize : Foo<$0>`, where +`$0` is a skolemized type. We would then look this up in the cache. + If we found a hit, the hit would tell us the immediate next step to -take in the selection process: i.e. apply impl #22, or apply where -clause `X : Foo`. Let's say in this case there is no hit. -Therefore, we search through impls and where clauses and so forth, and -we come to the conclusion that the only possible impl is this one, -with def-id 22: +take in the selection process (e.g. apply impl #22, or apply where +clause `X : Foo`). + +On the other hand, if there is no hit, we need to go through the [selection +process] from scratch. Suppose, we come to the conclusion that the only +possible impl is this one, with def-id 22: + +[selection process]: ./trait-resolution.html#selection ```rust impl Foo for usize { ... } // Impl #22 ``` -We would then record in the cache `usize : Foo<%0> ==> -ImplCandidate(22)`. Next we would confirm `ImplCandidate(22)`, which -would (as a side-effect) unify `$1` with `isize`. +We would then record in the cache `usize : Foo<$0> => ImplCandidate(22)`. Next +we would [confirm] `ImplCandidate(22)`, which would (as a side-effect) unify +`$t` with `isize`. + +[confirm]: ./trait-resolution.html#confirmation Now, at some later time, we might come along and see a `usize : -Foo<$3>`. When skolemized, this would yield `usize : Foo<%0>`, just as +Foo<$u>`. When skolemized, this would yield `usize : Foo<$0>`, just as before, and hence the cache lookup would succeed, yielding `ImplCandidate(22)`. We would confirm `ImplCandidate(22)` which would -(as a side-effect) unify `$3` with `isize`. +(as a side-effect) unify `$u` with `isize`. ## Where clauses and the local vs global cache One subtle interaction is that the results of trait lookup will vary depending on what where clauses are in scope. Therefore, we actually have *two* caches, a local and a global cache. The local cache is -attached to the `ParamEnv` and the global cache attached to the -`tcx`. We use the local cache whenever the result might depend on the +attached to the [`ParamEnv`], and the global cache attached to the +[`tcx`]. We use the local cache whenever the result might depend on the where clauses that are in scope. The determination of which cache to use is done by the method `pick_candidate_cache` in `select.rs`. At the moment, we use a very simple, conservative rule: if there are any @@ -51,3 +57,9 @@ and draw finer-grained distinctions, but that led to a serious of annoying and weird bugs like #22019 and #18290. This simple rule seems to be pretty clearly safe and also still retains a very high hit rate (~95% when compiling rustc). + +**TODO**: it looks like `pick_candidate_cache` no longer exists. In +general, is this section still accurate at all? + +[`ParamEnv`]: ./param_env.html +[`tcx`]: ./ty.html From a0ddc8aa8aa250580acfd9218022f3f95ec6284f Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 12 Feb 2018 17:55:06 -0600 Subject: [PATCH 106/648] Minor updates to specialization subchapter --- src/trait-specialization.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/trait-specialization.md b/src/trait-specialization.md index 6b8bc64dd..671e5e016 100644 --- a/src/trait-specialization.md +++ b/src/trait-specialization.md @@ -1,9 +1,12 @@ # Specialization +**TODO**: where does Chalk fit in? Should we mention/discuss it here? + Defined in the `specialize` module. The basic strategy is to build up a *specialization graph* during -coherence checking. Insertion into the graph locates the right place +coherence checking (recall that coherence checking looks for overlapping +impls). Insertion into the graph locates the right place to put an impl in the specialization hierarchy; if there is no right place (due to partial overlap but no containment), you get an overlap error. Specialization is consulted when selecting an impl (of course), From bfcdce6604b3ddc3f97146cbb8fbbce1588b2de1 Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Fri, 23 Feb 2018 10:18:16 -0800 Subject: [PATCH 107/648] Add some more information about the various tests that get run. CC #10 --- src/tests/intro.md | 74 ++++++++++++++++++++++++++++++++++++++------ src/tests/running.md | 7 ----- 2 files changed, 65 insertions(+), 16 deletions(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index 615673206..2388e4308 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -1,17 +1,18 @@ -# Using the compiler testing framework +# The compiler testing framework -The compiler has an extensive testing framework, masterminded by the -compiletest tool (sources in the [`src/tools/compiletest`]). This -section gives a brief overview of how the testing framework is setup, -and then gets into some of the details on -[how to run tests](./tests/running.html#ui) as well as -[how to add new tests](./tests/adding.html). +The Rust project runs a wide variety of different tests, orchestrated by the +build system (`x.py test`). The main test harness for testing the compiler +itself is a tool called compiletest (sources in the +[`src/tools/compiletest`]). This section gives a brief overview of how the +testing framework is setup, and then gets into some of the details on [how to +run tests](./tests/running.html#ui) as well as [how to add new +tests](./tests/adding.html). [`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest -## Test suites +## Compiletest test suites -The tests are located in the tree in the [`src/test`] +The compiletest tests are located in the tree in the [`src/test`] directory. Immediately within you will see a series of subdirectories (e.g. `ui`, `run-make`, and so forth). Each of those directories is called a **test suite** -- they house a group of tests that are run in @@ -48,6 +49,61 @@ that give more details. - `*-fulldeps` -- same as above, but indicates that the test depends on things other than `libstd` (and hence those things must be built) +## Other Tests + +The Rust build system handles running tests for various other things, +including: + +- **Tidy** -- This is a custom tool used for validating source code style and + formatting conventions, such as rejecting long lines. There is more + information in the [section on coding conventions](./conventions.html#formatting). + + Example: `./x.py test src/tools/tidy` + +- **Unittests** -- The Rust standard library and many of the Rust packages + include typical Rust `#[test]` unittests. Under the hood, `x.py` will run + `cargo test` on each package to run all the tests. + + Example: `./x.py test src/libstd` + +- **Doctests** -- Example code embedded within Rust documentation is executed + via `rustdoc --test`. Examples: + + `./x.py test src/doc` -- Runs `rustdoc --test` for all documentation in + `src/doc`. + + `./x.py test --doc src/libstd` -- Runs `rustdoc --test` on the standard + library. + +- **Linkchecker** -- A small tool for verifying `href` links within + documentation. + + Example: `./x.py test src/tools/linkchecker` + +- **Distcheck** -- This verifies that the source distribution tarball created + by the build system will unpack, build, and run all tests. + + Example: `./x.py test distcheck` + +- **Tool tests** -- Packages that are included with Rust have all of their + tests run as well (typically by running `cargo test` within their + directory). This includes things such as cargo, clippy, rustfmt, rls, miri, + bootstrap (testing the Rust build system itself), etc. + +- **Cargotest** -- This is a small tool which runs `cargo test` on a few + significant projects (such as `servo`, `ripgrep`, `tokei`, etc.) just to + ensure there aren't any significant regressions. + + Example: `./x.py test src/tools/cargotest` + +## Testing infrastructure + +TODO - bors, platforms, etc. + +## Crater + +TODO + ## Further reading The following blog posts may also be of interest: diff --git a/src/tests/running.md b/src/tests/running.md index 602aa51d9..98d590671 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -13,13 +13,6 @@ a very long time, and anyway bors / travis will do it for you. (Often, I will run this command in the background after opening a PR that I think is done, but rarely otherwise. -nmatsakis) -## Tidy - -When you run the full suite of tests via `./x.py test`, the first -thing that executes is a "tidy suite" that checks for long lines and -other formatting conventions. There is more information in the -[section on coding conventions](./conventions.html#formatting). - ## Running a subset of the test suites When working on a specific PR, you will usually want to run a smaller From b35bd94e82a8fa093f76cf8f697e1671b9a9bf00 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 13 Feb 2018 11:17:45 -0600 Subject: [PATCH 108/648] Start a code index --- src/SUMMARY.md | 1 + src/code-index.md | 9 +++++++++ 2 files changed, 10 insertions(+) create mode 100644 src/code-index.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index ea2a6cd98..04904bce7 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -28,3 +28,4 @@ - [Parameter Environments](./param_env.md) - [Generating LLVM IR](./trans.md) - [Glossary](./glossary.md) +- [Code Index](./code-index.md) diff --git a/src/code-index.md b/src/code-index.md new file mode 100644 index 000000000..5b452ef0f --- /dev/null +++ b/src/code-index.md @@ -0,0 +1,9 @@ +# Code Index + +rustc has a lot of important data structures. This is an attempt to give some +guidance on where to learn more about some of the key data structures of the +compiler. + +Item | Kind | Short description | Chapter | Declaration +----------------|----------|-----------------------------|--------------------|------------------- +`TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules](ty.html) | [src/librustc/ty/context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs) From 9f0be41a06a0d8f9afbf3db5eff6213c2e2a15da Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 23 Feb 2018 13:17:29 -0600 Subject: [PATCH 109/648] Add HIR MAP to glossary --- src/glossary.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/glossary.md b/src/glossary.md index 34b4bd446..13567a428 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -13,6 +13,7 @@ DAG | a directed acyclic graph is used during compilation t DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. HIR | the High-level IR, created by lowering and desugaring the AST ([see more](hir.html)) HirId | identifies a particular node in the HIR by combining a def-id with an "intra-definition offset". +HIR Map | The HIR map, accessible via tcx.hir, allows you to quickly navigate the HIR and convert between various forms of identifiers. 'gcx | the lifetime of the global arena ([see more](ty.html)) generics | the set of generic type parameters defined on a type or item ICE | internal compiler error. When the compiler crashes. From 18e8e255518a543fabd3c49de125311877f8cf58 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Thu, 15 Feb 2018 12:46:42 -0600 Subject: [PATCH 110/648] Add some codegen related terminology to glossary --- src/glossary.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/src/glossary.md b/src/glossary.md index a770bdf8e..5d76cf3cf 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -11,19 +11,20 @@ completeness | completeness is a technical term in type theory. Comp cx | we tend to use "cx" as an abbrevation for context. See also `tcx`, `infcx`, etc. DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](incremental-compilation.html)) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. +'gcx | the lifetime of the global arena ([see more](ty.html)) +generics | the set of generic type parameters defined on a type or item HIR | the High-level IR, created by lowering and desugaring the AST ([see more](hir.html)) HirId | identifies a particular node in the HIR by combining a def-id with an "intra-definition offset". HIR Map | The HIR map, accessible via tcx.hir, allows you to quickly navigate the HIR and convert between various forms of identifiers. -'gcx | the lifetime of the global arena ([see more](ty.html)) -generics | the set of generic type parameters defined on a type or item ICE | internal compiler error. When the compiler crashes. ICH | incremental compilation hash. ICHs are used as fingerprints for things such as HIR and crate metadata, to check if changes have been made. This is useful in incremental compilation to see if part of a crate has changed and should be recompiled. infcx | the inference context (see `librustc/infer`) -MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) -miri | an interpreter for MIR used for constant evaluation ([see more](./miri.html)) -obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) +IR | Intermediate Representation. A general term in compilers. During compilation, the code is transformed from raw source (ASCII text) to various IRs. In Rust, these are primarily HIR, MIR, and LLVM IR. Each IR is well-suited for some set of computations. For example, MIR is well-suited for the borrow checker, and LLVM IR is well-suited for codegen because LLVM accepts it. local crate | the crate currently being compiled. +LTO | Link-Time Optimizations. A set of optimizations offered by LLVM that occur just before the final binary is linked. These include optmizations like removing functions that are never used in the final program, for example. _ThinLTO_ is a variant of LTO that aims to be a bit more scalable and efficient, but possibly sacrifices some optimizations. You may also read issues in the Rust repo about "FatLTO", which is the loving nickname given to non-Thin LTO. LLVM documentation: [here][lto] and [here][thinlto] +[LLVM] | (actually not an acronym :P) an open-source compiler backend. It accepts LLVM IR and outputs native binaries. Various languages (e.g. Rust) can then implement a compiler front-end that output LLVM IR and use LLVM to compile to all the platforms LLVM supports. MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) +miri | an interpreter for MIR used for constant evaluation ([see more](./miri.html)) node-id or NodeId | an index identifying a particular node in the AST or HIR; gradually being phased out and replaced with `HirId`. obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) provider | the function that executes a query ([see more](query.html)) @@ -38,6 +39,12 @@ substs | the substitutions for a given generic type or item (e tcx | the "typing context", main data structure of the compiler ([see more](ty.html)) 'tcx | the lifetime of the currently active inference context ([see more](ty.html)) token | the smallest unit of parsing. Tokens are produced after lexing ([see more](the-parser.html)). +[TLS] | Thread-Local Storage. Variables may be defined so that each thread has its own copy (rather than all threads sharing the variable). This has some interactions with LLVM. Not all platforms support TLS. trans | the code to translate MIR into LLVM IR. trait reference | a trait and values for its type parameters ([see more](ty.html)). ty | the internal representation of a type ([see more](ty.html)). + +[LLVM]: https://llvm.org/ +[lto]: https://llvm.org/docs/LinkTimeOptimization.html +[thinlto]: https://clang.llvm.org/docs/ThinLTO.html +[TLS]: https://llvm.org/docs/LangRef.html#thread-local-storage-models From ae39f92b66c8d7e4513f8340c857c67d1e283613 Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Sat, 24 Feb 2018 12:50:48 -0800 Subject: [PATCH 111/648] Start documentation for test infrastructure. CC #10 --- src/tests/intro.md | 67 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 66 insertions(+), 1 deletion(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index 2388e4308..8ad2492b7 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -98,7 +98,72 @@ including: ## Testing infrastructure -TODO - bors, platforms, etc. +When a Pull Request is opened on Github, [Travis] will automatically launch a +build that will run all tests on a single configuration (x86-64 linux). In +essence, it runs `./x.py test` after building. + +The integration bot [bors] is used for coordinating merges to the master +branch. When a PR is approved, it goes into a [queue] where merges are tested +one at a time on a wide set of platforms using Travis and [Appveyor] +(currently over 50 different configurations). Most platforms only run the +build steps, some run a restricted set of tests, only a subset run the full +suite of tests (see Rust's [platform tiers]). + +[Travis]: https://travis-ci.org/rust-lang/rust +[bors]: https://github.com/servo/homu +[queue]: https://buildbot2.rust-lang.org/homu/queue/rust +[Appveyor]: https://ci.appveyor.com/project/rust-lang/rust +[platform tiers]: https://forge.rust-lang.org/platform-support.html + +## Testing with Docker images + +The Rust tree includes [Docker] image definitions for the platforms used on +Travis in [src/ci/docker]. The script [src/ci/docker/run.sh] is used to build +the Docker image, run it, build Rust within the image, and run the tests. + +> TODO: What is a typical workflow for testing/debugging on a platform that +> you don't have easy access to? Do people build Docker images and enter them +> to test things out? + +[Docker]: https://www.docker.com/ +[src/ci/docker]: https://github.com/rust-lang/rust/tree/master/src/ci/docker +[src/ci/docker/run.sh]: https://github.com/rust-lang/rust/blob/master/src/ci/docker/run.sh + +## Testing on emulators + +Some platforms are tested via an emulator for architectures that aren't +readily available. There is a set of tools for orchestrating running the +tests within the emulator. Platforms such as `arm-android` and +`arm-unknown-linux-gnueabihf` are set up to automatically run the tests under +emulation on Travis. The following will take a look at how a target's tests +are run under emulation. + +The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU +architecture. Included in the Rust tree are the tools [remote-test-client] +and [remote-test-server] which are programs for sending test programs and +libraries to the emulator, and running the tests within the emulator, and +reading the results. The Docker image is set up to launch +`remote-test-server` and the build tools use `remote-test-client` to +communicate with the server to coordinate running tests (see +[src/bootstrap/test.rs]). + +> TODO: What are the steps for manually running tests within an emulator? +> `./src/ci/docker/run.sh armhf-gnu` will do everything, but takes hours to +> run and doesn't offer much help with interacting within the emulator. +> +> Is there any support for emulating other (non-Android) platforms, such as +> running on an iOS emulator? +> +> Is there anything else interesting that can be said here about running tests +> remotely on real hardware? +> +> It's also unclear to me how the wasm or asm.js tests are run. + +[armhf-gnu]: https://github.com/rust-lang/rust/tree/master/src/ci/docker/armhf-gnu +[QEMU]: https://www.qemu.org/ +[remote-test-client]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-client +[remote-test-server]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-server +[src/bootstrap/test.rs]: https://github.com/rust-lang/rust/tree/master/src/bootstrap/test.rs ## Crater From 54873601a80b829ccee7a8f05c84a057842de89b Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 24 Feb 2018 17:14:56 -0500 Subject: [PATCH 112/648] rework the MIR intro section, breaking out passes and visitors --- src/SUMMARY.md | 4 + src/background.md | 122 ++++++++++++++ src/glossary.md | 2 + src/mir-background.md | 122 ++++++++++++++ src/mir-borrowck.md | 57 ++++++- src/mir-passes.md | 169 +++++++++++++++++++ src/mir-regionck.md | 376 ++++++++++++++++++++++++++++++++++++++++++ src/mir-visitor.md | 45 +++++ src/mir.md | 303 +++++++++++++++++++++++++--------- 9 files changed, 1121 insertions(+), 79 deletions(-) create mode 100644 src/background.md create mode 100644 src/mir-background.md create mode 100644 src/mir-passes.md create mode 100644 src/mir-regionck.md create mode 100644 src/mir-visitor.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 4ead05705..29ad79998 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -24,11 +24,15 @@ - [Type checking](./type-checking.md) - [The MIR (Mid-level IR)](./mir.md) - [MIR construction](./mir-construction.md) + - [MIR visitor](./mir-visitor.md) + - [MIR passes: getting the MIR for a function](./mir-passes.md) - [MIR borrowck](./mir-borrowck.md) + - [MIR-based region checking (NLL)](./mir-regionck.md) - [MIR optimizations](./mir-optimizations.md) - [Constant evaluation](./const-eval.md) - [miri const evaluator](./miri.md) - [Parameter Environments](./param_env.md) - [Generating LLVM IR](./trans.md) +- [Background material](./background.md) - [Glossary](./glossary.md) - [Code Index](./code-index.md) diff --git a/src/background.md b/src/background.md new file mode 100644 index 000000000..92ae6507a --- /dev/null +++ b/src/background.md @@ -0,0 +1,122 @@ +# Background topics + +This section covers a numbers of common compiler terms that arise in +this guide. We try to give the general definition while providing some +Rust-specific context. + + + +## What is a control-flow graph? + +A control-flow graph is a common term from compilers. If you've ever +used a flow-chart, then the concept of a control-flow graph will be +pretty familiar to you. It's a representation of your program that +exposes the underlying control flow in a very clear way. + +A control-flow graph is structured as a set of **basic blocks** +connected by edges. The key idea of a basic block is that it is a set +of statements that execute "together" -- that is, whenever you branch +to a basic block, you start at the first statement and then execute +all the remainder. Only at the end of the is there the possibility of +branching to more than one place (in MIR, we call that final statement +the **terminator**): + +``` +bb0: { + statement0; + statement1; + statement2; + ... + terminator; +} +``` + +Many expressions that you are used to in Rust compile down to multiple +basic blocks. For example, consider an if statement: + +```rust +a = 1; +if some_variable { + b = 1; +} else { + c = 1; +} +d = 1; +``` + +This would compile into four basic blocks: + +``` +BB0: { + a = 1; + if some_variable { goto BB1 } else { goto BB2 } +} + +BB1: { + b = 1; + goto BB3; +} + +BB2: { + c = 1; + goto BB3; +} + +BB3: { + d = 1; + ...; +} +``` + +When using a control-flow graph, a loop simply appears as a cycle in +the graph, and the `break` keyword translates into a path out of that +cycle. + + + +## What is a dataflow analysis? + +*to be written* + + + +## What is "universally quantified"? What about "existentially quantified"? + +*to be written* + + + +## What is co- and contra-variance? + +*to be written* + + + +## What is a "free region" or a "free variable"? What about "bound region"? + +Let's describe the concepts of free vs bound in terms of program +variables, since that's the thing we're most familiar with. + +- Consider this expression: `a + b`. In this expression, `a` and `b` + refer to local variables that are defined *outside* of the + expression. We say that those variables **appear free** in the + expression. To see why this term makes sense, consider the next + example. +- In contrast, consider this expression, which creates a closure: `|a, + b| a + b`. Here, the `a` and `b` in `a + b` refer to the arguments + that the closure will be given when it is called. We say that the + `a` and `b` there are **bound** to the closure, and that the closure + signature `|a, b|` is a **binder** for the names `a` and `b` + (because any references to `a` or `b` within refer to the variables + that it introduces). + +So there you have it: a variable "appears free" in some +expression/statement/whatever if it refers to something defined +outside of that expressions/statement/whatever. Equivalently, we can +then refer to the "free variables" of an expression -- which is just +the set of variables that "appear free". + +So what does this have to do with regions? Well, we can apply the +analogous concept to type and regions. For example, in the type `&'a +u32`, `'a` appears free. But in the type `for<'a> fn(&'a u32)`, it +does not. diff --git a/src/glossary.md b/src/glossary.md index 5d76cf3cf..2aa9b52f1 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -18,6 +18,7 @@ HirId | identifies a particular node in the HIR by combining HIR Map | The HIR map, accessible via tcx.hir, allows you to quickly navigate the HIR and convert between various forms of identifiers. ICE | internal compiler error. When the compiler crashes. ICH | incremental compilation hash. ICHs are used as fingerprints for things such as HIR and crate metadata, to check if changes have been made. This is useful in incremental compilation to see if part of a crate has changed and should be recompiled. +inference variable | when doing type or region inference, an "inference variable" is a kind of special type/region that represents value you are trying to find. Think of `X` in algebra. infcx | the inference context (see `librustc/infer`) IR | Intermediate Representation. A general term in compilers. During compilation, the code is transformed from raw source (ASCII text) to various IRs. In Rust, these are primarily HIR, MIR, and LLVM IR. Each IR is well-suited for some set of computations. For example, MIR is well-suited for the borrow checker, and LLVM IR is well-suited for codegen because LLVM accepts it. local crate | the crate currently being compiled. @@ -25,6 +26,7 @@ LTO | Link-Time Optimizations. A set of optimizations offer [LLVM] | (actually not an acronym :P) an open-source compiler backend. It accepts LLVM IR and outputs native binaries. Various languages (e.g. Rust) can then implement a compiler front-end that output LLVM IR and use LLVM to compile to all the platforms LLVM supports. MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) miri | an interpreter for MIR used for constant evaluation ([see more](./miri.html)) +newtype | a "newtype" is a wrapper around some other type (e.g., `struct Foo(T)` is a "newtype" for `T`). This is commonly used in Rust to give a stronger type for indices. node-id or NodeId | an index identifying a particular node in the AST or HIR; gradually being phased out and replaced with `HirId`. obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) provider | the function that executes a query ([see more](query.html)) diff --git a/src/mir-background.md b/src/mir-background.md new file mode 100644 index 000000000..38fba5d16 --- /dev/null +++ b/src/mir-background.md @@ -0,0 +1,122 @@ +# MIR Background topics + +This section covers a numbers of common compiler terms that arise when +talking about MIR and optimizations. We try to give the general +definition while providing some Rust-specific context. + + + +## What is a control-flow graph? + +A control-flow graph is a common term from compilers. If you've ever +used a flow-chart, then the concept of a control-flow graph will be +pretty familiar to you. It's a representation of your program that +exposes the underlying control flow in a very clear way. + +A control-flow graph is structured as a set of **basic blocks** +connected by edges. The key idea of a basic block is that it is a set +of statements that execute "together" -- that is, whenever you branch +to a basic block, you start at the first statement and then execute +all the remainder. Only at the end of the is there the possibility of +branching to more than one place (in MIR, we call that final statement +the **terminator**): + +``` +bb0: { + statement0; + statement1; + statement2; + ... + terminator; +} +``` + +Many expressions that you are used to in Rust compile down to multiple +basic blocks. For example, consider an if statement: + +```rust +a = 1; +if some_variable { + b = 1; +} else { + c = 1; +} +d = 1; +``` + +This would compile into four basic blocks: + +``` +BB0: { + a = 1; + if some_variable { goto BB1 } else { goto BB2 } +} + +BB1: { + b = 1; + goto BB3; +} + +BB2: { + c = 1; + goto BB3; +} + +BB3: { + d = 1; + ...; +} +``` + +When using a control-flow graph, a loop simply appears as a cycle in +the graph, and the `break` keyword translates into a path out of that +cycle. + + + +## What is a dataflow analysis? + +*to be written* + + + +## What is "universally quantified"? What about "existentially quantified"? + +*to be written* + + + +## What is co- and contra-variance? + +*to be written* + + + +## What is a "free region" or a "free variable"? What about "bound region"? + +Let's describe the concepts of free vs bound in terms of program +variables, since that's the thing we're most familiar with. + +- Consider this expression: `a + b`. In this expression, `a` and `b` + refer to local variables that are defined *outside* of the + expression. We say that those variables **appear free** in the + expression. To see why this term makes sense, consider the next + example. +- In contrast, consider this expression, which creates a closure: `|a, + b| a + b`. Here, the `a` and `b` in `a + b` refer to the arguments + that the closure will be given when it is called. We say that the + `a` and `b` there are **bound** to the closure, and that the closure + signature `|a, b|` is a **binder** for the names `a` and `b` + (because any references to `a` or `b` within refer to the variables + that it introduces). + +So there you have it: a variable "appears free" in some +expression/statement/whatever if it refers to something defined +outside of that expressions/statement/whatever. Equivalently, we can +then refer to the "free variables" of an expression -- which is just +the set of variables that "appear free". + +So what does this have to do with regions? Well, we can apply the +analogous concept to type and regions. For example, in the type `&'a +u32`, `'a` appears free. But in the type `for<'a> fn(&'a u32)`, it +does not. diff --git a/src/mir-borrowck.md b/src/mir-borrowck.md index 55bc9fc98..b632addc2 100644 --- a/src/mir-borrowck.md +++ b/src/mir-borrowck.md @@ -1 +1,56 @@ -# MIR borrowck +# MIR borrow check + +The borrow check is Rust's "secret sauce" -- it is tasked with +enforcing a number of properties: + +- That all variables are initialized before they are used. +- That you can't move the same value twice. +- That you can't move a value while it is borrowed. +- That you can't access a place while it is mutably borrowed (except through the reference). +- That you can't mutate a place while it is shared borrowed. +- etc + +At the time of this writing, the code is in a state of transition. The +"main" borrow checker still works by processing [the HIR](hir.html), +but that is being phased out in favor of the MIR-based borrow checker. +Doing borrow checking on MIR has two key advantages: + +- The MIR is *far* less complex than the HIR; the radical desugaring + helps prevent bugs in the borrow checker. (If you're curious, you + can see + [a list of bugs that the MIR-based borrow checker fixes here][47366].) +- Even more importantly, using the MIR enables ["non-lexical lifetimes"][nll], + which are regions derived from the control-flow graph. + +[47366]: https://github.com/rust-lang/rust/issues/47366 +[nll]: http://rust-lang.github.io/rfcs/2094-nll.html + +### Major phases of the borrow checker + +The borrow checker source is found in +[the `rustc_mir::borrow_check` module][b_c]. The main entry point is +the `mir_borrowck` query. At the time of this writing, MIR borrowck can operate +in several modes, but this text will describe only the mode when NLL is enabled +(what you get with `#![feature(nll)]`). + +[b_c]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir/borrow_check + +The overall flow of the borrow checker is as follows: + +- We first create a **local copy** C of the MIR. We will be modifying + this copy in place to modify the types and things to include + references to the new regions that we are computing. +- We then invoke `nll::replace_regions_in_mir` to modify this copy C. + Among other things, this function will replace all of the regions in + the MIR with fresh [inference variables](glossary.html). + - (More details can be found in [the regionck section](./mir-regionck.html).) +- Next, we perform a number of [dataflow analyses](./background.html#dataflow) + that compute what data is moved and when. The results of these analyses + are needed to do both borrow checking and region inference. +- Using the move data, we can then compute the values of all the regions in the MIR. + - (More details can be found in [the NLL section](./mir-regionck.html).) +- Finally, the borrow checker itself runs, taking as input (a) the + results of move analysis and (b) the regions computed by the region + checker. This allows is to figure out which loans are still in scope + at any particular point. + diff --git a/src/mir-passes.md b/src/mir-passes.md new file mode 100644 index 000000000..2fe471385 --- /dev/null +++ b/src/mir-passes.md @@ -0,0 +1,169 @@ +# MIR passes + +If you would like to get the MIR for a function (or constant, etc), +you can use the `optimized_mir(def_id)` query. This will give you back +the final, optimized MIR. For foreign def-ids, we simply read the MIR +from the other crate's metadata. But for local def-ids, the query will +construct the MIR and then iteratively optimize it by putting it +through various pipeline stages. This section describes those pipeline +stages and how you can extend them. + +To produce the `optimized_mir(D)` for a given def-id `D`, the MIR +passes through several suites of optimizations, each represented by a +query. Each suite consists of multiple optimizations and +transformations. These suites represent useful intermediate points +where we want to access the MIR for type checking or other purposes: + +- `mir_build(D)` – not a query, but this constructs the initial MIR +- `mir_const(D)` – applies some simple transformations to make MIR ready for constant evaluation; +- `mir_validated(D)` – applies some more transformations, making MIR ready for borrow checking; +- `optimized_mir(D)` – the final state, after all optimizations have been performed. + +### Seeing how the MIR changes as the compiler executes + +`-Zdump-mir=F` is a handy compiler options that will let you view the MIR +for each function at each stage of compilation. `-Zdump-mir` takes a **filter** +`F` which allows you to control which functions and which passes you are interesting +in. For example: + +```bash +> rustc -Zdump-mir=foo ... +``` + +This will dump the MIR for any function whose name contains `foo`; it +will dump the MIR both before and after every pass. Those files will +be created in the `mir_dump` directory. There will likely be quite a +lot of them! + +```bash +> cat > foo.rs +fn main() { + println!("Hello, world!"); +} +^D +> rustc -Zdump-mir=main foo.rs +> ls mir_dump/* | wc -l + 161 +``` + +The files have names like `rustc.main.000-000.CleanEndRegions.after.mir`. These +names have a number of parts: + +``` +rustc.main.000-000.CleanEndRegions.after.mir + ---- --- --- --------------- ----- either before or after + | | | name of the pass + | | index of dump within the pass (usually 0, but some passes dump intermediate states) + | index of the pass + def-path to the function etc being dumped +``` + +You can also make more selective filters. For example, `main & CleanEndRegions` will select +for things that reference *both* `main` and the pass `CleanEndRegions`: + +```bash +> rustc -Zdump-mir='main & CleanEndRegions' foo.rs +> ls mir_dump +rustc.main.000-000.CleanEndRegions.after.mir rustc.main.000-000.CleanEndRegions.before.mir +``` + +Filters can also have `|` parts to combine multiple sets of +`&`-filters. For example `main & CleanEndRegions | main & +NoLandingPads` will select *either* `main` and `CleanEndRegions` *or* +`main` and `NoLandingPads`: + +```bash +> rustc -Zdump-mir='main & CleanEndRegions | main & NoLandingPads' foo.rs +> ls mir_dump +rustc.main-promoted[0].002-000.NoLandingPads.after.mir +rustc.main-promoted[0].002-000.NoLandingPads.before.mir +rustc.main-promoted[0].002-006.NoLandingPads.after.mir +rustc.main-promoted[0].002-006.NoLandingPads.before.mir +rustc.main-promoted[1].002-000.NoLandingPads.after.mir +rustc.main-promoted[1].002-000.NoLandingPads.before.mir +rustc.main-promoted[1].002-006.NoLandingPads.after.mir +rustc.main-promoted[1].002-006.NoLandingPads.before.mir +rustc.main.000-000.CleanEndRegions.after.mir +rustc.main.000-000.CleanEndRegions.before.mir +rustc.main.002-000.NoLandingPads.after.mir +rustc.main.002-000.NoLandingPads.before.mir +rustc.main.002-006.NoLandingPads.after.mir +rustc.main.002-006.NoLandingPads.before.mir +``` + +(Here, the `main-promoted[0]` files refer to the MIR for "promoted constants" +that appeared within the `main` function.) + +### Implementing and registering a pass + +A `MirPass` is some bit of code that processes the MIR, typically -- +but not always -- transforming it along the way in some way. For +example, it might perform an optimization. The `MirPass` trait itself +is found in in [the `rustc_mir::transform` module][mirtransform], and +it basically consists of one method, `run_pass`, that simply gets an +`&mut Mir` (along with the tcx and some information about where it +came from). + +A good example of a basic MIR pass is [`NoLandingPads`], which walks the +MIR and removes all edges that are due to unwinding -- this is used +with when configured with `panic=abort`, which never unwinds. As you can see +from its source, a MIR pass is defined by first defining a dummy type, a struct +with no fields, something like: + +```rust +struct MyPass; +``` + +for which you then implement the `MirPass` trait. You can then insert +this pass into the appropriate list of passes found in a query like +`optimized_mir`, `mir_validated`, etc. (If this is an optimization, it +should go into the `optimized_mir` list.) + +If you are writing a pass, there's a good chance that you are going to +want to use a [MIR visitor] too -- those are a handy visitor that +walks the MIR for you and lets you make small edits here and there. + +### Stealing + +The intermediate queries `mir_const()` and `mir_validated()` yield up +a `&'tcx Steal>`, allocated using +`tcx.alloc_steal_mir()`. This indicates that the result may be +**stolen** by the next suite of optimizations – this is an +optimization to avoid cloning the MIR. Attempting to use a stolen +result will cause a panic in the compiler. Therefore, it is important +that you do not read directly from these intermediate queries except as +part of the MIR processing pipeline. + +Because of this stealing mechanism, some care must also be taken to +ensure that, before the MIR at a particular phase in the processing +pipeline is stolen, anyone who may want to read from it has already +done so. Concretely, this means that if you have some query `foo(D)` +that wants to access the result of `mir_const(D)` or +`mir_validated(D)`, you need to have the successor pass "force" +`foo(D)` using `ty::queries::foo::force(...)`. This will force a query +to execute even though you don't directly require its result. + +As an example, consider MIR const qualification. It wants to read the +result produced by the `mir_const()` suite. However, that result will +be **stolen** by the `mir_validated()` suite. If nothing was done, +then `mir_const_qualif(D)` would succeed if it came before +`mir_validated(D)`, but fail otherwise. Therefore, `mir_validated(D)` +will **force** `mir_const_qualif` before it actually steals, thus +ensuring that the reads have already happened: + +``` +mir_const(D) --read-by--> mir_const_qualif(D) + | ^ + stolen-by | + | (forces) + v | +mir_validated(D) ------------+ +``` + +This mechanism is a bit dodgy. There is a discussion of more elegant +alternatives in [rust-lang/rust#41710]. + +[rust-lang/rust#41710]: https://github.com/rust-lang/rust/issues/41710 +[mirtransform]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir/transform/mod.rs +[`NoLandingPads`]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir/transform/no_landing_pads.rs +[MIR visitor]: mir-visitor.html diff --git a/src/mir-regionck.md b/src/mir-regionck.md new file mode 100644 index 000000000..8be4fc8b9 --- /dev/null +++ b/src/mir-regionck.md @@ -0,0 +1,376 @@ +# MIR-based region checking (NLL) + +The MIR-based region checking code is located in +[the `rustc_mir::borrow_check::nll` module][nll]. (NLL, of course, +stands for "non-lexical lifetimes", a term that will hopefully be +deprecated once they become the standard kind of lifetime.) + +[nll]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir/borrow_check/nll + +The MIR-based region analysis consists of two major functions: + +- `replace_regions_in_mir`, invoked first, has two jobs: + - First, it analyzes the signature of the MIR and finds the set of + regions that appear in the MIR signature (e.g., `'a` in `fn + foo<'a>(&'a u32) { ... }`. These are called the "universal" or + "free" regions -- in particular, they are the regions that + [appear free][fvb] in the function body. + - Second, it replaces all the regions from the function body with + fresh inference variables. This is because (presently) those + regions are the results of lexical region inference and hence are + not of much interest. The intention is that -- eventually -- they + will be "erased regions" (i.e., no information at all), since we + don't be doing lexical region inference at all. +- `compute_regions`, invoked second: this is given as argument the + results of move analysis. It has the job of computing values for all + the inference variabes that `replace_regions_in_mir` introduced. + - To do that, it first runs the [MIR type checker](#mirtypeck). This + is basically a normal type-checker but specialized to MIR, which + is much simpler than full Rust of course. Running the MIR type + checker will however create **outlives constraints** between + region variables (e.g., that one variable must outlive another + one) to reflect the subtyping relationships that arise. + - It also adds **liveness constraints** that arise from where variables + are used. + - More details to come, though the [NLL RFC] also includes fairly thorough + (and hopefully readable) coverage. + +[fvb]: background.html#free-vs-bound +[NLL RFC]: http://rust-lang.github.io/rfcs/2094-nll.html + +## Universal regions + +*to be written* -- explain the `UniversalRegions` type + +## Region variables and constraints + +*to be written* -- describe the `RegionInferenceContext` and +the role of `liveness_constraints` vs other `constraints`, plus + +## Closures + + + +## The MIR type-check + +## Representing the "values" of a region variable + +The value of a region can be thought of as a **set**; we call the +domain of this set a `RegionElement`. In the code, the value for all +regions is maintained in +[the `rustc_mir::borrow_check::nll::region_infer` module][ri]. For +each region we maintain a set storing what elements are present in its +value (to make this efficient, we give each kind of element an index, +the `RegionElementIndex`, and use sparse bitsets). + +[ri]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir/borrow_check/nll/region_infer/ + +The kinds of region elements are as follows: + +- Each **location** in the MIR control-flow graph: a location is just + the pair of a basic block and an index. This identifies the point + **on entry** to the statement with that index (or the terminator, if + the index is equal to `statements.len()`). +- There is an element `end('a)` for each universal region `'a`, + corresponding to some portion of the caller's (or caller's caller, + etc) control-flow graph. +- Similarly, there is an element denoted `end('static)` corresponding + to the remainder of program execution after this function returns. +- There is an element `!1` for each skolemized region `!1`. This + corresponds (intuitively) to some unknown set of other elements -- + for details on skolemization, see the section + [skolemization and universes](#skol). + +## Causal tracking + +*to be written* -- describe how we can extend the values of a variable + with causal tracking etc + + + +## Skolemization and universes + +(This section describes ongoing work that hasn't landed yet.) + +From time to time we have to reason about regions that we can't +concretely know. For example, consider this program: + +```rust +// A function that needs a static reference +fn foo(x: &'static u32) { } + +fn bar(f: for<'a> fn(&'a u32)) { + // ^^^^^^^^^^^^^^^^^^^ a function that can accept **any** reference + let x = 22; + f(&x); +} + +fn main() { + bar(foo); +} +``` + +This program ought not to type-check: `foo` needs a static reference +for its argument, and `bar` wants to be given a function that that +accepts **any** reference (so it can call it with something on its +stack, for example). But *how* do we reject it and *why*? + +### Subtyping and skolemization + +When we type-check `main`, and in particular the call `bar(foo)`, we +are going to wind up with a subtyping relationship like this one: + + fn(&'static u32) <: for<'a> fn(&'a u32) + ---------------- ------------------- + the type of `foo` the type `bar` expects + +We handle this sort of subtyping by taking the variables that are +bound in the supertype and **skolemizing** them: this means that we +replace them with +[universally quantified](background.html#quantified) +representatives, written like `!1`. We call these regions "skolemized +regions" -- they represent, basically, "some unknown region". + +Once we've done that replacement, we have the following types: + + fn(&'static u32) <: fn(&'!1 u32) + +The key idea here is that this unknown region `'!1` is not related to +any other regions. So if we can prove that the subtyping relationship +is true for `'!1`, then it ought to be true for any region, which is +what we wanted. (This number `!1` is called a "universe", for reasons +we'll get into later.) + +So let's work through what happens next. To check if two functions are +subtypes, we check if their arguments have the desired relationship +(fn arguments are [contravariant](./background.html#variance), so +we swap the left and right here): + + &'!1 u32 <: &'static u32 + +According to the basic subtyping rules for a reference, this will be +true if `'!1: 'static`. That is -- if "some unknown region `!1`" lives +outlives `'static`. Now, this *might* be true -- after all, `'!1` +could be `'static` -- but we don't *know* that it's true. So this +should yield up an error (eventually). + +### Universes and skolemized region elements + +But where does that error come from? The way it happens is like this. +When we are constructing the region inference context, we can tell +from the type inference context how many skolemized variables exist +(the `InferCtxt` has an internal counter). For each of those, we +create a corresponding universal region variable `!n` and a "region +element" `skol(n)`. This corresponds to "some unknown set of other +elements". The value of `!n` is `{skol(n)}`. + +At the same time, we also give each existential variable a +**universe** (also taken from the `InferCtxt`). This universe +determines which skolemized elements may appear in its value: For +example, a variable in universe U3 may name `skol(1)`, `skol(2)`, and +`skol(3)`, but not `skol(4)`. Note that the universe of an inference +variable controls what region elements **can** appear in its value; it +does not say region elements **will** appear. + +### Skolemization and outlives constraints + +In the region inference engine, outlives constraints have the form: + + V1: V2 @ P + +where `V1` and `V2` are region indices, and hence map to some region +variable (which may be universally or existentially quantified). This +variable will have a universe, so let's call those universes `U(V1)` +and `U(V2)` respectively. (Actually, the only one we are going to care +about is `U(V1)`.) + +When we encounter this constraint, the ordinary procedure is to start +a DFS from `P`. We keep walking so long as the nodes we are walking +are present in `value(V2)` and we add those nodes to `value(V1)`. If +we reach a return point, we add in any `end(X)` elements. That part +remains unchanged. + +But then *after that* we want to iterate over the skolemized `skol(u)` +elements in V2 (each of those must be visible to `U(V2)`, but we +should be able to just assume that is true, we don't have to check +it). We have to ensure that `value(V1)` outlives each of those +skolemized elements. + +Now there are two ways that could happen. First, if `U(V1)` can see +the universe `u` (i.e., `u <= U(V1)`), then we can just add `skol(u1)` +to `value(V1)` and be done. But if not, then we have to approximate: +we may not know what set of elements `skol(u1)` represents, but we +should be able to compute some sort of **upper bound** for it -- +something that it is smaller than. For now, we'll just use `'static` +for that (since it is bigger than everything) -- in the future, we can +sometimes be smarter here (and in fact we have code for doing this +already in other contexts). Moreover, since `'static` is in U0, we +know that all variables can see it -- so basically if we find a that +`value(V2)` contains `skol(u)` for some universe `u` that `V1` can't +see, then we force `V1` to `'static`. + +### Extending the "universal regions" check + +After all constraints have been propagated, the NLL region inference +has one final check, where it goes over the values that wound up being +computed for each universal region and checks that they did not get +'too large'. In our case, we will go through each skolemized region +and check that it contains *only* the `skol(u)` element it is known to +outlive. (Later, we might be able to know that there are relationships +between two skolemized regions and take those into account, as we do +for universal regions from the fn signature.) + +Put another way, the "universal regions" check can be considered to be +checking constraints like: + + {skol(1)}: V1 + +where `{skol(1)}` is like a constant set, and V1 is the variable we +made to represent the `!1` region. + +## Back to our example + +OK, so far so good. Now let's walk through what would happen with our +first example: + + fn(&'static u32) <: fn(&'!1 u32) @ P // this point P is not imp't here + +The region inference engine will create a region element domain like this: + + { CFG; end('static); skol(1) } + --- ------------ ------- from the universe `!1` + | 'static is always in scope + all points in the CFG; not especially relevant here + +It will always create two universal variables, one representing +`'static` and one representing `'!1`. Let's call them Vs and V1. They +will have initial values like so: + + Vs = { CFG; end('static) } // it is in U0, so can't name anything else + V1 = { skol(1) } + +From the subtyping constraint above, we would have an outlives constraint like + + '!1: 'static @ P + +To process this, we would grow the value of V1 to include all of Vs: + + Vs = { CFG; end('static) } + V1 = { CFG; end('static), skol(1) } + +At that point, constraint propagation is done, because all the +outlives relationships are satisfied. Then we would go to the "check +universal regions" portion of the code, which would test that no +universal region grew too large. + +In this case, `V1` *did* grow too large -- it is not known to outlive +`end('static)`, nor any of the CFG -- so we would report an error. + +## Another example + +What about this subtyping relationship? + + for<'a> fn(&'a u32, &'a u32) + <: + for<'b, 'c> fn(&'b u32, &'c u32) + +Here we would skolemize the supertype, as before, yielding: + + for<'a> fn(&'a u32, &'a u32) + <: + fn(&'!1 u32, &'!2 u32) + +then we instantiate the variable on the left-hand side with an existential +in universe U2, yielding: + + fn(&'?3 u32, &'?3 u32) + <: + fn(&'!1 u32, &'!2 u32) + +Then we break this down further: + + &'!1 u32 <: &'?3 u32 + &'!2 u32 <: &'?3 u32 + +and even further, yield up our region constraints: + + '!1: '?3 + '!2: '?3 + +Note that, in this case, both `'!1` and `'!2` have to outlive the +variable `'?3`, but the variable `'?3` is not forced to outlive +anything else. Therefore, it simply starts and ends as the empty set +of elements, and hence the type-check succeeds here. + +(This should surprise you a little. It surprised me when I first +realized it. We are saying that if we are a fn that **needs both of +its arguments to have the same region**, we can accept being called +with **arguments with two distinct regions**. That seems intuitively +unsound. But in fact, it's fine, as I +[tried to explain in this issue on the Rust issue tracker long ago][ohdeargoditsallbroken]. +The reason is that even if we get called with arguments of two +distinct lifetimes, those two lifetimes have some intersection (the +call itself), and that intersection can be our value of `'a` that we +use as the common lifetime of our arguments. -nmatsakis) + +[ohdeargoditsallbroken]: https://github.com/rust-lang/rust/issues/32330#issuecomment-202536977 + +## Final example + +Let's look at one last example. We'll extend the previous one to have +a return type: + + for<'a> fn(&'a u32, &'a u32) -> &'a u32 + <: + for<'b, 'c> fn(&'b u32, &'c u32) -> &'b u32 + +Despite seeming very similar to the previous example, this case is +going to get an error. That's good: the problem is that we've gone +from a fn that promises to return one of its two arguments, to a fn +that is promising to return the first one. That is unsound. Let's see how it plays out. + +First, we skolemize the supertype: + + for<'a> fn(&'a u32, &'a u32) -> &'a u32 + <: + fn(&'!1 u32, &'!2 u32) -> &'!1 u32 + +Then we instantiate the subtype with existentials (in U2): + + fn(&'?3 u32, &'?3 u32) -> &'?3 u32 + <: + fn(&'!1 u32, &'!2 u32) -> &'!1 u32 + +And now we create the subtyping relationships: + + &'!1 u32 <: &'?3 u32 // arg 1 + &'!2 u32 <: &'?3 u32 // arg 2 + &'?3 u32 <: &'!1 u32 // return type + +And finally the outlives relationships. Here, let V1, V2, and V3 be the variables +we assign to `!1`, `!2`, and `?3` respectively: + + V1: V3 + V2: V3 + V3: V1 + +Those variables will have these initial values: + + V1 in U1 = {skol(1)} + V2 in U2 = {skol(2)} + V3 in U2 = {} + +Now because of the `V3: V1` constraint, we have to add `skol(1)` into `V3` (and indeed +it is visible from `V3`), so we get: + + V3 in U2 = {skol(1)} + +then we have this constraint `V2: V3`, so we wind up having to enlarge +`V2` to include `skol(1)` (which it can also see): + + V2 in U2 = {skol(1), skol(2)} + +Now contraint propagation is done, but when we check the outlives +relationships, we find that `V2` includes this new element `skol(1)`, +so we report an error. + diff --git a/src/mir-visitor.md b/src/mir-visitor.md new file mode 100644 index 000000000..824ddd5b2 --- /dev/null +++ b/src/mir-visitor.md @@ -0,0 +1,45 @@ +# MIR visitor + +The MIR visitor is a convenient tool for traversing the MIR and either +looking for things or making changes to it. The visitor traits are +defined in [the `rustc::mir::visit` module][m-v] -- there are two of +them, generated via a single macro: `Visitor` (which operates on a +`&Mir` and gives back shared references) and `MutVisitor` (which +operates on a `&mut Mir` and gives back mutable references). + +[m-v]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir/visit.rs + +To implement a visitor, you have to create a type that represents +your visitor. Typically, this type wants to "hang on" to whatever +state you will need while processing MIR: + +```rust +struct MyVisitor<...> { + tcx: TyCtxt<'cx, 'tcx, 'tcx>, + ... +} +``` + +and you then implement the `Visitor` or `MutVisitor` trait for that type: + +```rust +impl<'tcx> MutVisitor<'tcx> for NoLandingPads { + fn visit_foo(&mut self, ...) { + // ... + self.super_foo(...); + } +} +``` + +As shown above, within the impl, you can override any of the +`visit_foo` methods (e.g., `visit_terminator`) in order to write some +code that will execute whenever a `foo` is found. If you want to +recursively walk the contents of the `foo`, you then invoke the +`super_foo` method. (NB. You never want to override `super_foo`.) + +A very simple example of a visitor can be found in [`NoLandingPads`]. +That visitor doesn't even require any state: it just visits all +terminators and removes their `unwind` successors. + +[`NoLandingPads`]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir/transform/no_landing_pads.rs + diff --git a/src/mir.md b/src/mir.md index 34f6bbb8f..262082ed7 100644 --- a/src/mir.md +++ b/src/mir.md @@ -1,26 +1,229 @@ # The MIR (Mid-level IR) -MIR is Rust's _Mid-level Intermediate Representation_. It is constructed from -HIR (described in an earlier chapter). +MIR is Rust's _Mid-level Intermediate Representation_. It is +constructed from HIR (described in an earlier chapter). MIR was +introduced in [RFC 1211]. It is a radically simplified form of Rust +that is used for certain flow-sensitive safety checks -- notably the +borrow checker! -- and also for optimization and code generation. + +If you'd like a very high-level introduction to MIR, as well as some +of the compiler concepts that it relies on (such as control-flow +graphs and desugaring), you may enjoy the +[rust-lang blog post that introduced MIR][blog]. + +[blog]: https://blog.rust-lang.org/2016/04/19/MIR.html + +## Introduction to MIR MIR is defined in the [`src/librustc/mir/`][mir] module, but much of the code that manipulates it is found in [`src/librustc_mir`][mirmanip]. +[RFC 1211]: http://rust-lang.github.io/rfcs/1211-mir.html + +Some of the key characteristics of MIR are: + +- It is based on a [control-flow graph][cfg]. +- It does not have nested expressions. +- All types in MIR are fully explicit. + +[cfg]: ./background.html#cfg + +## Key MIR vocabulary + +This section introduces the key concepts of MIR, summarized here: + +- **Basic blocks**: units of the control-flow graph, consisting of: + - **statements:** actions with one successor + - **terminators:** actions with potentially multiple successors; always at the end of a block + - (if you're not familiar with the term basic block, see the [MIR background chapter][bg]) +- **Locals:** Memory locations alloated on the stack (conceptually, at + least), such as function arguments, local variables, and + temporaries. These are identified by an index, written with a + leading underscore, like `_1`. There is also a special "local" + (`_0`) allocated to store the return value. +- **Places:** expressions that identify a location in memory, like `_1` or `_1.f`. +- **Rvalues:** expressions that product a value. The "R" stands for + the fact that these are the "right-hand side" of an assignment. + - **Operands:** the arguments to an rvalue, which can either be a + constant (like `22`) or a place (like `_1`). + +You can get a feeling for how MIR is structed by translating simple +programs into MIR and ready the pretty printed output. In fact, the +playground makes this easy, since it supplies a MIR button that will +show you the MIR for your program. Try putting this program into play +(or [clicking on this link][sample-play]), and then clicking the "MIR" +button on the top: + +[sample-play]: https://play.rust-lang.org/?gist=30074856e62e74e91f06abd19bd72ece&version=stable + +```rust +fn main() { + let mut vec = Vec::new(); + vec.push(1); + vec.push(2); +} +``` + +You should see something like: + +``` +// WARNING: This output format is intended for human consumers only +// and is subject to change without notice. Knock yourself out. +fn main() -> () { + ... +} +``` + +This is the MIR format for the `main` function. + +**Variable declarations.** If we drill in a bit, we'll see it begins +with a bunch of variable declarations. They look like this: + +``` +let mut _0: (); // return place +scope 1 { + let mut _1: std::vec::Vec; // "vec" in scope 1 at src/main.rs:2:9: 2:16 +} +scope 2 { +} +let mut _2: (); +let mut _3: &mut std::vec::Vec; +let mut _4: (); +let mut _5: &mut std::vec::Vec; +``` + +You can see that variables in MIR don't have names, they have indices, +like `_0` or `_1`. We also intermingle the user's variables (e.g., +`_1`) with temporary values (e.g., `_2` or `_3`). You can tell the +difference between user-defined variables have a comment that gives +you their original name (`// "vec" in scope 1...`). + +**Basic blocks.** Reading further, we see our first **basic block** (naturally it may look +slightly different when you view it, and I am ignoring some of the comments): + +``` +bb0: { + StorageLive(_1); + _1 = const >::new() -> bb2; +} +``` + +A basic block is defined by a series of **statements** and a final **terminator**. +In this case, there is one statement: + +``` +StorageLive(_1); +``` + +This statement indicates that the variable `_1` is "live", meaning +that it may be used later -- this will persist until we encounter a +`StorageDead(_1)` statement, which indicates that the variable `_1` is +done being used. These "storage statements" are used by LLVM to +allocate stack space. -_NOTE: copy/pasted from README... needs editing_ +The **terminator** of the block `bb0` is the call to `Vec::new`: -# MIR definition and pass system +``` +_1 = const >::new() -> bb2; +``` + +Terminators are different from statements because they can have more +than one successor -- that is, control may flow to different +places. Function calls like the call to `Vec::new` are always +terminators because of the possibility of unwinding, although in the +case of `Vec::new` we are able to see that indeed unwinding is not +possible, and hence we list only one succssor block, `bb2`. + +If we look ahead to `bb2`, we will see it looks like this: + +``` +bb2: { + StorageLive(_3); + _3 = &mut _1; + _2 = const >::push(move _3, const 1i32) -> [return: bb3, unwind: bb4]; +} +``` + +Here there are two statements: another `StorageLive`, introducing the `_3` temporary, +and then an assignment: -This file contains the definition of the MIR datatypes along with the -various types for the "MIR Pass" system, which lets you easily -register and define new MIR transformations and analyses. +``` +_3 = &mut _1; +``` -Most of the code that operates on MIR can be found in the -`librustc_mir` crate or other crates. The code found here in -`librustc` is just the datatype definitions, along with the functions -which operate on MIR to be placed everywhere else. +Assignments in general have the form: -## MIR Data Types and visitor +``` + = +``` + +A place is an expression like `_3`, `_3.f` or `*_3` -- it denotes a +location in memory. An **Rvalue** is an expression that creates a +value: in this case, the rvalue is a mutable borrow expression, which +looks like `&mut `. So we can kind of define a grammar for +rvalues like so: + +``` + = & (mut)? + | + + | - + | ... + + = Constant + | copy Place + | move Place +``` + +As you can see from this grammar, rvalues cannot be nested -- they can +only reference places and constants. Moreover, when you use a place, +we indicate whether we are **copying it** (which requires that the +place have a type `T` where `T: Copy`) or **moving it** (which works +for a place of any type). So, for example, if we had the expression `x += a + b + c` in Rust, that would get compile to two statements and a +temporary: + +``` +TMP1 = a + b +x = TMP1 + c +``` + +([Try it and see, though you may want to do release mode to skip over the overflow checks.][play-abc]) + +[play-abc]: https://play.rust-lang.org/?gist=1751196d63b2a71f8208119e59d8a5b6&version=stable + +## MIR data types + +The MIR data types are defined in the [`src/librustc/mir/`][mir] +module. Each of the key concepts mentioned in the previous section +maps in a fairly straightforward way to a Rust type. + +The main MIR data type is `Mir`. It contains the data for a single +function (along with sub-instances of Mir for "promoted constants", +but [you can read about those below](#promoted)). + +- **Basic blocks**: The basic blocks are stored in the field + `basic_blocks`; this is a vector of `BasicBlockData` + structures. Nobody ever references a basic block directly: instead, + we pass around `BasicBlock` values, which are + [newtype'd] indices into this vector. +- **Statements** are represented by the type `Statement`. +- **Terminators** are represented by the `Terminator`. +- **Locals** are represented by a [newtype'd] index type `Local`. The + data for a local variable is found in the `Mir` (the `local_decls` + vector). There is also a special constant `RETURN_PLACE` identifying + the special "local" representing the return value. +- **Places** are identified by the enum `Place`. There are a few variants: + - Local variables like `_1` + - Static variables `FOO` + - **Projections**, which are fields or other things that "project + out" from a base place. So e.g. the place `_1.f` is a projection, + with `f` being the "projection element and `_1` being the base + path. `*_1` is also a projection, with the `*` being represented + by the `ProjectionElem::Deref` element. +- **Rvalues** are represented by the enum `Rvalue`. +- **Operands** are represented by the enum `Operand`. + +## MIR Visitor The main MIR data type is `rustc::mir::Mir`, defined in `mod.rs`. There is also the MIR visitor (in `visit.rs`) which allows you to walk @@ -32,74 +235,18 @@ routines for visiting the MIR CFG in [different standard orders][traversal] [traversal]: https://en.wikipedia.org/wiki/Tree_traversal -## MIR pass suites and their integration into the query system - -As a MIR *consumer*, you are expected to use one of the queries that -returns a "final MIR". As of the time of this writing, there is only -one: `optimized_mir(def_id)`, but more are expected to come in the -future. For foreign def-ids, we simply read the MIR from the other -crate's metadata. But for local def-ids, the query will construct the -MIR and then iteratively optimize it by putting it through various -pipeline stages. This section describes those pipeline stages and how -you can extend them. - -To produce the `optimized_mir(D)` for a given def-id `D`, the MIR -passes through several suites of optimizations, each represented by a -query. Each suite consists of multiple optimizations and -transformations. These suites represent useful intermediate points -where we want to access the MIR for type checking or other purposes: - -- `mir_build(D)` – not a query, but this constructs the initial MIR -- `mir_const(D)` – applies some simple transformations to make MIR ready for constant evaluation; -- `mir_validated(D)` – applies some more transformations, making MIR ready for borrow checking; -- `optimized_mir(D)` – the final state, after all optimizations have been performed. - -### Stealing - -The intermediate queries `mir_const()` and `mir_validated()` yield up -a `&'tcx Steal>`, allocated using -`tcx.alloc_steal_mir()`. This indicates that the result may be -**stolen** by the next suite of optimizations – this is an -optimization to avoid cloning the MIR. Attempting to use a stolen -result will cause a panic in the compiler. Therefore, it is important -that you do not read directly from these intermediate queries except as -part of the MIR processing pipeline. - -Because of this stealing mechanism, some care must also be taken to -ensure that, before the MIR at a particular phase in the processing -pipeline is stolen, anyone who may want to read from it has already -done so. Concretely, this means that if you have some query `foo(D)` -that wants to access the result of `mir_const(D)` or -`mir_validated(D)`, you need to have the successor pass "force" -`foo(D)` using `ty::queries::foo::force(...)`. This will force a query -to execute even though you don't directly require its result. - -As an example, consider MIR const qualification. It wants to read the -result produced by the `mir_const()` suite. However, that result will -be **stolen** by the `mir_validated()` suite. If nothing was done, -then `mir_const_qualif(D)` would succeed if it came before -`mir_validated(D)`, but fail otherwise. Therefore, `mir_validated(D)` -will **force** `mir_const_qualif` before it actually steals, thus -ensuring that the reads have already happened: - -``` -mir_const(D) --read-by--> mir_const_qualif(D) - | ^ - stolen-by | - | (forces) - v | -mir_validated(D) ------------+ -``` - -### Implementing and registering a pass - -To create a new MIR pass, you simply implement the `MirPass` trait for -some fresh singleton type `Foo`. Once you have implemented a trait for -your type `Foo`, you then have to insert `Foo` into one of the suites; -this is done in `librustc_driver/driver.rs` by invoking `push_pass(S, -Foo)` with the appropriate suite substituted for `S`. +## Representing constants + +TBD + + + +### Promoted constants + +TBD [mir]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir [mirmanip]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir [mir]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir +[newtype'd]: glossary.html From 906746500b74cf023ec576df749982415ff39776 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sun, 25 Feb 2018 20:55:56 -0500 Subject: [PATCH 113/648] apply mark-i-m's suggestions --- src/SUMMARY.md | 2 +- src/background.md | 20 ++--- src/glossary.md | 12 ++- src/mir-background.md | 122 ------------------------------ src/mir-borrowck.md | 8 +- src/mir-passes.md | 36 +++++---- src/mir-regionck.md | 167 +++++++++++++++++++++++++++++++++++------- src/mir-visitor.md | 10 +++ src/mir.md | 34 +++------ 9 files changed, 209 insertions(+), 202 deletions(-) delete mode 100644 src/mir-background.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 29ad79998..ab9da7295 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -24,7 +24,7 @@ - [Type checking](./type-checking.md) - [The MIR (Mid-level IR)](./mir.md) - [MIR construction](./mir-construction.md) - - [MIR visitor](./mir-visitor.md) + - [MIR visitor and traversal](./mir-visitor.md) - [MIR passes: getting the MIR for a function](./mir-passes.md) - [MIR borrowck](./mir-borrowck.md) - [MIR-based region checking (NLL)](./mir-regionck.md) diff --git a/src/background.md b/src/background.md index 92ae6507a..50c247774 100644 --- a/src/background.md +++ b/src/background.md @@ -17,9 +17,9 @@ A control-flow graph is structured as a set of **basic blocks** connected by edges. The key idea of a basic block is that it is a set of statements that execute "together" -- that is, whenever you branch to a basic block, you start at the first statement and then execute -all the remainder. Only at the end of the is there the possibility of -branching to more than one place (in MIR, we call that final statement -the **terminator**): +all the remainder. Only at the end of the block is there the +possibility of branching to more than one place (in MIR, we call that +final statement the **terminator**): ``` bb0: { @@ -88,7 +88,8 @@ cycle. ## What is co- and contra-variance? -*to be written* +Check out the subtyping chapter from the +[Rust Nomicon](https://doc.rust-lang.org/nomicon/subtyping.html). @@ -97,18 +98,17 @@ cycle. Let's describe the concepts of free vs bound in terms of program variables, since that's the thing we're most familiar with. -- Consider this expression: `a + b`. In this expression, `a` and `b` - refer to local variables that are defined *outside* of the - expression. We say that those variables **appear free** in the - expression. To see why this term makes sense, consider the next - example. -- In contrast, consider this expression, which creates a closure: `|a, +- Consider this expression, which creates a closure: `|a, b| a + b`. Here, the `a` and `b` in `a + b` refer to the arguments that the closure will be given when it is called. We say that the `a` and `b` there are **bound** to the closure, and that the closure signature `|a, b|` is a **binder** for the names `a` and `b` (because any references to `a` or `b` within refer to the variables that it introduces). +- Consider this expression: `a + b`. In this expression, `a` and `b` + refer to local variables that are defined *outside* of the + expression. We say that those variables **appear free** in the + expression (i.e., they are **free**, not **bound** (tied up)). So there you have it: a variable "appears free" in some expression/statement/whatever if it refers to something defined diff --git a/src/glossary.md b/src/glossary.md index 2aa9b52f1..81eb62bc9 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -6,11 +6,16 @@ The compiler uses a number of...idiosyncratic abbreviations and things. This glo Term | Meaning ------------------------|-------- AST | the abstract syntax tree produced by the syntax crate; reflects user syntax very closely. +binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and `|a| ...` is a binder for the parameter `a`. See [the background chapter for more](./background.html#free-vs-bound) +bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession `|a| a * 2`. See [the background chapter for more](./background.html#free-vs-bound) codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. completeness | completeness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). +control-flow graph | a representation of the control-flow of a program; see [the background chapter for more](./background.html#cfg) cx | we tend to use "cx" as an abbrevation for context. See also `tcx`, `infcx`, etc. DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](incremental-compilation.html)) +data-flow analysis | a static analysis that figures out what properties are true at each point in the control-flow of a program; see [the background chapter for more](./background.html#dataflow) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. +free variable | a "free variable" is one that is not bound within an expression or term; see [the background chapter for more](./background.html#free-vs-bound) 'gcx | the lifetime of the global arena ([see more](ty.html)) generics | the set of generic type parameters defined on a type or item HIR | the High-level IR, created by lowering and desugaring the AST ([see more](hir.html)) @@ -18,7 +23,7 @@ HirId | identifies a particular node in the HIR by combining HIR Map | The HIR map, accessible via tcx.hir, allows you to quickly navigate the HIR and convert between various forms of identifiers. ICE | internal compiler error. When the compiler crashes. ICH | incremental compilation hash. ICHs are used as fingerprints for things such as HIR and crate metadata, to check if changes have been made. This is useful in incremental compilation to see if part of a crate has changed and should be recompiled. -inference variable | when doing type or region inference, an "inference variable" is a kind of special type/region that represents value you are trying to find. Think of `X` in algebra. +inference variable | when doing type or region inference, an "inference variable" is a kind of special type/region that represents what you are trying to infer. Think of X in algebra. For example, if we are trying to infer the type of a variable in a program, we create an inference variable to represent that unknown type. infcx | the inference context (see `librustc/infer`) IR | Intermediate Representation. A general term in compilers. During compilation, the code is transformed from raw source (ASCII text) to various IRs. In Rust, these are primarily HIR, MIR, and LLVM IR. Each IR is well-suited for some set of computations. For example, MIR is well-suited for the borrow checker, and LLVM IR is well-suited for codegen because LLVM accepts it. local crate | the crate currently being compiled. @@ -27,14 +32,18 @@ LTO | Link-Time Optimizations. A set of optimizations offer MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) miri | an interpreter for MIR used for constant evaluation ([see more](./miri.html)) newtype | a "newtype" is a wrapper around some other type (e.g., `struct Foo(T)` is a "newtype" for `T`). This is commonly used in Rust to give a stronger type for indices. +NLL | [non-lexical lifetimes](./mir-regionck.html), an extension to Rust's borrowing system to make it be based on the control-flow graph. node-id or NodeId | an index identifying a particular node in the AST or HIR; gradually being phased out and replaced with `HirId`. obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) +promoted constants | constants extracted from a function and lifted to static scope; see [this section](./mir.html#promoted) for more details. provider | the function that executes a query ([see more](query.html)) +quantified | in math or logic, existential and universal quantification are used to ask questions like "is there any type T for which is true?" or "is this true for all types T?"; see [the background chapter for more](./background.html#quantified) query | perhaps some sub-computation during compilation ([see more](query.html)) region | another term for "lifetime" often used in the literature and in the borrow checker. sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. sigil | like a keyword but composed entirely of non-alphanumeric tokens. For example, `&` is a sigil for references. +skolemization | a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)` as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on skolemization and universes](./mir-regionck.html#skol) for more details. soundness | soundness is a technical term in type theory. Roughly, if a type system is sound, then if a program type-checks, it is type-safe; i.e. I can never (in safe rust) force a value into a variable of the wrong type. (see "completeness"). span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. substs | the substitutions for a given generic type or item (e.g. the `i32`, `u32` in `HashMap`) @@ -45,6 +54,7 @@ token | the smallest unit of parsing. Tokens are produced aft trans | the code to translate MIR into LLVM IR. trait reference | a trait and values for its type parameters ([see more](ty.html)). ty | the internal representation of a type ([see more](ty.html)). +variance | variance determines how changes to a generic type/lifetime parameter affect subtyping; for example, if `T` is a subtype of `U`, then `Vec` is a subtype `Vec` because `Vec` is *covariant* in its generic parameter. See [the background chapter for more](./background.html#variance). [LLVM]: https://llvm.org/ [lto]: https://llvm.org/docs/LinkTimeOptimization.html diff --git a/src/mir-background.md b/src/mir-background.md deleted file mode 100644 index 38fba5d16..000000000 --- a/src/mir-background.md +++ /dev/null @@ -1,122 +0,0 @@ -# MIR Background topics - -This section covers a numbers of common compiler terms that arise when -talking about MIR and optimizations. We try to give the general -definition while providing some Rust-specific context. - - - -## What is a control-flow graph? - -A control-flow graph is a common term from compilers. If you've ever -used a flow-chart, then the concept of a control-flow graph will be -pretty familiar to you. It's a representation of your program that -exposes the underlying control flow in a very clear way. - -A control-flow graph is structured as a set of **basic blocks** -connected by edges. The key idea of a basic block is that it is a set -of statements that execute "together" -- that is, whenever you branch -to a basic block, you start at the first statement and then execute -all the remainder. Only at the end of the is there the possibility of -branching to more than one place (in MIR, we call that final statement -the **terminator**): - -``` -bb0: { - statement0; - statement1; - statement2; - ... - terminator; -} -``` - -Many expressions that you are used to in Rust compile down to multiple -basic blocks. For example, consider an if statement: - -```rust -a = 1; -if some_variable { - b = 1; -} else { - c = 1; -} -d = 1; -``` - -This would compile into four basic blocks: - -``` -BB0: { - a = 1; - if some_variable { goto BB1 } else { goto BB2 } -} - -BB1: { - b = 1; - goto BB3; -} - -BB2: { - c = 1; - goto BB3; -} - -BB3: { - d = 1; - ...; -} -``` - -When using a control-flow graph, a loop simply appears as a cycle in -the graph, and the `break` keyword translates into a path out of that -cycle. - - - -## What is a dataflow analysis? - -*to be written* - - - -## What is "universally quantified"? What about "existentially quantified"? - -*to be written* - - - -## What is co- and contra-variance? - -*to be written* - - - -## What is a "free region" or a "free variable"? What about "bound region"? - -Let's describe the concepts of free vs bound in terms of program -variables, since that's the thing we're most familiar with. - -- Consider this expression: `a + b`. In this expression, `a` and `b` - refer to local variables that are defined *outside* of the - expression. We say that those variables **appear free** in the - expression. To see why this term makes sense, consider the next - example. -- In contrast, consider this expression, which creates a closure: `|a, - b| a + b`. Here, the `a` and `b` in `a + b` refer to the arguments - that the closure will be given when it is called. We say that the - `a` and `b` there are **bound** to the closure, and that the closure - signature `|a, b|` is a **binder** for the names `a` and `b` - (because any references to `a` or `b` within refer to the variables - that it introduces). - -So there you have it: a variable "appears free" in some -expression/statement/whatever if it refers to something defined -outside of that expressions/statement/whatever. Equivalently, we can -then refer to the "free variables" of an expression -- which is just -the set of variables that "appear free". - -So what does this have to do with regions? Well, we can apply the -analogous concept to type and regions. For example, in the type `&'a -u32`, `'a` appears free. But in the type `for<'a> fn(&'a u32)`, it -does not. diff --git a/src/mir-borrowck.md b/src/mir-borrowck.md index b632addc2..3c10191d4 100644 --- a/src/mir-borrowck.md +++ b/src/mir-borrowck.md @@ -37,9 +37,9 @@ in several modes, but this text will describe only the mode when NLL is enabled The overall flow of the borrow checker is as follows: -- We first create a **local copy** C of the MIR. We will be modifying - this copy in place to modify the types and things to include - references to the new regions that we are computing. +- We first create a **local copy** C of the MIR. In the coming steps, + we will modify this copy in place to modify the types and things to + include references to the new regions that we are computing. - We then invoke `nll::replace_regions_in_mir` to modify this copy C. Among other things, this function will replace all of the regions in the MIR with fresh [inference variables](glossary.html). @@ -51,6 +51,6 @@ The overall flow of the borrow checker is as follows: - (More details can be found in [the NLL section](./mir-regionck.html).) - Finally, the borrow checker itself runs, taking as input (a) the results of move analysis and (b) the regions computed by the region - checker. This allows is to figure out which loans are still in scope + checker. This allows us to figure out which loans are still in scope at any particular point. diff --git a/src/mir-passes.md b/src/mir-passes.md index 2fe471385..6d657ae70 100644 --- a/src/mir-passes.md +++ b/src/mir-passes.md @@ -4,9 +4,9 @@ If you would like to get the MIR for a function (or constant, etc), you can use the `optimized_mir(def_id)` query. This will give you back the final, optimized MIR. For foreign def-ids, we simply read the MIR from the other crate's metadata. But for local def-ids, the query will -construct the MIR and then iteratively optimize it by putting it -through various pipeline stages. This section describes those pipeline -stages and how you can extend them. +construct the MIR and then iteratively optimize it by applying a +series of passes. This section describes how those passes work and how +you can extend them. To produce the `optimized_mir(D)` for a given def-id `D`, the MIR passes through several suites of optimizations, each represented by a @@ -97,18 +97,19 @@ that appeared within the `main` function.) ### Implementing and registering a pass A `MirPass` is some bit of code that processes the MIR, typically -- -but not always -- transforming it along the way in some way. For -example, it might perform an optimization. The `MirPass` trait itself -is found in in [the `rustc_mir::transform` module][mirtransform], and -it basically consists of one method, `run_pass`, that simply gets an +but not always -- transforming it along the way somehow. For example, +it might perform an optimization. The `MirPass` trait itself is found +in in [the `rustc_mir::transform` module][mirtransform], and it +basically consists of one method, `run_pass`, that simply gets an `&mut Mir` (along with the tcx and some information about where it -came from). +came from). The MIR is therefore modified in place (which helps to +keep things efficient). -A good example of a basic MIR pass is [`NoLandingPads`], which walks the -MIR and removes all edges that are due to unwinding -- this is used -with when configured with `panic=abort`, which never unwinds. As you can see -from its source, a MIR pass is defined by first defining a dummy type, a struct -with no fields, something like: +A good example of a basic MIR pass is [`NoLandingPads`], which walks +the MIR and removes all edges that are due to unwinding -- this is +used when configured with `panic=abort`, which never unwinds. As you +can see from its source, a MIR pass is defined by first defining a +dummy type, a struct with no fields, something like: ```rust struct MyPass; @@ -120,8 +121,9 @@ this pass into the appropriate list of passes found in a query like should go into the `optimized_mir` list.) If you are writing a pass, there's a good chance that you are going to -want to use a [MIR visitor] too -- those are a handy visitor that -walks the MIR for you and lets you make small edits here and there. +want to use a [MIR visitor]. MIR visitors are a handy way to walk all +the parts of the MIR, either to search for something or to make small +edits. ### Stealing @@ -149,7 +151,9 @@ be **stolen** by the `mir_validated()` suite. If nothing was done, then `mir_const_qualif(D)` would succeed if it came before `mir_validated(D)`, but fail otherwise. Therefore, `mir_validated(D)` will **force** `mir_const_qualif` before it actually steals, thus -ensuring that the reads have already happened: +ensuring that the reads have already happened (remember that +[queries are memoized](./query.html), so executing a query twice +simply loads from a cache the second time): ``` mir_const(D) --read-by--> mir_const_qualif(D) diff --git a/src/mir-regionck.md b/src/mir-regionck.md index 8be4fc8b9..d9d854081 100644 --- a/src/mir-regionck.md +++ b/src/mir-regionck.md @@ -10,11 +10,11 @@ deprecated once they become the standard kind of lifetime.) The MIR-based region analysis consists of two major functions: - `replace_regions_in_mir`, invoked first, has two jobs: - - First, it analyzes the signature of the MIR and finds the set of - regions that appear in the MIR signature (e.g., `'a` in `fn - foo<'a>(&'a u32) { ... }`. These are called the "universal" or - "free" regions -- in particular, they are the regions that - [appear free][fvb] in the function body. + - First, it finds the set of regions that appear within the + signature of the function (e.g., `'a` in `fn foo<'a>(&'a u32) { + ... }`. These are called the "universal" or "free" regions -- in + particular, they are the regions that [appear free][fvb] in the + function body. - Second, it replaces all the regions from the function body with fresh inference variables. This is because (presently) those regions are the results of lexical region inference and hence are @@ -49,6 +49,8 @@ the role of `liveness_constraints` vs other `constraints`, plus ## Closures +*to be written* + ## The MIR type-check @@ -131,15 +133,14 @@ replace them with representatives, written like `!1`. We call these regions "skolemized regions" -- they represent, basically, "some unknown region". -Once we've done that replacement, we have the following types: +Once we've done that replacement, we have the following relation: fn(&'static u32) <: fn(&'!1 u32) The key idea here is that this unknown region `'!1` is not related to any other regions. So if we can prove that the subtyping relationship is true for `'!1`, then it ought to be true for any region, which is -what we wanted. (This number `!1` is called a "universe", for reasons -we'll get into later.) +what we wanted. So let's work through what happens next. To check if two functions are subtypes, we check if their arguments have the desired relationship @@ -154,6 +155,118 @@ outlives `'static`. Now, this *might* be true -- after all, `'!1` could be `'static` -- but we don't *know* that it's true. So this should yield up an error (eventually). +### What is a universe + +In the previous section, we introduced the idea of a skolemized +region, and we denoted it `!1`. We call this number `1` the **universe +index**. The idea of a "universe" is that it is a set of names that +are in scope within some type or at some point. Universes are formed +into a tree, where each child extends its parents with some new names. +So the **root universe** conceptually contains global names, such as +the the lifetime `'static` or the type `i32`. In the compiler, we also +put generic type parameters into this root universe. So consider +this function `bar`: + +```rust +struct Foo { } + +fn bar<'a, T>(t: &'a T) { + ... +} +``` + +Here, the root universe would consider of the lifetimes `'static` and +`'a`. In fact, although we're focused on lifetimes here, we can apply +the same concept to types, in which case the types `Foo` and `T` would +be in the root universe (along with other global types, like `i32`). +Basically, the root universe contains all the names that +[appear free](./background.html#free-vs-bound) in the body of `bar`. + +Now let's extend `bar` a bit by adding a variable `x`: + +```rust +fn bar<'a, T>(t: &'a T) { + let x: for<'b> fn(&'b u32) = ...; +} +``` + +Here, the name `'b` is not part of the root universe. Instead, when we +"enter" into this `for<'b>` (e.g., by skolemizing it), we will create +a child universe of the root, let's call it U1: + +``` +U0 (root universe) +│ +└─ U1 (child universe) +``` + +The idea is that this child universe U1 extends the root universe U0 +with a new name, which we are identifying by its universe number: +`!1`. + +Now let's extend `bar` a bit by adding one more variable, `y`: + +```rust +fn bar<'a, T>(t: &'a T) { + let x: for<'b> fn(&'b u32) = ...; + let y: for<'c> fn(&'b u32) = ...; +} +``` + +When we enter *this* type, we will again create a new universe, which +let's call `U2`. It's parent will be the root universe, and U1 will be +its sibling: + +``` +U0 (root universe) +│ +├─ U1 (child universe) +│ +└─ U2 (child universe) +``` + +This implies that, while in U2, we can name things from U0 or U2, but +not U1. + +**Giving existential variables a universe.** Now that we have this +notion of universes, we can use it to extend our type-checker and +things to prevent illegal names from leaking out. The idea is that we +give each inference (existential) variable -- whether it be a type or +a lifetime -- a universe. That variable's value can then only +reference names visible from that universe. So for example is a +lifetime variable is created in U0, then it cannot be assigned a value +of `!1` or `!2`, because those names are not visible from the universe +U0. + +**Representing universes with just a counter.** You might be surprised +to see that the compiler doesn't keep track of a full tree of +universes. Instead, it just keeps a counter -- and, to determine if +one universe can see another one, it just checks if the index is +greater. For example, U2 can see U0 because 2 >= 0. But U0 cannot see +U2, because 0 >= 2 is false. + +How can we get away with this? Doesn't this mean that we would allow +U2 to also see U1? The answer is that, yes, we would, **if that +question ever arose**. But because of the structure of our type +checker etc, there is no way for that to happen. In order for +something happening in the universe U1 to "communicate" with something +happening in U2, they would have to have a shared inference variable X +in common. And because everything in U1 is scoped to just U1 and its +children, that inference variable X would have to be in U0. And since +X is in U0, it cannot name anything from U1 (or U2). This is perhaps easiest +to see by using a kind of generic "logic" example: + +``` +exists { + forall { ... /* Y is in U1 ... */ } + forall { ... /* Z is in U2 ... */ } +} +``` + +Here, the only way for the two foralls to interact would be through X, +but neither Y nor Z are in scope when X is declared, so its value +cannot reference either of them. + ### Universes and skolemized region elements But where does that error come from? The way it happens is like this. @@ -179,10 +292,11 @@ In the region inference engine, outlives constraints have the form: V1: V2 @ P where `V1` and `V2` are region indices, and hence map to some region -variable (which may be universally or existentially quantified). This -variable will have a universe, so let's call those universes `U(V1)` -and `U(V2)` respectively. (Actually, the only one we are going to care -about is `U(V1)`.) +variable (which may be universally or existentially quantified). The +`P` here is a "point" in the control-flow graph; it's not important +for this section. This variable will have a universe, so let's call +those universes `U(V1)` and `U(V2)` respectively. (Actually, the only +one we are going to care about is `U(V1)`.) When we encounter this constraint, the ordinary procedure is to start a DFS from `P`. We keep walking so long as the nodes we are walking @@ -190,24 +304,24 @@ are present in `value(V2)` and we add those nodes to `value(V1)`. If we reach a return point, we add in any `end(X)` elements. That part remains unchanged. -But then *after that* we want to iterate over the skolemized `skol(u)` +But then *after that* we want to iterate over the skolemized `skol(x)` elements in V2 (each of those must be visible to `U(V2)`, but we should be able to just assume that is true, we don't have to check it). We have to ensure that `value(V1)` outlives each of those skolemized elements. Now there are two ways that could happen. First, if `U(V1)` can see -the universe `u` (i.e., `u <= U(V1)`), then we can just add `skol(u1)` +the universe `x` (i.e., `x <= U(V1)`), then we can just add `skol(x)` to `value(V1)` and be done. But if not, then we have to approximate: -we may not know what set of elements `skol(u1)` represents, but we -should be able to compute some sort of **upper bound** for it -- -something that it is smaller than. For now, we'll just use `'static` -for that (since it is bigger than everything) -- in the future, we can -sometimes be smarter here (and in fact we have code for doing this -already in other contexts). Moreover, since `'static` is in U0, we -know that all variables can see it -- so basically if we find a that -`value(V2)` contains `skol(u)` for some universe `u` that `V1` can't -see, then we force `V1` to `'static`. +we may not know what set of elements `skol(x)` represents, but we +should be able to compute some sort of **upper bound** B for it -- +some region B that outlives `skol(x)`. For now, we'll just use +`'static` for that (since it outlives everything) -- in the future, we +can sometimes be smarter here (and in fact we have code for doing this +already in other contexts). Moreover, since `'static` is in the root +universe U0, we know that all variables can see it -- so basically if +we find that `value(V2)` contains `skol(x)` for some universe `x` +that `V1` can't see, then we force `V1` to `'static`. ### Extending the "universal regions" check @@ -258,7 +372,7 @@ To process this, we would grow the value of V1 to include all of Vs: Vs = { CFG; end('static) } V1 = { CFG; end('static), skol(1) } -At that point, constraint propagation is done, because all the +At that point, constraint propagation is complete, because all the outlives relationships are satisfied. Then we would go to the "check universal regions" portion of the code, which would test that no universal region grew too large. @@ -280,8 +394,9 @@ Here we would skolemize the supertype, as before, yielding: <: fn(&'!1 u32, &'!2 u32) -then we instantiate the variable on the left-hand side with an existential -in universe U2, yielding: +then we instantiate the variable on the left-hand side with an +existential in universe U2, yielding the following (`?n` is a notation +for an existential variable): fn(&'?3 u32, &'?3 u32) <: diff --git a/src/mir-visitor.md b/src/mir-visitor.md index 824ddd5b2..3a8b06c54 100644 --- a/src/mir-visitor.md +++ b/src/mir-visitor.md @@ -43,3 +43,13 @@ terminators and removes their `unwind` successors. [`NoLandingPads`]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir/transform/no_landing_pads.rs +## Traversal + +In addition the visitor, [the `rustc::mir::traversal` module][t] +contains useful functions for walking the MIR CFG in +[different standard orders][traversal] (e.g. pre-order, reverse +post-order, and so forth). + +[t]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir/traversal.rs +[traversal]: https://en.wikipedia.org/wiki/Tree_traversal + diff --git a/src/mir.md b/src/mir.md index 262082ed7..6e7ac0691 100644 --- a/src/mir.md +++ b/src/mir.md @@ -1,10 +1,10 @@ # The MIR (Mid-level IR) MIR is Rust's _Mid-level Intermediate Representation_. It is -constructed from HIR (described in an earlier chapter). MIR was -introduced in [RFC 1211]. It is a radically simplified form of Rust -that is used for certain flow-sensitive safety checks -- notably the -borrow checker! -- and also for optimization and code generation. +constructed from [HIR](./hir.html). MIR was introduced in +[RFC 1211]. It is a radically simplified form of Rust that is used for +certain flow-sensitive safety checks -- notably the borrow checker! -- +and also for optimization and code generation. If you'd like a very high-level introduction to MIR, as well as some of the compiler concepts that it relies on (such as control-flow @@ -35,20 +35,20 @@ This section introduces the key concepts of MIR, summarized here: - **Basic blocks**: units of the control-flow graph, consisting of: - **statements:** actions with one successor - **terminators:** actions with potentially multiple successors; always at the end of a block - - (if you're not familiar with the term basic block, see the [MIR background chapter][bg]) + - (if you're not familiar with the term *basic block*, see the [background chapter][cfg]) - **Locals:** Memory locations alloated on the stack (conceptually, at least), such as function arguments, local variables, and temporaries. These are identified by an index, written with a leading underscore, like `_1`. There is also a special "local" (`_0`) allocated to store the return value. - **Places:** expressions that identify a location in memory, like `_1` or `_1.f`. -- **Rvalues:** expressions that product a value. The "R" stands for +- **Rvalues:** expressions that produce a value. The "R" stands for the fact that these are the "right-hand side" of an assignment. - **Operands:** the arguments to an rvalue, which can either be a constant (like `22`) or a place (like `_1`). You can get a feeling for how MIR is structed by translating simple -programs into MIR and ready the pretty printed output. In fact, the +programs into MIR and reading the pretty printed output. In fact, the playground makes this easy, since it supplies a MIR button that will show you the MIR for your program. Try putting this program into play (or [clicking on this link][sample-play]), and then clicking the "MIR" @@ -96,7 +96,9 @@ You can see that variables in MIR don't have names, they have indices, like `_0` or `_1`. We also intermingle the user's variables (e.g., `_1`) with temporary values (e.g., `_2` or `_3`). You can tell the difference between user-defined variables have a comment that gives -you their original name (`// "vec" in scope 1...`). +you their original name (`// "vec" in scope 1...`). The "scope" blocks +(e.g., `scope 1 { .. }`) describe the lexical structure of the source +program (which names were in scope when). **Basic blocks.** Reading further, we see our first **basic block** (naturally it may look slightly different when you view it, and I am ignoring some of the comments): @@ -223,27 +225,15 @@ but [you can read about those below](#promoted)). - **Rvalues** are represented by the enum `Rvalue`. - **Operands** are represented by the enum `Operand`. -## MIR Visitor - -The main MIR data type is `rustc::mir::Mir`, defined in `mod.rs`. -There is also the MIR visitor (in `visit.rs`) which allows you to walk -the MIR and override what actions will be taken at various points (you -can visit in either shared or mutable mode; the latter allows changing -the MIR in place). Finally `traverse.rs` contains various traversal -routines for visiting the MIR CFG in [different standard orders][traversal] -(e.g. pre-order, reverse post-order, and so forth). - -[traversal]: https://en.wikipedia.org/wiki/Tree_traversal - ## Representing constants -TBD +*to be written* ### Promoted constants -TBD +*to be written* [mir]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir From 64963f85af1723d9632ef69245e030c2487e7511 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 27 Feb 2018 14:16:18 -0500 Subject: [PATCH 114/648] fix typo --- src/mir-regionck.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/mir-regionck.md b/src/mir-regionck.md index d9d854081..571d7c673 100644 --- a/src/mir-regionck.md +++ b/src/mir-regionck.md @@ -20,7 +20,7 @@ The MIR-based region analysis consists of two major functions: regions are the results of lexical region inference and hence are not of much interest. The intention is that -- eventually -- they will be "erased regions" (i.e., no information at all), since we - don't be doing lexical region inference at all. + won't be doing lexical region inference at all. - `compute_regions`, invoked second: this is given as argument the results of move analysis. It has the job of computing values for all the inference variabes that `replace_regions_in_mir` introduced. From 03044a350e5cd5f4a1fc7dcc454fd9c4ecaade33 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 28 Feb 2018 15:28:57 -0500 Subject: [PATCH 115/648] address nits --- src/glossary.md | 2 +- src/mir-regionck.md | 9 +++++---- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/src/glossary.md b/src/glossary.md index 81eb62bc9..bff2a6557 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -43,7 +43,7 @@ region | another term for "lifetime" often used in the literat sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. sigil | like a keyword but composed entirely of non-alphanumeric tokens. For example, `&` is a sigil for references. -skolemization | a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)` as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on skolemization and universes](./mir-regionck.html#skol) for more details. +skolemization | a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)`) as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on skolemization and universes](./mir-regionck.html#skol) for more details. soundness | soundness is a technical term in type theory. Roughly, if a type system is sound, then if a program type-checks, it is type-safe; i.e. I can never (in safe rust) force a value into a variable of the wrong type. (see "completeness"). span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. substs | the substitutions for a given generic type or item (e.g. the `i32`, `u32` in `HashMap`) diff --git a/src/mir-regionck.md b/src/mir-regionck.md index 571d7c673..e7b12405a 100644 --- a/src/mir-regionck.md +++ b/src/mir-regionck.md @@ -12,7 +12,7 @@ The MIR-based region analysis consists of two major functions: - `replace_regions_in_mir`, invoked first, has two jobs: - First, it finds the set of regions that appear within the signature of the function (e.g., `'a` in `fn foo<'a>(&'a u32) { - ... }`. These are called the "universal" or "free" regions -- in + ... }`). These are called the "universal" or "free" regions -- in particular, they are the regions that [appear free][fvb] in the function body. - Second, it replaces all the regions from the function body with @@ -164,7 +164,8 @@ are in scope within some type or at some point. Universes are formed into a tree, where each child extends its parents with some new names. So the **root universe** conceptually contains global names, such as the the lifetime `'static` or the type `i32`. In the compiler, we also -put generic type parameters into this root universe. So consider +put generic type parameters into this root universe (in this sense, +there is not just one root universe, but one per item). So consider this function `bar`: ```rust @@ -175,7 +176,7 @@ fn bar<'a, T>(t: &'a T) { } ``` -Here, the root universe would consider of the lifetimes `'static` and +Here, the root universe would consist of the lifetimes `'static` and `'a`. In fact, although we're focused on lifetimes here, we can apply the same concept to types, in which case the types `Foo` and `T` would be in the root universe (along with other global types, like `i32`). @@ -214,7 +215,7 @@ fn bar<'a, T>(t: &'a T) { ``` When we enter *this* type, we will again create a new universe, which -let's call `U2`. It's parent will be the root universe, and U1 will be +we'll call `U2`. Its parent will be the root universe, and U1 will be its sibling: ``` From 3025c38976a88b5606e24a0ead66ff78d21658c6 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Wed, 28 Feb 2018 14:40:18 -0600 Subject: [PATCH 116/648] Fix funny glossary formatting --- src/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/glossary.md b/src/glossary.md index bff2a6557..187460f03 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -6,7 +6,7 @@ The compiler uses a number of...idiosyncratic abbreviations and things. This glo Term | Meaning ------------------------|-------- AST | the abstract syntax tree produced by the syntax crate; reflects user syntax very closely. -binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and `|a| ...` is a binder for the parameter `a`. See [the background chapter for more](./background.html#free-vs-bound) +binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and `\|a\| ...` is a binder for the parameter `a`. See [the background chapter for more](./background.html#free-vs-bound) bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession `|a| a * 2`. See [the background chapter for more](./background.html#free-vs-bound) codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. completeness | completeness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). From a797e14afaf6db2b23ee8a13fecaf61279160b9a Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 28 Feb 2018 14:45:51 -0600 Subject: [PATCH 117/648] fix table --- src/glossary.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/glossary.md b/src/glossary.md index 187460f03..9c4b66d66 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -6,8 +6,8 @@ The compiler uses a number of...idiosyncratic abbreviations and things. This glo Term | Meaning ------------------------|-------- AST | the abstract syntax tree produced by the syntax crate; reflects user syntax very closely. -binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and `\|a\| ...` is a binder for the parameter `a`. See [the background chapter for more](./background.html#free-vs-bound) -bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession `|a| a * 2`. See [the background chapter for more](./background.html#free-vs-bound) +binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and `|a| ...` is a binder for the parameter `a`. See [the background chapter for more](./background.html#free-vs-bound) +bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession `|a| a * 2`. See [the background chapter for more](./background.html#free-vs-bound) codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. completeness | completeness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). control-flow graph | a representation of the control-flow of a program; see [the background chapter for more](./background.html#cfg) From 017796049dbaa091909406646eed568ae4a50645 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 28 Feb 2018 14:58:24 -0600 Subject: [PATCH 118/648] actually fix table with hack --- src/glossary.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/glossary.md b/src/glossary.md index 9c4b66d66..7152cba36 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -6,8 +6,8 @@ The compiler uses a number of...idiosyncratic abbreviations and things. This glo Term | Meaning ------------------------|-------- AST | the abstract syntax tree produced by the syntax crate; reflects user syntax very closely. -binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and `|a| ...` is a binder for the parameter `a`. See [the background chapter for more](./background.html#free-vs-bound) -bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession `|a| a * 2`. See [the background chapter for more](./background.html#free-vs-bound) +binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and \|`a`\|` ...` is a binder for the parameter `a`. See [the background chapter for more](./background.html#free-vs-bound) +bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession \|`a`\|` a * 2`. See [the background chapter for more](./background.html#free-vs-bound) codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. completeness | completeness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). control-flow graph | a representation of the control-flow of a program; see [the background chapter for more](./background.html#cfg) From 12e382d2d5d8e3f3fe2c845e3134084126204438 Mon Sep 17 00:00:00 2001 From: vishalsodani Date: Thu, 1 Mar 2018 10:04:23 +0530 Subject: [PATCH 119/648] Add info about ParseSess for code index #50 --- src/code-index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/code-index.md b/src/code-index.md index 5b452ef0f..73e23d824 100644 --- a/src/code-index.md +++ b/src/code-index.md @@ -7,3 +7,4 @@ compiler. Item | Kind | Short description | Chapter | Declaration ----------------|----------|-----------------------------|--------------------|------------------- `TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules](ty.html) | [src/librustc/ty/context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs) +`ParseSess` | struct | This struct contains information about a parsing session | [The parser](the-parser.html) | [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs) From 7fa92ec4a528833984dce289ef4bcbe887b433c4 Mon Sep 17 00:00:00 2001 From: vishalsodani Date: Fri, 2 Mar 2018 09:22:41 +0530 Subject: [PATCH 120/648] Add info about CodeMap for #50 --- src/code-index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/code-index.md b/src/code-index.md index 73e23d824..bbfc233e8 100644 --- a/src/code-index.md +++ b/src/code-index.md @@ -8,3 +8,4 @@ Item | Kind | Short description | Chapter | ----------------|----------|-----------------------------|--------------------|------------------- `TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules](ty.html) | [src/librustc/ty/context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs) `ParseSess` | struct | This struct contains information about a parsing session | [The parser](the-parser.html) | [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs) +`CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser](the-parser.html) | [src/libsyntax/codemap.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs) \ No newline at end of file From f1bda065d8ad9cbf0d328ffed1a48acaf48718dc Mon Sep 17 00:00:00 2001 From: vishalsodani Date: Fri, 2 Mar 2018 11:03:20 +0530 Subject: [PATCH 121/648] Add info about TraitDef for #50 --- src/code-index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/code-index.md b/src/code-index.md index bbfc233e8..07aad732c 100644 --- a/src/code-index.md +++ b/src/code-index.md @@ -8,4 +8,5 @@ Item | Kind | Short description | Chapter | ----------------|----------|-----------------------------|--------------------|------------------- `TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules](ty.html) | [src/librustc/ty/context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs) `ParseSess` | struct | This struct contains information about a parsing session | [The parser](the-parser.html) | [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs) -`CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser](the-parser.html) | [src/libsyntax/codemap.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs) \ No newline at end of file +`CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser](the-parser.html) | [src/libsyntax/codemap.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs) +`TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules](ty.html) | [src/librustc/ty/trait_def.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/trait_def.rs) From fd0223e50b6feaf1fa821c915adef6e651951c43 Mon Sep 17 00:00:00 2001 From: vishalsodani Date: Fri, 2 Mar 2018 19:42:26 +0530 Subject: [PATCH 122/648] keep alphabetical order --- src/code-index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/code-index.md b/src/code-index.md index 07aad732c..b1eb3546a 100644 --- a/src/code-index.md +++ b/src/code-index.md @@ -6,7 +6,7 @@ compiler. Item | Kind | Short description | Chapter | Declaration ----------------|----------|-----------------------------|--------------------|------------------- -`TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules](ty.html) | [src/librustc/ty/context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs) -`ParseSess` | struct | This struct contains information about a parsing session | [The parser](the-parser.html) | [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs) `CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser](the-parser.html) | [src/libsyntax/codemap.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs) +`ParseSess` | struct | This struct contains information about a parsing session | [The parser](the-parser.html) | [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs) `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules](ty.html) | [src/librustc/ty/trait_def.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/trait_def.rs) +`TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules](ty.html) | [src/librustc/ty/context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs) From 23b3e84c1136eaa010b62cdee584edb413690c87 Mon Sep 17 00:00:00 2001 From: Michal 'vorner' Vaner Date: Sun, 18 Feb 2018 18:03:46 +0100 Subject: [PATCH 123/648] Second pass of name resolution Just small little tweaks --- src/name-resolution.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/name-resolution.md b/src/name-resolution.md index 25ce3e29e..5095b750a 100644 --- a/src/name-resolution.md +++ b/src/name-resolution.md @@ -3,7 +3,12 @@ The name resolution is a separate pass in the compiler. Its input is the syntax tree, produced by parsing input files. It produces links from all the names in the source to relevant places where the name was introduced. It also generates -helpful error messages, like typo suggestions or traits to import. +helpful error messages, like typo suggestions, traits to import or lints about +unused items. + +A successful run of the name resolution (`Resolver::resolve_crate`) creates kind +of an index the rest of the compilation may use to ask about the present names +(through the `hir::lowering::Resolver` interface). The name resolution lives in the `librustc_resolve` crate, with the meat in `lib.rs` and some helpers or symbol-type specific logic in the other modules. @@ -72,6 +77,8 @@ fn do_something(val: T) { // <- New rib in both types and values (1) Because the rules for different namespaces are a bit different, each namespace has its own independent rib stack that is constructed in parallel to the others. +In addition, there's also a rib stack for local labels (eg. names of loops or +blocks), which isn't a full namespace in its own right. ## Overall strategy From 8a005f5dac2743212ec045e2116a95e67f78ea7c Mon Sep 17 00:00:00 2001 From: Phlosioneer Date: Tue, 6 Mar 2018 20:01:09 -0500 Subject: [PATCH 124/648] Fix typo in conventions.md --- src/conventions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/conventions.md b/src/conventions.md index f7ef86f11..db531c58c 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -132,7 +132,7 @@ we do not currently run rustfmt all the time, that can introduce a lot of noise into your commit. Please isolate that into its own commit. This also makes rebases a lot less painful, since rustfmt tends to cause a lot of merge conflicts, and having those isolated -into their own commit makes htem easier to resolve. +into their own commit makes them easier to resolve. **No merges.** We do not allow merge commits into our history, other than those by bors. If you get a merge conflict, rebase instead via a From 6660e4466a0263aadf04f754b1ec85d49ccd2825 Mon Sep 17 00:00:00 2001 From: "Adam C. Foltzer" Date: Mon, 5 Mar 2018 12:18:33 -0800 Subject: [PATCH 125/648] Fix typos --- src/incremental-compilation.md | 2 +- src/query.md | 2 +- src/trait-resolution.md | 4 ++-- src/type-inference.md | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src/incremental-compilation.md b/src/incremental-compilation.md index dc4d06c6c..6f5b48cca 100644 --- a/src/incremental-compilation.md +++ b/src/incremental-compilation.md @@ -114,7 +114,7 @@ This can lead to ICEs and other problems in the compiler. ## Improvements to the basic algorithm -In the description basic algorithm, we said that at the end of +In the description of the basic algorithm, we said that at the end of compilation we would save the results of all the queries that were performed. In practice, this can be quite wasteful – many of those results are very cheap to recompute, and serializing and deserializing diff --git a/src/query.md b/src/query.md index 481b4fed3..33d2b3bac 100644 --- a/src/query.md +++ b/src/query.md @@ -141,7 +141,7 @@ They return the result of the query. #### How providers are setup When the tcx is created, it is given the providers by its creator using -the `Providers` struct. This struct is generate by the macros here, but it +the `Providers` struct. This struct is generated by the macros here, but it is basically a big list of function pointers: ```rust diff --git a/src/trait-resolution.md b/src/trait-resolution.md index 1cad488ac..28176d740 100644 --- a/src/trait-resolution.md +++ b/src/trait-resolution.md @@ -263,13 +263,13 @@ Confirmation is where an error would be reported because the impl specified that `Target` would be `usize`, but the obligation reported `char`. Hence the result of selection would be an error. -Note that the candidate impl is chosen base on the `Self` type, but +Note that the candidate impl is chosen based on the `Self` type, but confirmation is done based on (in this case) the `Target` type parameter. ### Selection during translation As mentioned above, during type checking, we do not store the results of trait -selection. At trans time, repeat the trait selection to choose a particular +selection. At trans time, we repeat the trait selection to choose a particular impl for each method call. In this second selection, we do not consider any where-clauses to be in scope because we know that each resolution will resolve to a particular impl. diff --git a/src/type-inference.md b/src/type-inference.md index c4307ce7f..8812d34d7 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -39,7 +39,7 @@ for some fresh `'cx` and `'tcx` – the latter corresponds to the lifetime of this temporary arena, and the `'cx` is the lifetime of the `InferCtxt` itself. (Again, see the [`ty` chapter][ty-ch] for more details on this setup.) -The `tcx.infer_ctxt` method actually returns a build, which means +The `tcx.infer_ctxt` method actually returns a builder, which means there are some kinds of configuration you can do before the `infcx` is created. See `InferCtxtBuilder` for more information. From f04757a901faa72f6ce3711e87ce3c323ec91885 Mon Sep 17 00:00:00 2001 From: vishalsodani Date: Sat, 3 Mar 2018 09:58:45 +0530 Subject: [PATCH 126/648] Add info about StringReader for #50 --- src/code-index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/code-index.md b/src/code-index.md index b1eb3546a..2395feaf6 100644 --- a/src/code-index.md +++ b/src/code-index.md @@ -8,5 +8,6 @@ Item | Kind | Short description | Chapter | ----------------|----------|-----------------------------|--------------------|------------------- `CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser](the-parser.html) | [src/libsyntax/codemap.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs) `ParseSess` | struct | This struct contains information about a parsing session | [The parser](the-parser.html) | [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs) +`StringReader` | struct | This struct helps in lexing source code into tokens | [The parser](the-parser.html) | [src/libsyntax/parse/lexer/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/lexer/mod.rs) `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules](ty.html) | [src/librustc/ty/trait_def.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/trait_def.rs) `TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules](ty.html) | [src/librustc/ty/context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs) From 86c5f6c9b4cc70fefa87b40c91b431e3a31b4949 Mon Sep 17 00:00:00 2001 From: Vishal Sodani Date: Fri, 9 Mar 2018 13:52:57 +0530 Subject: [PATCH 127/648] Better explanation for StringReader --- src/code-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/code-index.md b/src/code-index.md index 2395feaf6..6a500abba 100644 --- a/src/code-index.md +++ b/src/code-index.md @@ -8,6 +8,6 @@ Item | Kind | Short description | Chapter | ----------------|----------|-----------------------------|--------------------|------------------- `CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser](the-parser.html) | [src/libsyntax/codemap.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs) `ParseSess` | struct | This struct contains information about a parsing session | [The parser](the-parser.html) | [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs) -`StringReader` | struct | This struct helps in lexing source code into tokens | [The parser](the-parser.html) | [src/libsyntax/parse/lexer/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/lexer/mod.rs) +`StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser](the-parser.html) | [src/libsyntax/parse/lexer/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/lexer/mod.rs) `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules](ty.html) | [src/librustc/ty/trait_def.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/trait_def.rs) `TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules](ty.html) | [src/librustc/ty/context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs) From 479d914ff009a0372427b42f3796f5937297644f Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 27 Feb 2018 05:50:43 -0500 Subject: [PATCH 128/648] work on traits chapters --- src/SUMMARY.md | 13 +- src/trait-resolution.md | 7 +- src/traits-associated-types.md | 143 +++++++++++++++++ src/traits-bibliography.md | 28 ++++ src/traits-canonicalization.md | 93 +++++++++++ src/traits-goals-and-clauses.md | 148 ++++++++++++++++++ src/traits-lowering-rules.md | 267 ++++++++++++++++++++++++++++++++ src/traits-lowering-to-logic.md | 182 ++++++++++++++++++++++ src/traits-regions.md | 3 + src/traits-wf.md | 11 ++ src/traits.md | 18 +++ src/type-inference.md | 2 + 12 files changed, 912 insertions(+), 3 deletions(-) create mode 100644 src/traits-associated-types.md create mode 100644 src/traits-bibliography.md create mode 100644 src/traits-canonicalization.md create mode 100644 src/traits-goals-and-clauses.md create mode 100644 src/traits-lowering-rules.md create mode 100644 src/traits-lowering-to-logic.md create mode 100644 src/traits-regions.md create mode 100644 src/traits-wf.md create mode 100644 src/traits.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index ab9da7295..1c5bc284d 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -17,10 +17,19 @@ - [The HIR (High-level IR)](./hir.md) - [The `ty` module: representing types](./ty.md) - [Type inference](./type-inference.md) -- [Trait resolution](./trait-resolution.md) +- [Trait solving (old-style)](./trait-resolution.md) - [Higher-ranked trait bounds](./trait-hrtb.md) - [Caching subtleties](./trait-caching.md) - - [Speciailization](./trait-specialization.md) + - [Specialization](./trait-specialization.md) +- [Trait solving (new-style)](./traits.md) + - [Canonicalization](./traits-canonicalization.md) + - [Lowering to logic](./traits-lowering-to-logic.md) + - [Goals and clauses](./traits-goals-and-clauses.md) + - [Lowering rules](./traits-lowering-rules.md) + - [Equality and associated types](./traits-associated-types.md) + - [Region constraints](./traits-regions.md) + - [Well-formedness checking](./traits-wf.md) + - [Bibliography](./traits-bibliography.md) - [Type checking](./type-checking.md) - [The MIR (Mid-level IR)](./mir.md) - [MIR construction](./mir-construction.md) diff --git a/src/trait-resolution.md b/src/trait-resolution.md index 28176d740..cccfcd54b 100644 --- a/src/trait-resolution.md +++ b/src/trait-resolution.md @@ -1,8 +1,13 @@ -# Trait resolution +# Trait resolution (old-style) This chapter describes the general process of _trait resolution_ and points out some non-obvious things. +**Note:** This chapter (and its subchapters) describe how the trait +solver **currently** works. However, we are in the process of +designing a new trait solver. If you'd prefer to read about *that*, +see [the traits chapter](./traits.html). + ## Major concepts Trait resolution is the process of pairing up an impl with each diff --git a/src/traits-associated-types.md b/src/traits-associated-types.md new file mode 100644 index 000000000..a640bf032 --- /dev/null +++ b/src/traits-associated-types.md @@ -0,0 +1,143 @@ +# Equality and associated types + +This section covers how the trait system handles equality between +associated types. The full system consists of several moving parts, +which we will introduce one by one: + +- Projection and the `Normalize` predicate +- Skolemization +- The `ProjectionEq` predicate +- Integration with unification + +## Associated type projection and normalization + +When a trait defines an associated type (e.g., +[the `Item` type in the `IntoIterator` trait][intoiter-item]), that +type can be referenced by the user using an **associated type +projection** like ` as IntoIterator>::Item`. (Often, +though, people will use the shorthand syntax `T::Item` -- presently, +that syntax is expanded during +["type collection"](./type-checking.html) into the explicit form, +though that is something we may want to change in the future.) + +In some cases, associated type projections can be **normalized** -- +that is, simplified -- based on the types given in an impl. So, to +continue with our example, the impl of `IntoIterator` for `Option` +declares (among other things) that `Item = T`: + +```rust +impl IntoIterator for Option { + type Item = T; + .. +} +``` + +This means we can normalize the projection ` as +IntoIterator>::Item` to just `u32`. + +In this case, the projection was a "monomorphic" one -- that is, it +did not have any type parameters. Monomorphic projections are special +because they can **always** be fully normalized -- but often we can +normalize other associated type projections as well. For example, +` as IntoIterator>::Item` (where `?T` is an inference +variable) can be normalized to just `?T`. + +In our logic, normalization is defined by a predicate +`Normalize`. The `Normalize` clauses arise only from +impls. For example, the `impl` of `IntoIterator` for `Option` that +we saw above would be lowered to a program clause like so: + + forall { + Normalize( as IntoIterator>::Item -> T) + } + +(An aside: since we do not permit quantification over traits, this is +really more like a family of predicates, one for each associated +type.) + +We could apply that rule to normalize either of the examples that +we've seen so far. + +## Skolemized associated types + +Sometimes however we want to work with associated types that cannot be +normalized. For example, consider this function: + +```rust +fn foo(...) { ... } +``` + +In this context, how would we normalize the type `T::Item`? Without +knowing what `T` is, we can't really do so. To represent this case, we +introduce a type called a **skolemized associated type +projection**. This is written like so `(IntoIterator::Item)`. You +may note that it looks a lot like a regular type (e.g., `Option`), +except that the "name" of the type is `(IntoIterator::Item)`. This is +not an accident: skolemized associated type projections work just like +ordinary types like `Vec` when it comes to unification. That is, +they are only considered equal if (a) they are both references to the +same associated type, like `IntoIterator::Item` and (b) their type +arguments are equal. + +Skolemized associated types are never written directly by the user. +They are used internally by the trait system only, as we will see +shortly. + +## Projection equality + +So far we have seen two ways to answer the question of "When can we +consider an associated type projection equal to another type?": + +- the `Normalize` predicate could be used to transform associated type + projections when we knew which impl was applicable; +- **skolemized** associated types can be used when we don't. + +We now introduce the `ProjectionEq` predicate to bring those two cases +together. The `ProjectionEq` predicate looks like so: + + ProjectionEq(::Item = U) + +and we will see that it can be proven *either* via normalization or +skolemization. As part of lowering an associated type declaration from +some trait, we create two program clauses for `ProjectionEq`: + + forall { + ProjectionEq(::Item = U) :- + Normalize(::Item -> U) + } + + forall { + ProjectionEq(::Item = (IntoIterator::Item)) + } + +These are the only two `ProjectionEq` program clauses we ever make for +any given associated item. + +## Integration with unification + +Now we are ready to discuss how associated type equality integrates +with unification. As described in the +[type inference](./type-inference.html) section, unification is +basically a procedure with a signature like this: + + Unify(A, B) = Result<(Subgoals, RegionConstraints), NoSolution> + +In other words, we try to unify two things A and B. That procedure +might just fail, in which case we get back `Err(NoSolution)`. This +would happen, for example, if we tried to unify `u32` and `i32`. + +The key point is that, on success, unification can also give back to +us a set of subgoals that still remain to be proven (it can also give +back region constraints, but those are not relevant here). + +Whenever unification encounters an (unskolemized!) associated type +projection P being equated with some other type T, it always succeeds, +but it produces a subgoal `ProjectionEq(P = T)` that is propagated +back up. Thus it falls to the ordinary workings of the trait system +to process that constraint. + +(If we unify two projections P1 and P2, then unification produces a +variable X and asks us to prove that `ProjectionEq(P1 = X)` and +`ProjectionEq(P2 = X)`. That used to be needed in an older system to +prevent cycles; I rather doubt it still is. -nmatsakis) + diff --git a/src/traits-bibliography.md b/src/traits-bibliography.md new file mode 100644 index 000000000..d9ff912c7 --- /dev/null +++ b/src/traits-bibliography.md @@ -0,0 +1,28 @@ +# Bibliography + +If you'd like to read more background material, here are some +recommended texts and papers: + +[Programming with Higher-order Logic][phl], by Dale Miller and Gopalan +Nadathur, covers the key concepts of Lambda prolog. Although it's a +slim little volume, it's the kind of book where you learn something +new every time you open it. + +[phl]: https://www.amazon.com/Programming-Higher-Order-Logic-Dale-Miller/dp/052187940X + + + +["A proof procedure for the logic of Hereditary Harrop formulas"][pphhf], +by Gopalan Nadathur. This paper covers the basics of universes, +environments, and Lambda Prolog-style proof search. Quite readable. + +[pphhf]: https://dl.acm.org/citation.cfm?id=868380 + + + +["A new formulation of tabled resolution with delay"][nftrd], by +[Theresa Swift]. This paper gives a kind of abstract treatment of the +SLG formulation that is the basis for our on-demand solver. + +[nftrd]: https://dl.acm.org/citation.cfm?id=651202 +[ts]: http://www3.cs.stonybrook.edu/~tswift/ diff --git a/src/traits-canonicalization.md b/src/traits-canonicalization.md new file mode 100644 index 000000000..35352d605 --- /dev/null +++ b/src/traits-canonicalization.md @@ -0,0 +1,93 @@ +# Canonicalization + +Canonicalization is the process of **isolating** an inference value +from its context. It is really based on a very simple concept: every +[inference variable](./type-inference.html#vars) is always in one of two +states: either it is **unbound**, in which case we don't know yet what +type it is, or it is **bound**, in which case we do. So to isolate +some thing T from its environment, we just walk down and find the +unbound variables that appear in T; those variables get renumbered in +a canonical order (left to right, for the most part, but really it +doesn't matter as long as it is consistent). + +So, for example, if we have the type `X = (?T, ?U)`, where `?T` and +`?U` are distinct, unbound inference variables, then the canonical +form of `X` would be `(?0, ?1)`, where `?0` and `?1` represent these +**canonical placeholders**. Note that the type `Y = (?U, ?T)` also +canonicalizes to `(?0, ?1)`. But the type `Z = (?T, ?T)` would +canonicalize to `(?0, ?0)` (as would `(?U, ?U)`). In other words, the +exact identity of the inference variables is not important -- unless +they are repeated. + +We use this to improve caching as well as to detect cycles and other +things during trait resolution. Roughly speaking, the idea is that if +two trait queries have the same canonicalize form, then they will get +the same answer -- modulo the precise identities of the variables +involved. + +To see how it works, imagine that we are asking to solve the following +trait query: `?A: Foo<'static, ?B>`, where `?A` and `?B` are unbound. +This query contains two unbound variables, but it also contains the +lifetime `'static`. The trait system generally ignores all lifetimes +and treats them equally, so when canonicalizing, we will *also* +replace any [free lifetime](./background.html#free-vs-bound) with a +canonical variable. Therefore, we get the following result: + + ?0: Foo<'?1, ?2> + +Sometimes we write this differently, like so: + + for { ?0: Foo<'?1, ?2> } + +This `for<>` gives some information about each of the canonical +variables within. In this case, I am saying that `?0` is a type +(`T`), `?1` is a lifetime (`L`), and `?2` is also a type (`T`). The +`canonicalize` method *also* gives back a `CanonicalVarValues` array +with the "original values" for each canonicalized variable: + + [?A, 'static, ?B] + +Now we do the query and get back some result R. As part of that +result, we'll have an array of values for the canonical inputs. For +example, the canonical result might be: + +``` +for<2> { + values = [ Vec, '1, ?0 ] + ^^ ^^ ^^ these are variables in the result! + ... +} +``` + +Note that this result is itself canonical and may include some +variables (in this case, `?0`). + +What we want to do conceptually is to (a) instantiate each of the +canonical variables in the result with a fresh inference variable +and then (b) unify the values in the result with the original values. +Doing step (a) would yield a result of + +``` +{ + values = [ Vec, '?X, ?C ] + ^^ ^^^ fresh inference variables in `self` + .. +} +``` + +Step (b) would then unify: + +``` +?A with Vec +'static with '?X +?B with ?C +``` + +(What we actually do is a mildly optimized variant of that: Rather +than eagerly instantiating all of the canonical values in the result +with variables, we instead walk the vector of values, looking for +cases where the value is just a canonical variable. In our example, +`values[2]` is `?C`, so that we means we can deduce that `?C := ?B and +`'?X := 'static`. This gives us a partial set of values. Anything for +which we do not find a value, we create an inference variable.) + diff --git a/src/traits-goals-and-clauses.md b/src/traits-goals-and-clauses.md new file mode 100644 index 000000000..ad1655726 --- /dev/null +++ b/src/traits-goals-and-clauses.md @@ -0,0 +1,148 @@ +# Goals and clauses + +In logic programming terms, a **goal** is something that you must +prove and a **clause** is something that you know is true. As +described in the [lowering to logic](./traits-lowering-to-logic.html) +chapter, Rust's trait solver is based on an extension of hereditary +harrop (HH) clauses, which extend traditional Prolog Horn clauses with +a few new superpowers. + +## Goals and clauses meta structure + +In Rust's solver, **goals** and **clauses** have the following forms +(note that the two definitions reference one another): + + Goal = DomainGoal + | Goal && Goal + | Goal || Goal + | exists { Goal } // existential quantification + | forall { Goal } // universal quantification + | if (Clause) { Goal } // implication + + Clause = DomainGoal + | Clause :- Goal // if can prove Goal, then Clause is true + | Clause && Clause + | forall { Clause } + + K = // a "kind" + | + +The proof procedure for these sorts of goals is actually quite +straightforward. Essentially, it's a form of depth-first search. The +paper +["A Proof Procedure for the Logic of Hereditary Harrop Formulas"][pphhf] +gives the details. + +[pphhf]: ./traits-bibliography.html#pphhf + + + +## Domain goals + +To define the set of *domain goals* in our system, we need to first +introduce a few simple formulations. A **trait reference** consists of +the name of a trait allow with a suitable set of inputs P0..Pn: + + TraitRef = P0: TraitName + +So, for example, `u32: Display` is a trait reference, as is `Vec: +IntoIterator`. Note that Rust surface syntax also permits some extra +things, like associated type bindings (`Vec: IntoIterator`), that are not part of a trait reference. + +A **projection** consists of a an associated item reference along with +its inputs P0..Pm: + + Projection = >::AssocItem + +Given that, we can define a `DomainGoal` as follows: + + DomainGoal = Implemented(TraitRef) + | ProjectionEq(Projection = Type) + | Normalize(Projection -> Type) + | FromEnv(TraitRef) + | FromEnv(Projection = Type) + | WellFormed(Type) + | WellFormed(TraitRef) + | WellFormed(Projection = Type) + | Outlives(Type, Region) + | Outlives(Region, Region) + +- `Implemented(TraitRef)` -- true if the given trait is + implemented for the given input types and lifetimes +- `FromEnv(TraitEnv)` -- true if the given trait is *assumed* to be implemented; + that is, if it can be derived from the in-scope where clauses + - as we'll see in the section on lowering, `FromEnv(X)` implies + `Implemented(X)` but not vice versa. This distinction is crucial + to [implied bounds]. +- `ProjectionEq(Projection = Type)` -- the given associated type `Projection` is equal + to `Type`; see [the section on associated types](./traits-associated-types.html) +- `Normalize(Projection -> Type)` -- the given associated type `Projection` can be normalized + to `Type` + - as discussed in [the section on associated types](./traits-associated-types.html), + `Normalize` implies `ProjectionEq` but not vice versa +- `WellFormed(..)` -- these goals imply that the given item is + *well-formed* + - well-formedness is important to [implied bounds]. + + + +## Coinductive goals + +Most goals in our system are "inductive". In an inductive goal, +circular reasoning is disallowed. Consider this example clause: + + Implemented(Foo: Bar) :- + Implemented(Foo: Bar). + +Considered inductively, this clause is useless: if we are trying to +prove `Implemented(Foo: Bar)`, we would then recursively have to prove +`Implemented(Foo: Bar)`, and that cycle would continue ad infinitum +(the trait solver will terminate here, it would just consider that +`Implemented(Foo: Bar)` is not known to be true). + +However, some goals are *co-inductive*. Simply put, this means that +cycles are OK. So, if `Bar` were a co-inductive trait, then the rule +above would be perfectly valid, and it would indicate that +`Implemented(Foo: Bar)` is true. + +*Auto traits* are one example in Rust where co-inductive goals are used. +Consider the `Send` trait, and imagine that we have this struct: + +```rust +struct Foo { + next: Option> +} +``` + +The default rules for auto traits say that `Foo` is `Send` if the +types of its fields are `Send`. Therefore, we would have a rule like + + Implemented(Foo: Send) :- + Implemented(Option>: Send). + +As you can probably imagine, proving that `Option>: Send` is +going to wind up circularly requiring us to prove that `Foo: Send` +again. So this would be an example where we wind up in a cycle -- but +that's ok, we *do* consider `Foo: Send` to hold, even though it +references itself. + +In general, co-inductive traits are used in Rust trait solving when we +want to enumerate a fixed set of possibilities. In the case of auto +traits, we are enumerating the set of reachable types from a given +starting point (i.e., `Foo` can reach values of type +`Option>`, which implies it can reach values of type +`Box`, and then of type `Foo`, and then the cycle is complete). + +In addition to auto traits, `WellFormed` predicates are co-inductive. +These are used to achieve a similar "enumerate all the cases" pattern, +as described in the section on [implied bounds]. + +[implied bounds]: ./traits-lowering-rules.html#implied-bounds + +## Incomplete chapter + +Some topics yet to be written: + +- Elaborate on the proof procedure +- SLG solving -- introduce negative reasoning diff --git a/src/traits-lowering-rules.md b/src/traits-lowering-rules.md new file mode 100644 index 000000000..d096b4364 --- /dev/null +++ b/src/traits-lowering-rules.md @@ -0,0 +1,267 @@ +# Lowering rules + +This section gives the complete lowering rules for Rust traits into +[program clauses][pc]. These rules reference the [domain goals][dg] defined in +an earlier section. + +[pc]: ./traits-goals-and-clauses.html +[dg]: ./traits-goals-and-clauses.html#domain-goals + +## Notation + +The nonterminal `Pi` is used to mean some generic *parameter*, either a +named lifetime like `'a` or a type paramter like `A`. + +The nonterminal `Ai` is used to mean some generic *argument*, which +might be a lifetime like `'a` or a type like `Vec`. + +When defining the lowering rules, we will give goals and clauses in +the [notation given in this section](./traits-goals-and-clauses.html). +We sometimes insert "macros" like `LowerWhereClause!` into these +definitions; these macros reference other sections within this chapter. + +## Lowering where clauses + +When used in a goal position, where clauses can be mapped directly +[domain goals][dg], as follows: + +- `A0: Foo` maps to `Implemented(A0: Foo)`. +- `A0: Foo` maps to `ProjectionEq(>::Item = T)` +- `T: 'r` maps to `Outlives(T, 'r)` +- `'a: 'b` maps to `Outlives('a, 'b)` + +In the rules below, we will use `WC` to indicate where clauses that +appear in Rust syntax; we will then use the same `WC` to indicate +where those where clauses appear as goals in the program clauses that +we are producing. In that case, the mapping above is used to convert +from the Rust syntax into goals. + +### Transforming the lowered where clauses + +In addition, in the rules below, we sometimes do some transformations +on the lowered where clauses, as defined here: + +- `FromEnv(WC)` -- this indicates that: + - `Implemented(TraitRef)` becomes `FromEnv(TraitRef)` + - `ProjectionEq(Projection = Ty)` becomes `FromEnv(Projection = Ty)` + - other where-clauses are left intact +- `WellFormed(WC)` -- this indicates that: + - `Implemented(TraitRef)` becomes `WellFormed(TraitRef)` + - `ProjectionEq(Projection = Ty)` becomes `WellFormed(Projection = Ty)` + +*TODO*: I suspect that we want to alter the outlives relations too, +but Chalk isn't modeling those right now. + +## Lowering traits + +Given a trait definition + +```rust +trait Trait // P0 == Self +where WC +{ + // trait items +} +``` + +we will produce a number of declarations. This section is focused on +the program clauses for the trait header (i.e., the stuff outside the +`{}`); the [section on trait items](#trait-items) covers the stuff +inside the `{}`. + +### Trait header + +From the trait itself we mostly make "meta" rules that setup the +relationships between different kinds of domain goals. The first dush +rule from the trait header creates the mapping between the `FromEnv` +and `Implemented` predicates: + + forall { + Implemented(Self: Trait) :- FromEnv(Self: Trait + +#### Implied bounds + +The next few clauses have to do with implied bounds (see also +[RFC 2089]). For each trait, we produce two clauses: + +[RFC 2089]: https://rust-lang.github.io/rfcs/2089-implied-bounds.html + + // For each where clause WC: + forall { + FromEnv(WC) :- FromEnv(Self: Trait { + WellFormed(Self: Trait) :- Implemented(Self: Trait), WellFormed(WC) + } + +This `WellFormed` rule states that `T: Trait` is well-formed if (a) +`T: Trait` is implemented and (b) all the where-clauses declared on +`Trait` are well-formed (and hence they are implemented). Remember +that the `WellFormed` predicate is +[coinductive](./traits-goals-and-clauses.html#coinductive); in this +case, it is serving as a kind of "carrier" that allows us to enumerate +all the where clauses that are transitively implied by `T: Trait`. + +An example: + +```rust +trait Foo: A + Bar { } +trait Bar: B + Foo { } +trait A { } +trait B { } +``` + +Here, the transitive set of implications for `T: Foo` are `T: A`, `T: Bar`, and `T: B`. +And indeed if we were to try to prove `WellFormed(T: Foo)`, we would have to prove each +one of those: + +- `WellFormed(T: Foo)` + - `Implemented(T: Foo)` + - `WellFormed(T: A)` + - `Implemented(T: A)` + - `WellFormed(T: Bar)` + - `Implemented(T: Bar)` + - `WellFormed(T: B)` + - `Implemented(T: Bar)` + - `WellFormed(T: Foo)` -- cycle, true coinductively + +This `WellFormed` predicate is only used when proving that impls are +well-formed -- basically, for each impl of some trait ref `TraitRef`, +we must that `WellFormed(TraitRef)`. This in turn justifies the +implied bounds rules that allow us to extend the set of `FromEnv` +items. + + + +## Lowering trait items + +### Associated type declarations + +Given a trait that declares a (possibly generic) associated type: + +```rust +trait Trait // P0 == Self +where WC +{ + type AssocType: Bounds where WC1; +} +``` + +We will produce a number of program clases. The first two define +the rules by which `ProjectionEq` can succeed; these two clauses are discussed +in detail in the [section on associated types](./traits-associated-types.html). + + // ProjectionEq can succeed by normalizing: + forall { + ProjectionEq(>::AssocType = U) :- + Normalize(>::AssocType -> U) + } + + // ProjectionEq can succeed by skolemizing: + forall { + ProjectionEq( + >::AssocType = + (Trait::AssocType) + ) :- + // But only if the trait is implemented, and the conditions from + // the associated type are met as well: + Implemented(Self: Trait), + WC1 + } + +The next rule covers implied bounds for the projection. In particular, +the `Bounds` declared on the associated type must be proven to hold to +show that the impl is well-formed, and hence we can rely on them from +elsewhere. + + // XXX how exactly should we set this up? Have to be careful; + // presumably this has to be a kind of `FromEnv` setup. + +### Lowering function and constant declarations + +Chalk didn't model functions and constants, but I would eventually +like to treat them exactly like normalization. See [the section on function/constant +values below](#constant-vals) for more details. + +## Lowering impls + +Given an impl of a trait: + +```rust +impl Trait for A0 +where WC +{ + // zero or more impl items +} +``` + +Let `TraitRef` be the trait reference `A0: Trait`. Then we +will create the following rules: + + forall { + Implemented(TraitRef) :- WC + } + +In addition, we will lower all of the *impl items*. + +## Lowering impl items + +### Associated type values + +Given an impl that contains: + +```rust +impl Trait for A0 +where WC +{ + type AssocType: Bounds where WC1 = T; +} +``` + +We produce the following rule: + + forall { + forall { + Normalize(>::AssocType -> T) :- + WC, WC1 + } + } + +Note that `WC` and `WC1` both encode where-clauses that the impl can +rely on, whereas the bounds `Bounds` on the associated type are things +that the impl must prove (see the well-formedness checking). + + + +### Function and constant values + +Chalk didn't model functions and constants, but I would eventually +like to treat them exactly like normalization. This presumably +involves adding a new kind of parameter (constant), and then having a +`NormalizeValue` domain goal. This is *to be written* because the +details are a bit up in the air. + + diff --git a/src/traits-lowering-to-logic.md b/src/traits-lowering-to-logic.md new file mode 100644 index 000000000..7b1c2ed6b --- /dev/null +++ b/src/traits-lowering-to-logic.md @@ -0,0 +1,182 @@ +# Lowering to logic + +The key observation here is that the Rust trait system is basically a +kind of logic, and it can be mapped onto standard logical inference +rules. We can then look for solutions to those inference rules in a +very similar fashion to how e.g. a [Prolog] solver works. It turns out +that we can't *quite* use Prolog rules (also called Horn clauses) but +rather need a somewhat more expressive variant. + +[Prolog]: https://en.wikipedia.org/wiki/Prolog + +## Rust traits and logic + +One of the first observations is that the Rust trait system is +basically a kind of logic. As such, we can map our struct, trait, and +impl declarations into logical inference rules. For the most part, +these are basically Horn clauses, though we'll see that to capture the +full richness of Rust -- and in particular to support generic +programming -- we have to go a bit further than standard Horn clauses. + +To see how this mapping works, let's start with an example. Imagine +we declare a trait and a few impls, like so: + +```rust +trait Clone { } +impl Clone for usize { } +impl Clone for Vec where T: Clone { } +``` + +We could map these declarations to some Horn clauses, written in a +Prolog-like notation, as follows: + +``` +Clone(usize). +Clone(Vec) :- Clone(?T). +``` + +In Prolog terms, we might say that `Clone(Foo)` -- where `Foo` is some +Rust type -- is a *predicate* that represents the idea that the type +`Foo` implements `Clone`. These rules are **program clauses**; they +state the conditions under which that predicate can be proven (i.e., +considered true). So the first rule just says "Clone is implemented +for `usize`". The next rule says "for any type `?T`, Clone is +implemented for `Vec` if clone is implemented for `?T`". So +e.g. if we wanted to prove that `Clone(Vec>)`, we would do +so by applying the rules recursively: + +- `Clone(Vec>)` is provable if: + - `Clone(Vec)` is provable if: + - `Clone(usize)` is provable. (Which is is, so we're all good.) + +But now suppose we tried to prove that `Clone(Vec)`. This would +fail (after all, I didn't give an impl of `Clone` for `Bar`): + +- `Clone(Vec)` is provable if: + - `Clone(Bar)` is provable. (But it is not, as there are no applicable rules.) + +We can easily extend the example above to cover generic traits with +more than one input type. So imagine the `Eq` trait, which declares +that `Self` is equatable with a value of type `T`: + +```rust +trait Eq { ... } +impl Eq for usize { } +impl> Eq> for Vec { } +``` + +That could be mapped as follows: + +``` +Eq(usize, usize). +Eq(Vec, Vec) :- Eq(?T, ?U). +``` + +So far so good. + +## Type-checking normal functions + +OK, now that we have defined some logical rules that are able to +express when traits are implemented and to handle associated types, +let's turn our focus a bit towards **type-checking**. Type-checking is +interesting because it is what gives us the goals that we need to +prove. That is, everything we've seen so far has been about how we +derive the rules by which we can prove goals from the traits and impls +in the program; but we are also interesting in how derive the goals +that we need to prove, and those come from type-checking. + +Consider type-checking the function `foo()` here: + +```rust +fn foo() { bar::() } +fn bar() { } +``` + +This function is very simple, of course: all it does is to call +`bar::()`. Now, looking at the definition of `bar()`, we can see +that it has one where-clause `U: Eq`. So, that means that `foo()` will +have to prove that `usize: Eq` in order to show that it can call `bar()` +with `usize` as the type argument. + +If we wanted, we could write a Prolog predicate that defines the +conditions under which `bar()` can be called. We'll say that those +conditions are called being "well-formed": + +``` +barWellFormed(?U) :- Eq(?U). +``` + +Then we can say that `foo()` type-checks if the reference to +`bar::` (that is, `bar()` applied to the type `usize`) is +well-formed: + +``` +fooTypeChecks :- barWellFormed(usize). +``` + +If we try to prove the goal `fooTypeChecks`, it will succeed: + +- `fooTypeChecks` is provable if: + - `barWellFormed(usize)`, which is provable if: + - `Eq(usize)`, which is provable because of an impl. + +Ok, so far so good. Let's move on to type-checking a more complex function. + +## Type-checking generic functions: beyond Horn clauses + +In the last section, we used standard Prolog horn-clauses (augmented with Rust's +notion of type equality) to type-check some simple Rust functions. But that only +works when we are type-checking non-generic functions. If we want to type-check +a generic function, it turns out we need a stronger notion of goal than Prolog +can be provide. To see what I'm talking about, let's revamp our previous +example to make `foo` generic: + +```rust +fn foo() { bar::() } +fn bar() { } +``` + +To type-check the body of `foo`, we need to be able to hold the type +`T` "abstract". That is, we need to check that the body of `foo` is +type-safe *for all types `T`*, not just for some specific type. We might express +this like so: + +``` +fooTypeChecks :- + // for all types T... + forall { + // ...if we assume that Eq(T) is provable... + if (Eq(T)) { + // ...then we can prove that `barWellFormed(T)` holds. + barWellFormed(T) + } + }. +``` + +This notation I'm using here is the notation I've been using in my +prototype implementation; it's similar to standard mathematical +notation but a bit Rustified. Anyway, the problem is that standard +Horn clauses don't allow universal quantification (`forall`) or +implication (`if`) in goals (though many Prolog engines do support +them, as an extension). For this reason, we need to accept something +called "first-order hereditary harrop" (FOHH) clauses -- this long +name basically means "standard Horn clauses with `forall` and `if` in +the body". But it's nice to know the proper name, because there is a +lot of work describing how to efficiently handle FOHH clauses; see for +example Gopalan Nadathur's excellent +["A Proof Procedure for the Logic of Hereditary Harrop Formulas"][pphhf] +in [the bibliography]. + +[the bibliography]: ./traits-bibliography.html +[pphhf]: ./traits-bibliography.html#pphhf + +It turns out that supporting FOHH is not really all that hard. And +once we are able to do that, we can easily describe the type-checking +rule for generic functions like `foo` in our logic. + +## Source + +This page is a lightly adapted version of a +[blog post by Nicholas Matsakis][lrtl]. + +[lrtl]: http://smallcultfollowing.com/babysteps/blog/2017/01/26/lowering-rust-traits-to-logic/ diff --git a/src/traits-regions.md b/src/traits-regions.md new file mode 100644 index 000000000..baa3582b6 --- /dev/null +++ b/src/traits-regions.md @@ -0,0 +1,3 @@ +# Region constraints + +*to be written* diff --git a/src/traits-wf.md b/src/traits-wf.md new file mode 100644 index 000000000..c002b8cc1 --- /dev/null +++ b/src/traits-wf.md @@ -0,0 +1,11 @@ +# Well-formedness checking + +This chapter is mostly *to be written*. WF checking, in short, has the +job of checking that the various declarations in a Rust program are +well-formed. This is the basis for implied bounds, and partly for that +reason, this checking can be surprisingly subtle! (For example, we +have to be sure that each impl proves the WF conditions declared on +the trait.) + + + diff --git a/src/traits.md b/src/traits.md new file mode 100644 index 000000000..28e27e82e --- /dev/null +++ b/src/traits.md @@ -0,0 +1,18 @@ +# Trait solving (new-style) + +🚧 This chapter describes "new-style" trait solving. This is still in +the process of being implemented; this chapter serves as a kind of +in-progress design document. If you would prefer to read about how the +current trait solver works, check out +[this chapter](./trait-resolution.html).🚧 + +Trait solving is based around a few key ideas: + +- [Canonicalization](./traits-canonicalization.html), which allows us to + extract types that contain inference variables in them from their + inference context, work with them, and then bring the results back + into the original context. +- [Lowering to logic](./traits-lowering-to-logic.html), which expresses + Rust traits in terms of standard logical terms. + +*more to come* diff --git a/src/type-inference.md b/src/type-inference.md index 8812d34d7..3795ea70e 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -43,6 +43,8 @@ The `tcx.infer_ctxt` method actually returns a builder, which means there are some kinds of configuration you can do before the `infcx` is created. See `InferCtxtBuilder` for more information. + + ## Inference variables The main purpose of the inference context is to house a bunch of From e4aaa9c7f9e23d24e8f5d566143c232631970014 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 9 Mar 2018 04:14:18 -0500 Subject: [PATCH 129/648] clarify how there are two traits chapters --- src/trait-resolution.md | 2 +- src/traits.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/trait-resolution.md b/src/trait-resolution.md index cccfcd54b..d6edcf68a 100644 --- a/src/trait-resolution.md +++ b/src/trait-resolution.md @@ -6,7 +6,7 @@ some non-obvious things. **Note:** This chapter (and its subchapters) describe how the trait solver **currently** works. However, we are in the process of designing a new trait solver. If you'd prefer to read about *that*, -see [the traits chapter](./traits.html). +see [*this* traits chapter](./traits.html). ## Major concepts diff --git a/src/traits.md b/src/traits.md index 28e27e82e..0098704d8 100644 --- a/src/traits.md +++ b/src/traits.md @@ -4,7 +4,7 @@ the process of being implemented; this chapter serves as a kind of in-progress design document. If you would prefer to read about how the current trait solver works, check out -[this chapter](./trait-resolution.html).🚧 +[this other chapter](./trait-resolution.html).🚧 Trait solving is based around a few key ideas: From 6f99f12c2acbceae307c43e53a05071aa434380b Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 9 Mar 2018 04:16:34 -0500 Subject: [PATCH 130/648] link to traits working group tracking issue --- src/traits.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/src/traits.md b/src/traits.md index 0098704d8..f0e67ffd4 100644 --- a/src/traits.md +++ b/src/traits.md @@ -1,10 +1,15 @@ # Trait solving (new-style) -🚧 This chapter describes "new-style" trait solving. This is still in -the process of being implemented; this chapter serves as a kind of +🚧 This chapter describes "new-style" trait solving. This is still in the +[process of being implemented][wg]; this chapter serves as a kind of in-progress design document. If you would prefer to read about how the current trait solver works, check out -[this other chapter](./trait-resolution.html).🚧 +[this other chapter](./trait-resolution.html). (By the way, if you +would like to help in hacking on the new solver, you will find +instructions for getting involved in the +[Traits Working Group tracking issue][wg].) 🚧 + +[wg]: https://github.com/rust-lang/rust/issues/48416 Trait solving is based around a few key ideas: From fd92491375f30b89050ebe51130112d1cee8896e Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 9 Mar 2018 04:19:36 -0500 Subject: [PATCH 131/648] don't say 'thing' --- src/traits-canonicalization.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/src/traits-canonicalization.md b/src/traits-canonicalization.md index 35352d605..8ab73a7ff 100644 --- a/src/traits-canonicalization.md +++ b/src/traits-canonicalization.md @@ -2,13 +2,15 @@ Canonicalization is the process of **isolating** an inference value from its context. It is really based on a very simple concept: every -[inference variable](./type-inference.html#vars) is always in one of two -states: either it is **unbound**, in which case we don't know yet what -type it is, or it is **bound**, in which case we do. So to isolate -some thing T from its environment, we just walk down and find the -unbound variables that appear in T; those variables get renumbered in -a canonical order (left to right, for the most part, but really it -doesn't matter as long as it is consistent). +[inference variable](./type-inference.html#vars) is always in one of +two states: either it is **unbound**, in which case we don't know yet +what type it is, or it is **bound**, in which case we do. So to +isolate some data-structure T that contains types/regions from its +environment, we just walk down and find the unbound variables that +appear in T; those variables get replaced with "canonical variables", +starting from zero and numbered in a fixed order (left to right, for +the most part, but really it doesn't matter as long as it is +consistent). So, for example, if we have the type `X = (?T, ?U)`, where `?T` and `?U` are distinct, unbound inference variables, then the canonical From 3c8a827a37a5c48661fdaf4e5b34806f528bffd9 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 9 Mar 2018 04:30:48 -0500 Subject: [PATCH 132/648] expand reorder topic list slightly --- src/SUMMARY.md | 2 +- src/traits-canonicalization.md | 5 +++-- src/traits.md | 22 ++++++++++++++++------ 3 files changed, 20 insertions(+), 9 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 1c5bc284d..9c1920d32 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -22,8 +22,8 @@ - [Caching subtleties](./trait-caching.md) - [Specialization](./trait-specialization.md) - [Trait solving (new-style)](./traits.md) - - [Canonicalization](./traits-canonicalization.md) - [Lowering to logic](./traits-lowering-to-logic.md) + - [Canonical queries](./traits-canonicalization.md) - [Goals and clauses](./traits-goals-and-clauses.md) - [Lowering rules](./traits-lowering-rules.md) - [Equality and associated types](./traits-associated-types.md) diff --git a/src/traits-canonicalization.md b/src/traits-canonicalization.md index 8ab73a7ff..9b71c8470 100644 --- a/src/traits-canonicalization.md +++ b/src/traits-canonicalization.md @@ -24,8 +24,9 @@ they are repeated. We use this to improve caching as well as to detect cycles and other things during trait resolution. Roughly speaking, the idea is that if two trait queries have the same canonicalize form, then they will get -the same answer -- modulo the precise identities of the variables -involved. +the same answer. That answer will be expressed in terms of the +canonical variables (`?0`, `?1`), which we can then map back to the +original variables (`?T`, `?U`). To see how it works, imagine that we are asking to solve the following trait query: `?A: Foo<'static, ?B>`, where `?A` and `?B` are unbound. diff --git a/src/traits.md b/src/traits.md index f0e67ffd4..18cb0c649 100644 --- a/src/traits.md +++ b/src/traits.md @@ -13,11 +13,21 @@ instructions for getting involved in the Trait solving is based around a few key ideas: -- [Canonicalization](./traits-canonicalization.html), which allows us to - extract types that contain inference variables in them from their - inference context, work with them, and then bring the results back - into the original context. - [Lowering to logic](./traits-lowering-to-logic.html), which expresses Rust traits in terms of standard logical terms. - -*more to come* + - The [goals and clauses](./traits-goals-and-clauses.html) chapter + describes the precise lowering rules we use. +- [Canonical queries](./traits-canonicalization.html), which allow us + to solve trait problems (like "is `Foo` implemented for the type + `Bar`?") once, and then apply that same result independently in many + different inference contexts. +- [Lazy normalization](./traits-associated-types.html), which is the + technique we use to accommodate associated types when figuring out + whether types are equal. +- [Region constraints](./traits-regions.md), which are accumulated + during trait solving but mostly ignored. This means that trait + solving effectively ignores the precise regions involved, always -- + but we still remember the constraints on them so that those + constraints can be checked by thet type checker. + +Note: this is not a complete list of topics. See the sidebar for more. From c3a35021d6e607ecc7b05573875b49d476ded8f6 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 10 Mar 2018 05:32:47 -0500 Subject: [PATCH 133/648] add background material on trait queries --- src/SUMMARY.md | 8 +- src/traits-canonical-queries.md | 228 ++++++++++++++++++++++++++++++++ src/traits-canonicalization.md | 7 +- src/traits-slg.md | 1 + src/traits.md | 2 +- 5 files changed, 241 insertions(+), 5 deletions(-) create mode 100644 src/traits-canonical-queries.md create mode 100644 src/traits-slg.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 9c1920d32..95a5ee577 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -23,12 +23,14 @@ - [Specialization](./trait-specialization.md) - [Trait solving (new-style)](./traits.md) - [Lowering to logic](./traits-lowering-to-logic.md) - - [Canonical queries](./traits-canonicalization.md) - - [Goals and clauses](./traits-goals-and-clauses.md) - - [Lowering rules](./traits-lowering-rules.md) + - [Goals and clauses](./traits-goals-and-clauses.md) + - [Lowering rules](./traits-lowering-rules.md) + - [Canonical queries](./traits-canonical-queries.md) + - [Canonicalization](./traits-canonicalization.md) - [Equality and associated types](./traits-associated-types.md) - [Region constraints](./traits-regions.md) - [Well-formedness checking](./traits-wf.md) + - [The SLG solver](./traits-slg.md) - [Bibliography](./traits-bibliography.md) - [Type checking](./type-checking.md) - [The MIR (Mid-level IR)](./mir.md) diff --git a/src/traits-canonical-queries.md b/src/traits-canonical-queries.md new file mode 100644 index 000000000..ec861eb47 --- /dev/null +++ b/src/traits-canonical-queries.md @@ -0,0 +1,228 @@ +# Canonical queries + +The "start" of the trait system is the **canonical query** (these are +both queries in the more general sense of the word -- something you +would like to know the answer to -- and in the +[rustc-specific sense](./query.html)). The idea is that the type +checker or other parts of the system, may in the course of doing their +thing want to know whether some trait is implemented for some type +(e.g., is `u32: Debug` true?). Or they may want to +[normalize some associated type](./traits-associated-types.html). + +This section covers queries at a fairly high level of abstraction. The +subsections look a bit more closely at how these ideas are implemented +in rustc. + +## The traditional, interactive Prolog query + +In a traditional Prolog system, when you start a query, the solver +will run off and start supplying you with every possible answer it can +find. So given something like this: + + ?- Vec: AsRef + +The solver might answer: + + Vec: AsRef<[i32]> + continue? (y/n) + +This `continue` bit is interesting. The idea in Prolog is that the +solver is finding **all possible** instantiations of your query that +are true. In this case, if we instantiate `?U = [i32]`, then the query +is true (note that a traditional Prolog interface does not, directly, +tell us a value for `?U`, but we can infer one by unifying the +response with our original query -- Rust's solver gives back a +substitution instead). If we were to hit `y`, the solver might then +give us another possible answer: + + Vec: AsRef> + continue? (y/n) + +This answer derives from the fact that there is a reflexive impl +(`impl AsRef for T`) for `AsRef`. If were to hit `y` again, +then we might get back a negative response: + + no + +Naturally, in some cases, there may be no possible answers, and hence +the solver will just give me back `no` right away: + + ?- Box: Copy + no + +In some cases, there might be an infinite number of responses. So for +example if I gave this query, and I kept hitting `y`, then the solver +would never stop giving me back answers: + + ?- Vec: Clone + Vec: Clone + continue? (y/n) + Vec>: Clone + continue? (y/n) + Vec>>: Clone + continue? (y/n) + Vec>>>: Clone + continue? (y/n) + +As you can imagine, the solver will gleefully keep adding another +layer of `Box` until we ask it to stop, or it runs out of memory. + +Another interesting thing is that queries might still have variables +in them. For example: + + ?- Rc: Clone + +might produce the answer: + + Rc: Clone + continue? (y/n) + +After all, `Rc` is true **no matter what type `?T` is**. + +## A trait query in rustc + +The trait queries in rustc work somewhat differently. Instead of +trying to enumerate **all possible** answers for you, they are looking +for an **unambiguous** answer. In particular, when they tells you the +value for a type variable, that means that this is the **only possible +instantiation** that you could use, given the current set of impls and +where-clauses, that would be provable. (Internally within the solver, +though, they can potentially enumerate all possible answers. See +[the description of the SLG solver](./traits-slg.html) for details.) + +The response to a trait query in rustc is typically a +`Result, NoSolution>` (where the `T` will vary a bit +depending on the query itself). The `Err(NoSolution)` case indicates +that the query was false and had no answers (e.g., `Box: Copy`). +Otherwise, the `QueryResult` gives back information about the possible answer(s) +we did find. It consists of four parts: + +- **Certainty:** tells you how sure we are of this answer. It can have two values: + - `Proven` means that the result is known to be true. + - This might be the result for trying to prove `Vec: Clone`, + say, or `Rc: Clone`. + - `Ambiguous` means that there were things we could not yet prove to + be either true *or* false, typically because more type information + was needed. (We'll see an example shortly.) + - This might be the result for trying to prove `Vec: Clone`. +- **Var values:** Values for each of the unbound inference variables + (like `?T`) that appeared in your original query. (Remember that in Prolog, + we had to infer these.) + - As we'll see in the example below, we can get back var values even + for `Ambiguous` cases. +- **Region constraints:** these are relations that must hold between + the lifetimes that you supplied as inputs. We'll ignore these here, + but see the + [section on handling regions in traits](./traits-regions.html) for + more details. +- **Value:** The query result also comes with a value of type `T`. For + some specialized queries -- like normalizing associated types -- + this is used to carry back an extra result, but it's often just + `()`. + +### Examples + +Let's work through an example query to see what all the parts mean. +Consider [the `Borrow` trait][borrow]. This trait has a number of +impls; among them, there are these two (for clarify, I've written the +`Sized` bounds explicitly): + +[borrow]: https://doc.rust-lang.org/std/borrow/trait.Borrow.html + +```rust +impl Borrow for T where T: ?Sized +impl Borrow<[T]> for Vec where T: Sized +``` + +**Example 1.** Imagine we are type-checking this (rather artificial) +bit of code: + +```rust +fn foo(a: A, vec_b: Option) where A: Borrow { } + +fn main() { + let mut t: Vec<_> = vec![]; // Type: Vec + let mut u: Option<_> = None; // Type: Option + foo(t, u); // Example 1: requires `Vec: Borrow` + ... +} +``` + +As the comments indicate, we first create two variables `t` and `u`; +`t` is an empty vector and `u` is a `None` option. Both of these +variables have unbound inference variables in their type: `?T` +represents the elements in the vector `t` and `?U` represents the +value stored in the option `u`. Next, we invoke `foo`; comparing the +signature of `foo` to its arguments, we wind up with `A = Vec` and +`B = ?U`.Therefore, the where clause on `foo` requires that `Vec: +Borrow`. This is thus our first example trait query. + +There are many possible solutions to the query `Vec: Borrow`; +for example: + +- `?U = Vec`, +- `?U = [?T]`, +- `?T = u32, ?U = [u32]` +- and so forth. + +Therefore, the result we get back would be as follows (I'm going to +ignore region constraints and the "value"): + +- Certainty: `Ambiguous` -- we're not sure yet if this holds +- Var values: `[?T = ?T, ?U = ?U]` -- we learned nothing about the values of the variables + +In short, the query result says that it is too soon to say much about +whether this trait is proven. During type-checking, this is not an +immediate error: instead, the type checker would hold on to this +requirement (`Vec: Borrow`) and wait. As we'll see in the next +example, it may happen that `?T` and `?U` wind up constrained from +other sources, in which case we can try the trait query again. + +**Example 2.** We can now extend our previous example a bit, +and assign a value to `u`: + +```rust +fn foo(a: A, vec_b: Option) where A: Borrow { } + +fn main() { + // What we saw before: + let mut t: Vec<_> = vec![]; // Type: Vec + let mut u: Option<_> = None; // Type: Option + foo(t, u); // `Vec: Borrow` => ambiguous + + // New stuff: + u = Some(vec![]); // ?U = Vec +} +``` + +As a result of this assignment, the type of `u` is forced to be +`Option>`, where `?V` represents the element type of the +vector. This in turn implies that `?U` is [unified] to `Vec`. + +[unified]: ./type-checking.html + +Let's suppose that the type checker decides to revisit the +"as-yet-unproven" trait obligation we saw before, `Vec: +Borrow`. `?U` is no longer an unbound inference variable; it now +has a value, &. So, if we "refresh" the query with that value, we get: + + Vec: Borrow> + +This time, there is only one impl that applies, the reflexive impl: + + impl Borrow for T where T: ?Sized + +Therefore, the trait checker will answer: + +- Certainty: `Proven` +- Var values: `[?T = ?T, ?V = ?T]` + +Here, it is saying that we have indeed proven that the obligation +holds, and we also know that `?T` and `?V` are the same type (but we +don't know what that type is yet!). + +(In fact, as the function ends here, the type checker would give an +error at this point, since the element types of `t` and `u` are still +not yet known, even though they are known to be the same.) + + diff --git a/src/traits-canonicalization.md b/src/traits-canonicalization.md index 9b71c8470..94b1d6b85 100644 --- a/src/traits-canonicalization.md +++ b/src/traits-canonicalization.md @@ -1,7 +1,10 @@ # Canonicalization Canonicalization is the process of **isolating** an inference value -from its context. It is really based on a very simple concept: every +from its context. It is a key part of implementing +[canonical queries][cq]. + +Canonicalization is really based on a very simple concept: every [inference variable](./type-inference.html#vars) is always in one of two states: either it is **unbound**, in which case we don't know yet what type it is, or it is **bound**, in which case we do. So to @@ -12,6 +15,8 @@ starting from zero and numbered in a fixed order (left to right, for the most part, but really it doesn't matter as long as it is consistent). +[cq]: ./traits-canonical-queries.html + So, for example, if we have the type `X = (?T, ?U)`, where `?T` and `?U` are distinct, unbound inference variables, then the canonical form of `X` would be `(?0, ?1)`, where `?0` and `?1` represent these diff --git a/src/traits-slg.md b/src/traits-slg.md new file mode 100644 index 000000000..cdd52ece9 --- /dev/null +++ b/src/traits-slg.md @@ -0,0 +1 @@ +# The SLG solver diff --git a/src/traits.md b/src/traits.md index 18cb0c649..6994f33d2 100644 --- a/src/traits.md +++ b/src/traits.md @@ -24,7 +24,7 @@ Trait solving is based around a few key ideas: - [Lazy normalization](./traits-associated-types.html), which is the technique we use to accommodate associated types when figuring out whether types are equal. -- [Region constraints](./traits-regions.md), which are accumulated +- [Region constraints](./traits-regions.html), which are accumulated during trait solving but mostly ignored. This means that trait solving effectively ignores the precise regions involved, always -- but we still remember the constraints on them so that those From 22b5ec479684ad8349b2173d784b9588f4512b84 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 10 Mar 2018 05:45:36 -0500 Subject: [PATCH 134/648] update for notation --- src/traits-canonicalization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/traits-canonicalization.md b/src/traits-canonicalization.md index 94b1d6b85..db2f939f6 100644 --- a/src/traits-canonicalization.md +++ b/src/traits-canonicalization.md @@ -60,7 +60,7 @@ result, we'll have an array of values for the canonical inputs. For example, the canonical result might be: ``` -for<2> { +for { values = [ Vec, '1, ?0 ] ^^ ^^ ^^ these are variables in the result! ... From 53115310f72cf5fdf46a607e79aa9819957fcf34 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 10 Mar 2018 06:44:29 -0500 Subject: [PATCH 135/648] rework canon section substantially to spell out steps more clearly --- src/traits-canonical-queries.md | 2 + src/traits-canonicalization.md | 175 +++++++++++++++++++++++++++----- 2 files changed, 151 insertions(+), 26 deletions(-) diff --git a/src/traits-canonical-queries.md b/src/traits-canonical-queries.md index ec861eb47..ee66a73db 100644 --- a/src/traits-canonical-queries.md +++ b/src/traits-canonical-queries.md @@ -90,6 +90,8 @@ where-clauses, that would be provable. (Internally within the solver, though, they can potentially enumerate all possible answers. See [the description of the SLG solver](./traits-slg.html) for details.) + + The response to a trait query in rustc is typically a `Result, NoSolution>` (where the `T` will vary a bit depending on the query itself). The `Err(NoSolution)` case indicates diff --git a/src/traits-canonicalization.md b/src/traits-canonicalization.md index db2f939f6..fc55fac0d 100644 --- a/src/traits-canonicalization.md +++ b/src/traits-canonicalization.md @@ -2,7 +2,8 @@ Canonicalization is the process of **isolating** an inference value from its context. It is a key part of implementing -[canonical queries][cq]. +[canonical queries][cq], and you may wish to read the parent chapter +to get more context. Canonicalization is really based on a very simple concept: every [inference variable](./type-inference.html#vars) is always in one of @@ -33,6 +34,8 @@ the same answer. That answer will be expressed in terms of the canonical variables (`?0`, `?1`), which we can then map back to the original variables (`?T`, `?U`). +## Canonicalizing the query + To see how it works, imagine that we are asking to solve the following trait query: `?A: Foo<'static, ?B>`, where `?A` and `?B` are unbound. This query contains two unbound variables, but it also contains the @@ -48,54 +51,174 @@ Sometimes we write this differently, like so: for { ?0: Foo<'?1, ?2> } This `for<>` gives some information about each of the canonical -variables within. In this case, I am saying that `?0` is a type -(`T`), `?1` is a lifetime (`L`), and `?2` is also a type (`T`). The -`canonicalize` method *also* gives back a `CanonicalVarValues` array -with the "original values" for each canonicalized variable: +variables within. In this case, each `T` indicates a type variable, +so `?0` and `?2` are types; the `L` indicates a lifetime varibale, so +`?1` is a lifetime. The `canonicalize` method *also* gives back a +`CanonicalVarValues` array OV with the "original values" for each +canonicalized variable: [?A, 'static, ?B] + +We'll need this vector OV later, when we process the query response. + +## Executing the query + +Once we've constructed the canonical query, we can try to solve it. +To do so, we will wind up creating a fresh inference context and +**instantiating** the canonical query in that context. The idea is that +we create a substitution S from the canonical form containing a fresh +inference variable (of suitable kind) for each canonical variable. +So, for our example query: -Now we do the query and get back some result R. As part of that -result, we'll have an array of values for the canonical inputs. For -example, the canonical result might be: + for { ?0: Foo<'?1, ?2> } + +the substitution S might be: + + S = [?A, '?B, ?C] + +We can then replace the bound canonical variables (`?0`, etc) with +these inference variables, yielding the following fully instantiated +query: + + ?A: Foo<'?B, ?C> + +Remember that substitution S though! We're going to need it later. + +OK, now that we have a fresh inference context and an instantiated +query, we can go ahead and try to solve it. The trait solver itself is +explained in more detail in [another section](./traits-slg.html), but +suffice to say that it will compute a [certainty value][cqqr] (`Proven` or +`Ambiguous`) and have side-effects on the inference variables we've +created. For example, if there were only one impl of `Foo`, like so: + +[cqqr]: ./traits-canonical-queries.html#query-response ``` -for { - values = [ Vec, '1, ?0 ] - ^^ ^^ ^^ these are variables in the result! - ... -} +impl<'a, X> Foo<'a, X> for Vec +where X: 'a +{ ... } ``` -Note that this result is itself canonical and may include some -variables (in this case, `?0`). +then we might wind up with a certainty value of `Proven`, as well as +creating fresh inference variables `'?D` and `?E` (to represent the +parameters on the impl) and unifying as follows: + +- `'?B = '?D` +- `?A = Vec` +- `?C = ?E` + +We would also accumulate the region constraint `?E: '?D`, due to the +where clause. + +In order to create our final query result, we have to "lift" these +values out of the query's inference context and into something that +can be reapplied in our original inference context. We do that by +**re-applying canonicalization**, but to the **query result**. -What we want to do conceptually is to (a) instantiate each of the -canonical variables in the result with a fresh inference variable -and then (b) unify the values in the result with the original values. -Doing step (a) would yield a result of +## Canonicalizing the query result + +As discussed in [the parent section][cqqr], most trait queries wind up +with a result that brings together a "certainty value" `certainty`, a +result substitution `var_values`, and some region constraints. To +create this, we wind up re-using the substitution S that we created +when first instantiating our query. To refresh your memory, we had a query + + for { ?0: Foo<'?1, ?2> } + +for which we made a substutition S: + + S = [?A, '?B, ?C] + +We then did some work which unified some of those variables with other things. If we +"refresh" S with the latest results, we get: + + S = [Vec, '?D, ?E] + +These are precisely the new values for the three input variables from +our original query. Note though that they include some new variables +(like `?E`). We can make those go away by canonicalizing again! We don't +just canonicalize S, though, we canonicalize the whole query response QR: + + QR = { + certainty: Proven, // or whatever + var_values: [Vec, '?D, ?E] // this is S + region_constraints: [?E: '?D], // from the impl + value: (), // for our purposes, just (), but + // in some cases this might have + // a type or other info + } + +The result would be as follows: + + Canonical(QR) = for { + certainty: Proven, + var_values: [Vec, '?1, ?2] + region_constraints: [?2: '?1], + value: (), + } + +(One subtle point: when we canonicalize the query **result**, we do not +use any special treatment for free lifetimes. Note that both +references to `'?D`, for example, were converted into the same +canonical variable (`?1`). This is in contrast to the original query, +where we canonicalized every free lifetime into a fresh canonical +variable.) + +Now, this result must be reapplied in each context where needed. + +## Processing the canonicalized query result + +In the previous section we produced a canonical query result. We now have +to apply that result in our original context. If you recall, way back in the +beginning, we were trying to prove this query: + + ?A: Foo<'static, ?B> + +We canonicalized that into this: + + for { ?0: Foo<'?1, ?2> } + +and now we got back a canonical response: + + for { + certainty: Proven, + var_values: [Vec, '?1, ?2] + region_constraints: [?2: '?1], + value: (), + } + +We now want to apply that response to our context. Conceptually, how +we do that is to (a) instantiate each of the canonical variables in +the result with a fresh inference variable, (b) unify the values in +the result with the original values, and then (c) record the region +constraints for later. Doing step (a) would yield a result of ``` { - values = [ Vec, '?X, ?C ] - ^^ ^^^ fresh inference variables in `self` - .. -} + certainty: Proven, + var_values: [Vec, '?D, ?C] + ^^ ^^^ fresh inference variables + region_constraints: [?C: '?D], + value: (), +} ``` Step (b) would then unify: ``` ?A with Vec -'static with '?X +'static with '?D ?B with ?C ``` -(What we actually do is a mildly optimized variant of that: Rather +And finally the region constraint of `?C: 'static` would be recorded +for later verification. + +(What we *actually* do is a mildly optimized variant of that: Rather than eagerly instantiating all of the canonical values in the result with variables, we instead walk the vector of values, looking for cases where the value is just a canonical variable. In our example, `values[2]` is `?C`, so that we means we can deduce that `?C := ?B and -`'?X := 'static`. This gives us a partial set of values. Anything for +`'?D := 'static`. This gives us a partial set of values. Anything for which we do not find a value, we create an inference variable.) From 95f99572771272df3a196662211a8d95b6ca21ef Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 10 Mar 2018 07:04:16 -0500 Subject: [PATCH 136/648] numerous edits --- src/SUMMARY.md | 6 +++--- src/glossary.md | 3 +++ src/traits-associated-types.md | 2 ++ src/traits-goals-and-clauses.md | 18 +++++++++++++---- src/traits-lowering-rules.md | 35 +++++++++++++++++---------------- src/traits-lowering-to-logic.md | 23 ++++++++++++---------- 6 files changed, 53 insertions(+), 34 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 95a5ee577..6612372be 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -24,11 +24,11 @@ - [Trait solving (new-style)](./traits.md) - [Lowering to logic](./traits-lowering-to-logic.md) - [Goals and clauses](./traits-goals-and-clauses.md) - - [Lowering rules](./traits-lowering-rules.md) + - [Equality and associated types](./traits-associated-types.md) + - [Region constraints](./traits-regions.md) - [Canonical queries](./traits-canonical-queries.md) - [Canonicalization](./traits-canonicalization.md) - - [Equality and associated types](./traits-associated-types.md) - - [Region constraints](./traits-regions.md) + - [Lowering rules](./traits-lowering-rules.md) - [Well-formedness checking](./traits-wf.md) - [The SLG solver](./traits-slg.md) - [Bibliography](./traits-bibliography.md) diff --git a/src/glossary.md b/src/glossary.md index 7152cba36..e542d4e35 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -31,10 +31,12 @@ LTO | Link-Time Optimizations. A set of optimizations offer [LLVM] | (actually not an acronym :P) an open-source compiler backend. It accepts LLVM IR and outputs native binaries. Various languages (e.g. Rust) can then implement a compiler front-end that output LLVM IR and use LLVM to compile to all the platforms LLVM supports. MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) miri | an interpreter for MIR used for constant evaluation ([see more](./miri.html)) +normalize | a general term for converting to a more canonical form, but in the case of rustc typically refers to [associated type normalization](./traits-associated-types.html#normalize) newtype | a "newtype" is a wrapper around some other type (e.g., `struct Foo(T)` is a "newtype" for `T`). This is commonly used in Rust to give a stronger type for indices. NLL | [non-lexical lifetimes](./mir-regionck.html), an extension to Rust's borrowing system to make it be based on the control-flow graph. node-id or NodeId | an index identifying a particular node in the AST or HIR; gradually being phased out and replaced with `HirId`. obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) +projection | a general term for a "relative path", e.g. `x.f` is a "field projection", and `T::Item` is an ["associated type projection"](./traits-goals-and-clauses.html#trait-ref) promoted constants | constants extracted from a function and lifted to static scope; see [this section](./mir.html#promoted) for more details. provider | the function that executes a query ([see more](query.html)) quantified | in math or logic, existential and universal quantification are used to ask questions like "is there any type T for which is true?" or "is this true for all types T?"; see [the background chapter for more](./background.html#quantified) @@ -49,6 +51,7 @@ span | a location in the user's source code, used for error substs | the substitutions for a given generic type or item (e.g. the `i32`, `u32` in `HashMap`) tcx | the "typing context", main data structure of the compiler ([see more](ty.html)) 'tcx | the lifetime of the currently active inference context ([see more](ty.html)) +trait reference | the name of a trait along with a suitable set of input type/lifetimes ([see more](./traits-goals-and-clauses.html#trait-ref)) token | the smallest unit of parsing. Tokens are produced after lexing ([see more](the-parser.html)). [TLS] | Thread-Local Storage. Variables may be defined so that each thread has its own copy (rather than all threads sharing the variable). This has some interactions with LLVM. Not all platforms support TLS. trans | the code to translate MIR into LLVM IR. diff --git a/src/traits-associated-types.md b/src/traits-associated-types.md index a640bf032..6ac3fc512 100644 --- a/src/traits-associated-types.md +++ b/src/traits-associated-types.md @@ -20,6 +20,8 @@ that syntax is expanded during ["type collection"](./type-checking.html) into the explicit form, though that is something we may want to change in the future.) + + In some cases, associated type projections can be **normalized** -- that is, simplified -- based on the types given in an impl. So, to continue with our example, the impl of `IntoIterator` for `Option` diff --git a/src/traits-goals-and-clauses.md b/src/traits-goals-and-clauses.md index ad1655726..92e1b67d9 100644 --- a/src/traits-goals-and-clauses.md +++ b/src/traits-goals-and-clauses.md @@ -12,12 +12,14 @@ a few new superpowers. In Rust's solver, **goals** and **clauses** have the following forms (note that the two definitions reference one another): - Goal = DomainGoal + Goal = DomainGoal // defined in the section below | Goal && Goal | Goal || Goal | exists { Goal } // existential quantification | forall { Goal } // universal quantification | if (Clause) { Goal } // implication + | true // something that's trivially true + | ambiguous // something that's never provable Clause = DomainGoal | Clause :- Goal // if can prove Goal, then Clause is true @@ -39,9 +41,11 @@ gives the details. ## Domain goals + + To define the set of *domain goals* in our system, we need to first introduce a few simple formulations. A **trait reference** consists of -the name of a trait allow with a suitable set of inputs P0..Pn: +the name of a trait along with a suitable set of inputs P0..Pn: TraitRef = P0: TraitName @@ -50,7 +54,9 @@ IntoIterator`. Note that Rust surface syntax also permits some extra things, like associated type bindings (`Vec: IntoIterator`), that are not part of a trait reference. -A **projection** consists of a an associated item reference along with + + +A **projection** consists of an associated item reference along with its inputs P0..Pm: Projection = >::AssocItem @@ -77,7 +83,9 @@ Given that, we can define a `DomainGoal` as follows: to [implied bounds]. - `ProjectionEq(Projection = Type)` -- the given associated type `Projection` is equal to `Type`; see [the section on associated types](./traits-associated-types.html) -- `Normalize(Projection -> Type)` -- the given associated type `Projection` can be normalized + - in general, proving `ProjectionEq(TraitRef::Item = Type)` also + requires proving `Implemented(TraitRef)` +- `Normalize(Projection -> Type)` -- the given associated type `Projection` can be [normalized][n] to `Type` - as discussed in [the section on associated types](./traits-associated-types.html), `Normalize` implies `ProjectionEq` but not vice versa @@ -85,6 +93,8 @@ Given that, we can define a `DomainGoal` as follows: *well-formed* - well-formedness is important to [implied bounds]. +[n]: ./traits-associated-types.html#normalize + ## Coinductive goals diff --git a/src/traits-lowering-rules.md b/src/traits-lowering-rules.md index d096b4364..46827dce5 100644 --- a/src/traits-lowering-rules.md +++ b/src/traits-lowering-rules.md @@ -1,8 +1,8 @@ # Lowering rules This section gives the complete lowering rules for Rust traits into -[program clauses][pc]. These rules reference the [domain goals][dg] defined in -an earlier section. +[program clauses][pc]. It is a kind of reference. These rules +reference the [domain goals][dg] defined in an earlier section. [pc]: ./traits-goals-and-clauses.html [dg]: ./traits-goals-and-clauses.html#domain-goals @@ -22,7 +22,7 @@ definitions; these macros reference other sections within this chapter. ## Lowering where clauses -When used in a goal position, where clauses can be mapped directly +When used in a goal position, where clauses can be mapped directly to [domain goals][dg], as follows: - `A0: Foo` maps to `Implemented(A0: Foo)`. @@ -72,12 +72,12 @@ inside the `{}`. ### Trait header From the trait itself we mostly make "meta" rules that setup the -relationships between different kinds of domain goals. The first dush +relationships between different kinds of domain goals. The first such rule from the trait header creates the mapping between the `FromEnv` and `Implemented` predicates: forall { - Implemented(Self: Trait) :- FromEnv(Self: Trait) :- FromEnv(Self: Trait) } @@ -114,7 +114,7 @@ to be **well-formed**: // For each where clause WC: forall { - WellFormed(Self: Trait) :- Implemented(Self: Trait), WellFormed(WC) + WellFormed(Self: Trait) :- Implemented(Self: Trait) && WellFormed(WC) } This `WellFormed` rule states that `T: Trait` is well-formed if (a) @@ -150,7 +150,7 @@ one of those: This `WellFormed` predicate is only used when proving that impls are well-formed -- basically, for each impl of some trait ref `TraitRef`, -we must that `WellFormed(TraitRef)`. This in turn justifies the +we must show that `WellFormed(TraitRef)`. This in turn justifies the implied bounds rules that allow us to extend the set of `FromEnv` items. @@ -170,9 +170,10 @@ where WC } ``` -We will produce a number of program clases. The first two define +We will produce a number of program clauses. The first two define the rules by which `ProjectionEq` can succeed; these two clauses are discussed -in detail in the [section on associated types](./traits-associated-types.html). +in detail in the [section on associated types](./traits-associated-types.html),, +but reproduced here for reference: // ProjectionEq can succeed by normalizing: forall { @@ -180,7 +181,8 @@ in detail in the [section on associated types](./traits-associated-types.html). Normalize(>::AssocType -> U) } - // ProjectionEq can succeed by skolemizing: + // ProjectionEq can succeed by skolemizing, see "associated type" + // chapter for more: forall { ProjectionEq( >::AssocType = @@ -188,13 +190,13 @@ in detail in the [section on associated types](./traits-associated-types.html). ) :- // But only if the trait is implemented, and the conditions from // the associated type are met as well: - Implemented(Self: Trait), - WC1 + Implemented(Self: Trait) + && WC1 } The next rule covers implied bounds for the projection. In particular, the `Bounds` declared on the associated type must be proven to hold to -show that the impl is well-formed, and hence we can rely on them from +show that the impl is well-formed, and hence we can rely on them elsewhere. // XXX how exactly should we set this up? Have to be careful; @@ -237,7 +239,7 @@ Given an impl that contains: impl Trait for A0 where WC { - type AssocType: Bounds where WC1 = T; + type AssocType where WC1 = T; } ``` @@ -246,13 +248,12 @@ We produce the following rule: forall { forall { Normalize(>::AssocType -> T) :- - WC, WC1 + WC && WC1 } } Note that `WC` and `WC1` both encode where-clauses that the impl can -rely on, whereas the bounds `Bounds` on the associated type are things -that the impl must prove (see the well-formedness checking). +rely on. diff --git a/src/traits-lowering-to-logic.md b/src/traits-lowering-to-logic.md index 7b1c2ed6b..4e4e7cae9 100644 --- a/src/traits-lowering-to-logic.md +++ b/src/traits-lowering-to-logic.md @@ -33,6 +33,9 @@ Prolog-like notation, as follows: ``` Clone(usize). Clone(Vec) :- Clone(?T). + +// The notation `A :- B` means "A is true if B is true". +// Or, put another way, B implies A. ``` In Prolog terms, we might say that `Clone(Foo)` -- where `Foo` is some @@ -82,20 +85,20 @@ let's turn our focus a bit towards **type-checking**. Type-checking is interesting because it is what gives us the goals that we need to prove. That is, everything we've seen so far has been about how we derive the rules by which we can prove goals from the traits and impls -in the program; but we are also interesting in how derive the goals +in the program; but we are also interested in how to derive the goals that we need to prove, and those come from type-checking. Consider type-checking the function `foo()` here: ```rust fn foo() { bar::() } -fn bar() { } +fn bar>() { } ``` This function is very simple, of course: all it does is to call `bar::()`. Now, looking at the definition of `bar()`, we can see -that it has one where-clause `U: Eq`. So, that means that `foo()` will -have to prove that `usize: Eq` in order to show that it can call `bar()` +that it has one where-clause `U: Eq`. So, that means that `foo()` will +have to prove that `usize: Eq` in order to show that it can call `bar()` with `usize` as the type argument. If we wanted, we could write a Prolog predicate that defines the @@ -103,7 +106,7 @@ conditions under which `bar()` can be called. We'll say that those conditions are called being "well-formed": ``` -barWellFormed(?U) :- Eq(?U). +barWellFormed(?U) :- Eq(?U, ?U). ``` Then we can say that `foo()` type-checks if the reference to @@ -118,7 +121,7 @@ If we try to prove the goal `fooTypeChecks`, it will succeed: - `fooTypeChecks` is provable if: - `barWellFormed(usize)`, which is provable if: - - `Eq(usize)`, which is provable because of an impl. + - `Eq(usize, usize)`, which is provable because of an impl. Ok, so far so good. Let's move on to type-checking a more complex function. @@ -132,8 +135,8 @@ can be provide. To see what I'm talking about, let's revamp our previous example to make `foo` generic: ```rust -fn foo() { bar::() } -fn bar() { } +fn foo>() { bar::() } +fn bar>() { } ``` To type-check the body of `foo`, we need to be able to hold the type @@ -145,8 +148,8 @@ this like so: fooTypeChecks :- // for all types T... forall { - // ...if we assume that Eq(T) is provable... - if (Eq(T)) { + // ...if we assume that Eq(T, T) is provable... + if (Eq(T, T)) { // ...then we can prove that `barWellFormed(T)` holds. barWellFormed(T) } From 7a78a998cea262950d04f291618ba7ea7540d517 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 10 Mar 2018 07:06:36 -0500 Subject: [PATCH 137/648] add implied bounds placeholder --- src/SUMMARY.md | 1 + src/traits-implied-bounds.md | 9 +++++++++ 2 files changed, 10 insertions(+) create mode 100644 src/traits-implied-bounds.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 6612372be..8aa450b7b 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -25,6 +25,7 @@ - [Lowering to logic](./traits-lowering-to-logic.md) - [Goals and clauses](./traits-goals-and-clauses.md) - [Equality and associated types](./traits-associated-types.md) + - [Implied bounds](./traits-implied-bounds.md) - [Region constraints](./traits-regions.md) - [Canonical queries](./traits-canonical-queries.md) - [Canonicalization](./traits-canonicalization.md) diff --git a/src/traits-implied-bounds.md b/src/traits-implied-bounds.md new file mode 100644 index 000000000..26a63bf32 --- /dev/null +++ b/src/traits-implied-bounds.md @@ -0,0 +1,9 @@ +# Implied Bounds + +*to be written* + +Cover: + +- Why the `FromEnv` setup etc is the way it is +- Perhaps move some of the material from 'lowering rules' in to here +- Show various examples where you could go wrong From 994a38b19802a8982f413ba6c769c4ccecb80888 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 10 Mar 2018 07:09:25 -0500 Subject: [PATCH 138/648] update the summary landing page --- src/traits.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/traits.md b/src/traits.md index 6994f33d2..b75e7fec6 100644 --- a/src/traits.md +++ b/src/traits.md @@ -16,8 +16,10 @@ Trait solving is based around a few key ideas: - [Lowering to logic](./traits-lowering-to-logic.html), which expresses Rust traits in terms of standard logical terms. - The [goals and clauses](./traits-goals-and-clauses.html) chapter - describes the precise lowering rules we use. -- [Canonical queries](./traits-canonicalization.html), which allow us + describes the precise form of rules we use, and + [lowering rules](./lowering-rules.html) gives the complete set of + lowering rules in a more reference-like form. +- [Canonical queries](./traits-canonical-queries.html), which allow us to solve trait problems (like "is `Foo` implemented for the type `Bar`?") once, and then apply that same result independently in many different inference contexts. From d9f0a45a045c7901cf2c36b4134f5f96db5e959c Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 10 Mar 2018 07:10:44 -0500 Subject: [PATCH 139/648] fix link --- src/traits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/traits.md b/src/traits.md index b75e7fec6..175e6418b 100644 --- a/src/traits.md +++ b/src/traits.md @@ -17,7 +17,7 @@ Trait solving is based around a few key ideas: Rust traits in terms of standard logical terms. - The [goals and clauses](./traits-goals-and-clauses.html) chapter describes the precise form of rules we use, and - [lowering rules](./lowering-rules.html) gives the complete set of + [lowering rules](./traits-lowering-rules.html) gives the complete set of lowering rules in a more reference-like form. - [Canonical queries](./traits-canonical-queries.html), which allow us to solve trait problems (like "is `Foo` implemented for the type From 49e8e092c12a6bf97ec240eb65f8d6713e4088a4 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Sat, 10 Mar 2018 07:11:11 -0500 Subject: [PATCH 140/648] trailing whitespace --- src/SUMMARY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 8aa450b7b..d4ee59abf 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -21,7 +21,7 @@ - [Higher-ranked trait bounds](./trait-hrtb.md) - [Caching subtleties](./trait-caching.md) - [Specialization](./trait-specialization.md) -- [Trait solving (new-style)](./traits.md) +- [Trait solving (new-style)](./traits.md) - [Lowering to logic](./traits-lowering-to-logic.md) - [Goals and clauses](./traits-goals-and-clauses.md) - [Equality and associated types](./traits-associated-types.md) From 5dfe4f26c5a4e2ad56ff4c52a9d3dadeef8a9430 Mon Sep 17 00:00:00 2001 From: Shanavas M Date: Sat, 10 Mar 2018 18:41:11 +0300 Subject: [PATCH 141/648] fix typo --- src/tests/intro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index 8ad2492b7..eb7294c83 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -27,7 +27,7 @@ that give more details. - [`ui`](./tests/adding.html#ui) -- tests that check the exact stdout/stderr from compilation and/or running the test - `run-pass` -- tests that are expected to compile and execute successfully (no panics) - - `run-pass-valgrind` -- tests that ought to run with valrind + - `run-pass-valgrind` -- tests that ought to run with valgrind - `run-fail` -- tests that are expected to compile but then panic during execution - `compile-fail` -- tests that are expected to fail compilation. - `parse-fail` -- tests that are expected to fail to parse From ce45556483b70edf882417060f079f79e02309f9 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sun, 11 Mar 2018 21:05:29 -0500 Subject: [PATCH 142/648] Fix a few typos --- src/traits-canonical-queries.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/src/traits-canonical-queries.md b/src/traits-canonical-queries.md index ee66a73db..3fea5fbd9 100644 --- a/src/traits-canonical-queries.md +++ b/src/traits-canonical-queries.md @@ -20,7 +20,7 @@ will run off and start supplying you with every possible answer it can find. So given something like this: ?- Vec: AsRef - + The solver might answer: Vec: AsRef<[i32]> @@ -71,27 +71,27 @@ Another interesting thing is that queries might still have variables in them. For example: ?- Rc: Clone - + might produce the answer: Rc: Clone continue? (y/n) - -After all, `Rc` is true **no matter what type `?T` is**. + +After all, `Rc` is true **no matter what type `?T` is**. + + ## A trait query in rustc The trait queries in rustc work somewhat differently. Instead of trying to enumerate **all possible** answers for you, they are looking -for an **unambiguous** answer. In particular, when they tells you the +for an **unambiguous** answer. In particular, when they tell you the value for a type variable, that means that this is the **only possible instantiation** that you could use, given the current set of impls and where-clauses, that would be provable. (Internally within the solver, though, they can potentially enumerate all possible answers. See [the description of the SLG solver](./traits-slg.html) for details.) - - The response to a trait query in rustc is typically a `Result, NoSolution>` (where the `T` will vary a bit depending on the query itself). The `Err(NoSolution)` case indicates @@ -191,7 +191,7 @@ fn main() { let mut t: Vec<_> = vec![]; // Type: Vec let mut u: Option<_> = None; // Type: Option foo(t, u); // `Vec: Borrow` => ambiguous - + // New stuff: u = Some(vec![]); // ?U = Vec } @@ -206,10 +206,10 @@ vector. This in turn implies that `?U` is [unified] to `Vec`. Let's suppose that the type checker decides to revisit the "as-yet-unproven" trait obligation we saw before, `Vec: Borrow`. `?U` is no longer an unbound inference variable; it now -has a value, &. So, if we "refresh" the query with that value, we get: +has a value, `Vec`. So, if we "refresh" the query with that value, we get: Vec: Borrow> - + This time, there is only one impl that applies, the reflexive impl: impl Borrow for T where T: ?Sized From 0da5774acb75498d34b0f7df0e489bd99f2e1ea7 Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Wed, 7 Mar 2018 22:18:05 +0800 Subject: [PATCH 143/648] Added a very rough rustc-driver chapter --- src/SUMMARY.md | 1 + src/rustc-driver.md | 66 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+) create mode 100644 src/rustc-driver.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index d4ee59abf..841a9b491 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -9,6 +9,7 @@ - [Using `compiletest` + commands to control test execution](./compiletest.md) - [Walkthrough: a typical contribution](./walkthrough.md) - [High-level overview of the compiler source](./high-level-overview.md) +- [The Rustc Driver](./rustc-driver.md) - [Queries: demand-driven compilation](./query.md) - [Incremental compilation](./incremental-compilation.md) - [The parser](./the-parser.md) diff --git a/src/rustc-driver.md b/src/rustc-driver.md new file mode 100644 index 000000000..b21cc3662 --- /dev/null +++ b/src/rustc-driver.md @@ -0,0 +1,66 @@ +# The Rustc Driver + +The [`rustc_driver`] is essentially `rustc`'s `main()` function. It acts as +the glue for running the various phases of the compiler in the correct order, +managing state such as the [`CodeMap`] \(maps AST nodes to source code), +[`Session`] \(general build context and error messaging) and the [`TyCtxt`] +\(the "typing context", allowing you to query the type system and other cool +stuff). The `rustc_driver` crate also provides external users with a method +for running code at particular times during the compilation process, allowing +third parties to effectively use `rustc`'s internals as a library for +analysing a crate. + +For those using `rustc` as a library, the `run_compiler()` function is the main +entrypoint to the compiler. Its main parameters are a list of command-line +arguments and a reference to something which implements the `CompilerCalls` +trait. A `CompilerCalls` creates the overall `CompileController`, letting it +govern which compiler passes are run and attach callbacks to be fired at the end +of each phase. + +From `rustc_driver`'s perspective, the main phases of the compiler are: + +1. *Parse Input:* Initial crate parsing +2. *Configure and Expand:* Resolve `#[cfg]` attributes and expand macros +3. *Run Analysis Passes:* Run the resolution, typechecking, region checking + and other miscellaneous analysis passes on the crate +4. *Translate to LLVM:* Turn the analysed program into executable code + +The `CompileController` then gives users the ability to inspect the ongoing +compilation process + +- after parsing +- after AST expansion +- after HIR lowering +- after analysis, and +- when compilation is done + +The `CompileState`'s various `state_after_*()` constructors can be inspected to +determine what bits of information are available to which callback. + +## A Note On Lifetimes + +The Rust compiler is a fairly large program containing lots of big data +structures (e.g. the AST, HIR, and the type system) and as such, arenas and +references are heavily relied upon to minimize unnecessary memory use. This +manifests itself in the way people can plug into the compiler, preferring a +"push"-style API (callbacks) instead of the more Rust-ic "pull" style (think +the `Iterator` trait). + +For example the [`CompileState`], the state passed to callbacks after each +phase, is essentially just a box of optional references to pieces inside the +compiler. The lifetime bound on the `CompilerCalls` trait then helps to ensure +compiler internals don't "escape" the compiler (e.g. if you tried to keep a +reference to the AST after the compiler is finished), while still letting users +record *some* state for use after the `run_compiler()` function finishes. + +Thread-local storage and interning are used a lot through the compiler to reduce +duplication while also preventing a lot of the ergonomic issues due to many +pervasive lifetimes. The `rustc::ty::tls` module is used to access these +thread-locals, although you should rarely need to touch it. + + +[`rustc_driver`]: https://github.com/rust-lang/rust/tree/master/src/librustc_driver +[`CompileState`]: https://github.com/rust-lang/rust/blob/master/src/librustc_driver/driver.rs +[`Session`]: https://github.com/rust-lang/rust/blob/master/src/librustc/session/mod.rs +[`TyCtxt`]: https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs +[`CodeMap`]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs \ No newline at end of file From 121d05098c7f4f4fe0813b4f783747e4008c092a Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Wed, 7 Mar 2018 22:31:22 +0800 Subject: [PATCH 144/648] Renamed appendices and added @nrc's guide --- src/SUMMARY.md | 10 +- src/{background.md => appendix-background.md} | 2 +- src/{code-index.md => appendix-code-index.md} | 2 +- src/{glossary.md => appendix-glossary.md} | 17 +- src/appendix-stupid-stats.md | 396 ++++++++++++++++++ src/mir-borrowck.md | 2 +- src/mir-regionck.md | 8 +- src/mir.md | 2 +- 8 files changed, 419 insertions(+), 20 deletions(-) rename src/{background.md => appendix-background.md} (99%) rename src/{code-index.md => appendix-code-index.md} (98%) rename src/{glossary.md => appendix-glossary.md} (94%) create mode 100644 src/appendix-stupid-stats.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 841a9b491..eb889d36b 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -46,6 +46,10 @@ - [miri const evaluator](./miri.md) - [Parameter Environments](./param_env.md) - [Generating LLVM IR](./trans.md) -- [Background material](./background.md) -- [Glossary](./glossary.md) -- [Code Index](./code-index.md) + +--- + +- [Appendix A: Stupid Stats](./appendix-stupid-stats.md) +- [Appendix B: Background material](./appendix-background.md) +- [Appendix C: Glossary](./appendix-glossary.md) +- [Appendix D: Code Index](./appendix-code-index.md) diff --git a/src/background.md b/src/appendix-background.md similarity index 99% rename from src/background.md rename to src/appendix-background.md index 50c247774..c69e7d93d 100644 --- a/src/background.md +++ b/src/appendix-background.md @@ -1,4 +1,4 @@ -# Background topics +# Appendix B: Background topics This section covers a numbers of common compiler terms that arise in this guide. We try to give the general definition while providing some diff --git a/src/code-index.md b/src/appendix-code-index.md similarity index 98% rename from src/code-index.md rename to src/appendix-code-index.md index 6a500abba..64a40f74d 100644 --- a/src/code-index.md +++ b/src/appendix-code-index.md @@ -1,4 +1,4 @@ -# Code Index +# Appendix D: Code Index rustc has a lot of important data structures. This is an attempt to give some guidance on where to learn more about some of the key data structures of the diff --git a/src/glossary.md b/src/appendix-glossary.md similarity index 94% rename from src/glossary.md rename to src/appendix-glossary.md index e542d4e35..1914adec6 100644 --- a/src/glossary.md +++ b/src/appendix-glossary.md @@ -1,21 +1,20 @@ -Glossary --------- +# Appendix C: Glossary The compiler uses a number of...idiosyncratic abbreviations and things. This glossary attempts to list them and give you a few pointers for understanding them better. Term | Meaning ------------------------|-------- AST | the abstract syntax tree produced by the syntax crate; reflects user syntax very closely. -binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and \|`a`\|` ...` is a binder for the parameter `a`. See [the background chapter for more](./background.html#free-vs-bound) -bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession \|`a`\|` a * 2`. See [the background chapter for more](./background.html#free-vs-bound) +binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and \|`a`\|` ...` is a binder for the parameter `a`. See [the background chapter for more](./appendix-background.html#free-vs-bound) +bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession \|`a`\|` a * 2`. See [the background chapter for more](./appendix-background.html#free-vs-bound) codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. completeness | completeness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). -control-flow graph | a representation of the control-flow of a program; see [the background chapter for more](./background.html#cfg) +control-flow graph | a representation of the control-flow of a program; see [the background chapter for more](./appendix-background.html#cfg) cx | we tend to use "cx" as an abbrevation for context. See also `tcx`, `infcx`, etc. DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](incremental-compilation.html)) -data-flow analysis | a static analysis that figures out what properties are true at each point in the control-flow of a program; see [the background chapter for more](./background.html#dataflow) +data-flow analysis | a static analysis that figures out what properties are true at each point in the control-flow of a program; see [the background chapter for more](./appendix-background.html#dataflow) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. -free variable | a "free variable" is one that is not bound within an expression or term; see [the background chapter for more](./background.html#free-vs-bound) +free variable | a "free variable" is one that is not bound within an expression or term; see [the background chapter for more](./appendix-background.html#free-vs-bound) 'gcx | the lifetime of the global arena ([see more](ty.html)) generics | the set of generic type parameters defined on a type or item HIR | the High-level IR, created by lowering and desugaring the AST ([see more](hir.html)) @@ -39,7 +38,7 @@ obligation | something that must be proven by the trait system ([s projection | a general term for a "relative path", e.g. `x.f` is a "field projection", and `T::Item` is an ["associated type projection"](./traits-goals-and-clauses.html#trait-ref) promoted constants | constants extracted from a function and lifted to static scope; see [this section](./mir.html#promoted) for more details. provider | the function that executes a query ([see more](query.html)) -quantified | in math or logic, existential and universal quantification are used to ask questions like "is there any type T for which is true?" or "is this true for all types T?"; see [the background chapter for more](./background.html#quantified) +quantified | in math or logic, existential and universal quantification are used to ask questions like "is there any type T for which is true?" or "is this true for all types T?"; see [the background chapter for more](./appendix-background.html#quantified) query | perhaps some sub-computation during compilation ([see more](query.html)) region | another term for "lifetime" often used in the literature and in the borrow checker. sess | the compiler session, which stores global data used throughout compilation @@ -57,7 +56,7 @@ token | the smallest unit of parsing. Tokens are produced aft trans | the code to translate MIR into LLVM IR. trait reference | a trait and values for its type parameters ([see more](ty.html)). ty | the internal representation of a type ([see more](ty.html)). -variance | variance determines how changes to a generic type/lifetime parameter affect subtyping; for example, if `T` is a subtype of `U`, then `Vec` is a subtype `Vec` because `Vec` is *covariant* in its generic parameter. See [the background chapter for more](./background.html#variance). +variance | variance determines how changes to a generic type/lifetime parameter affect subtyping; for example, if `T` is a subtype of `U`, then `Vec` is a subtype `Vec` because `Vec` is *covariant* in its generic parameter. See [the background chapter for more](./appendix-background.html#variance). [LLVM]: https://llvm.org/ [lto]: https://llvm.org/docs/LinkTimeOptimization.html diff --git a/src/appendix-stupid-stats.md b/src/appendix-stupid-stats.md new file mode 100644 index 000000000..405577e3c --- /dev/null +++ b/src/appendix-stupid-stats.md @@ -0,0 +1,396 @@ +# Appendix A: A tutorial on creating a drop-in replacement for rustc + +Many tools benefit from being a drop-in replacement for a compiler. By this, I +mean that any user of the tool can use `mytool` in all the ways they would +normally use `rustc` - whether manually compiling a single file or as part of a +complex make project or Cargo build, etc. That could be a lot of work; +rustc, like most compilers, takes a large number of command line arguments which +can affect compilation in complex and interacting ways. Emulating all of this +behaviour in your tool is annoying at best, especically if you are making many +of the same calls into librustc that the compiler is. + +The kind of things I have in mind are tools like rustdoc or a future rustfmt. +These want to operate as closely as possible to real compilation, but have +totally different outputs (documentation and formatted source code, +respectively). Another use case is a customised compiler. Say you want to add a +custom code generation phase after macro expansion, then creating a new tool +should be easier than forking the compiler (and keeping it up to date as the +compiler evolves). + +I have gradually been trying to improve the API of librustc to make creating a +drop-in tool easier to produce (many others have also helped improve these +interfaces over the same time frame). It is now pretty simple to make a tool +which is as close to rustc as you want it to be. In this tutorial I'll show +how. + +Note/warning, everything I talk about in this tutorial is internal API for +rustc. It is all extremely unstable and likely to change often and in +unpredictable ways. Maintaining a tool which uses these APIs will be non- +trivial, although hopefully easier than maintaining one that does similar things +without using them. + +This tutorial starts with a very high level view of the rustc compilation +process and of some of the code that drives compilation. Then I'll describe how +that process can be customised. In the final section of the tutorial, I'll go +through an example - stupid-stats - which shows how to build a drop-in tool. + + +## Overview of the compilation process + +Compilation using rustc happens in several phases. We start with parsing, this +includes lexing. The output of this phase is an AST (abstract syntax tree). +There is a single AST for each crate (indeed, the entire compilation process +operates over a single crate). Parsing abstracts away details about individual +files which will all have been read in to the AST in this phase. At this stage +the AST includes all macro uses, attributes will still be present, and nothing +will have been eliminated due to `cfg`s. + +The next phase is configuration and macro expansion. This can be thought of as a +function over the AST. The unexpanded AST goes in and an expanded AST comes out. +Macros and syntax extensions are expanded, and `cfg` attributes will cause some +code to disappear. The resulting AST won't have any macros or macro uses left +in. + +The code for these first two phases is in [libsyntax](https://github.com/rust-lang/rust/tree/master/src/libsyntax). + +After this phase, the compiler allocates ids to each node in the AST +(technically not every node, but most of them). If we are writing out +dependencies, that happens now. + +The next big phase is analysis. This is the most complex phase and +uses the bulk of the code in rustc. This includes name resolution, type +checking, borrow checking, type and lifetime inference, trait selection, method +selection, linting, and so forth. Most error detection is done in this phase +(although parse errors are found during parsing). The 'output' of this phase is +a bunch of side tables containing semantic information about the source program. +The analysis code is in [librustc](https://github.com/rust-lang/rust/tree/master/src/librustc) +and a bunch of other crates with the 'librustc_' prefix. + +Next is translation, this translates the AST (and all those side tables) into +LLVM IR (intermediate representation). We do this by calling into the LLVM +libraries, rather than actually writing IR directly to a file. The code for this is in +[librustc_trans](https://github.com/rust-lang/rust/tree/master/src/librustc_trans). + +The next phase is running the LLVM backend. This runs LLVM's optimisation passes +on the generated IR and then generates machine code. The result is object files. +This phase is all done by LLVM, it is not really part of the rust compiler. The +interface between LLVM and rustc is in [librustc_llvm](https://github.com/rust-lang/rust/tree/master/src/librustc_llvm). + +Finally, we link the object files into an executable. Again we outsource this to +other programs and it's not really part of the rust compiler. The interface is +in [librustc_back](https://github.com/rust-lang/rust/tree/master/src/librustc_back) +(which also contains some things used primarily during translation). + +All these phases are coordinated by the driver. To see the exact sequence, look +at the `compile_input` function in [librustc_driver/driver.rs](https://github.com/rust-lang/rust/tree/master/src/librustc_driver/driver.rs). +The driver (which is found in [librust_driver](https://github.com/rust-lang/rust/tree/master/src/librustc_driver)) +handles all the highest level coordination of compilation - handling command +line arguments, maintaining compilation state (primarily in the `Session`), and +calling the appropriate code to run each phase of compilation. It also handles +high level coordination of pretty printing and testing. To create a drop-in +compiler replacement or a compiler replacement, we leave most of compilation +alone and customise the driver using its APIs. + + +## The driver customisation APIs + +There are two primary ways to customise compilation - high level control of the +driver using `CompilerCalls` and controlling each phase of compilation using a +`CompileController`. The former lets you customise handling of command line +arguments etc., the latter lets you stop compilation early or execute code +between phases. + + +### `CompilerCalls` + +`CompilerCalls` is a trait that you implement in your tool. It contains a fairly +ad-hoc set of methods to hook in to the process of processing command line +arguments and driving the compiler. For details, see the comments in +[librustc_driver/lib.rs](https://github.com/rust-lang/rust/tree/master/src/librustc_driver/lib.rs). +I'll summarise the methods here. + +`early_callback` and `late_callback` let you call arbitrary code at different +points - early is after command line arguments have been parsed, but before +anything is done with them; late is pretty much the last thing before +compilation starts, i.e., after all processing of command line arguments, etc. is +done. Currently, you get to choose whether compilation stops or continues at +each point, but you don't get to change anything the driver has done. You can +record some info for later, or perform other actions of your own. + +`some_input` and `no_input` give you an opportunity to modify the primary input +to the compiler (usually the input is a file containing the top module for a +crate, but it could also be a string). You could record the input or perform +other actions of your own. + +Ignore `parse_pretty`, it is unfortunate and hopefully will get improved. There +is a default implementation, so you can pretend it doesn't exist. + +`build_controller` returns a `CompileController` object for more fine-grained +control of compilation, it is described next. + +We might add more options in the future. + + +### `CompilerController` + +`CompilerController` is a struct consisting of `PhaseController`s and flags. +Currently, there is only flag, `make_glob_map` which signals whether to produce +a map of glob imports (used by save-analysis and potentially other tools). There +are probably flags in the session that should be moved here. + +There is a `PhaseController` for each of the phases described in the above +summary of compilation (and we could add more in the future for finer-grained +control). They are all `after_` a phase because they are checked at the end of a +phase (again, that might change), e.g., `CompilerController::after_parse` +controls what happens immediately after parsing (and before macro expansion). + +Each `PhaseController` contains a flag called `stop` which indicates whether +compilation should stop or continue, and a callback to be executed at the point +indicated by the phase. The callback is called whether or not compilation +continues. + +Information about the state of compilation is passed to these callbacks in a +`CompileState` object. This contains all the information the compiler has. Note +that this state information is immutable - your callback can only execute code +using the compiler state, it can't modify the state. (If there is demand, we +could change that). The state available to a callback depends on where during +compilation the callback is called. For example, after parsing there is an AST +but no semantic analysis (because the AST has not been analysed yet). After +translation, there is translation info, but no AST or analysis info (since these +have been consumed/forgotten). + + +## An example - stupid-stats + +Our example tool is very simple, it simply collects some simple and not very +useful statistics about a program; it is called stupid-stats. You can find +the (more heavily commented) complete source for the example on [Github](https://github.com/nick29581/stupid-stats/blob/master/src). +To build, just do `cargo build`. To run on a file `foo.rs`, do `cargo run +foo.rs` (assuming you have a Rust program called `foo.rs`. You can also pass any +command line arguments that you would normally pass to rustc). When you run it +you'll see output similar to + +``` +In crate: foo, + +Found 12 uses of `println!`; +The most common number of arguments is 1 (67% of all functions); +25% of functions have four or more arguments. +``` + +To make things easier, when we talk about functions, we're excluding methods and +closures. + +You can also use the executable as a drop-in replacement for rustc, because +after all, that is the whole point of this exercise. So, however you use rustc +in your makefile setup, you can use `target/stupid` (or whatever executable you +end up with) instead. That might mean setting an environment variable or it +might mean renaming your executable to `rustc` and setting your PATH. Similarly, +if you're using Cargo, you'll need to rename the executable to rustc and set the +PATH. Alternatively, you should be able to use +[multirust](https://github.com/brson/multirust) to get around all the PATH stuff +(although I haven't actually tried that). + +(Note that this example prints to stdout. I'm not entirely sure what Cargo does +with stdout from rustc under different circumstances. If you don't see any +output, try inserting a `panic!` after the `println!`s to error out, then Cargo +should dump stupid-stats' stdout to Cargo's stdout). + +Let's start with the `main` function for our tool, it is pretty simple: + +``` +fn main() { + let args: Vec<_> = std::env::args().collect(); + rustc_driver::run_compiler(&args, &mut StupidCalls::new()); + std::env::set_exit_status(0); +} +``` + +The first line grabs any command line arguments. The second line calls the +compiler driver with those arguments. The final line sets the exit code for the +program. + +The only interesting thing is the `StupidCalls` object we pass to the driver. +This is our implementation of the `CompilerCalls` trait and is what will make +this tool different from rustc. + +`StupidCalls` is a mostly empty struct: + +``` +struct StupidCalls { + default_calls: RustcDefaultCalls, +} +``` + +This tool is so simple that it doesn't need to store any data here, but usually +you would. We embed a `RustcDefaultCalls` object to delegate to in our impl when +we want exactly the same behaviour as the Rust compiler. Mostly you don't want +to do that (or at least don't need to) in a tool. However, Cargo calls rustc +with the `--print file-names`, so we delegate in `late_callback` and `no_input` +to keep Cargo happy. + +Most of the rest of the impl of `CompilerCalls` is trivial: + +``` +impl<'a> CompilerCalls<'a> for StupidCalls { + fn early_callback(&mut self, + _: &getopts::Matches, + _: &config::Options, + _: &diagnostics::registry::Registry, + _: ErrorOutputType) + -> Compilation { + Compilation::Continue + } + + fn late_callback(&mut self, + m: &getopts::Matches, + s: &Session, + i: &Input, + odir: &Option, + ofile: &Option) + -> Compilation { + self.default_calls.late_callback(m, s, i, odir, ofile); + Compilation::Continue + } + + fn some_input(&mut self, + input: Input, + input_path: Option) + -> (Input, Option) { + (input, input_path) + } + + fn no_input(&mut self, + m: &getopts::Matches, + o: &config::Options, + odir: &Option, + ofile: &Option, + r: &diagnostics::registry::Registry) + -> Option<(Input, Option)> { + self.default_calls.no_input(m, o, odir, ofile, r); + + // This is not optimal error handling. + panic!("No input supplied to stupid-stats"); + } + + fn build_controller(&mut self, _: &Session) -> driver::CompileController<'a> { + ... + } +} +``` + +We don't do anything for either of the callbacks, nor do we change the input if +the user supplies it. If they don't, we just `panic!`, this is the simplest way +to handle the error, but not very user-friendly, a real tool would give a +constructive message or perform a default action. + +In `build_controller` we construct our `CompileController`. We only want to +parse, and we want to inspect macros before expansion, so we make compilation +stop after the first phase (parsing). The callback after that phase is where the +tool does it's actual work by walking the AST. We do that by creating an AST +visitor and making it walk the AST from the top (the crate root). Once we've +walked the crate, we print the stats we've collected: + +``` +fn build_controller(&mut self, _: &Session) -> driver::CompileController<'a> { + // We mostly want to do what rustc does, which is what basic() will return. + let mut control = driver::CompileController::basic(); + // But we only need the AST, so we can stop compilation after parsing. + control.after_parse.stop = Compilation::Stop; + + // And when we stop after parsing we'll call this closure. + // Note that this will give us an AST before macro expansions, which is + // not usually what you want. + control.after_parse.callback = box |state| { + // Which extracts information about the compiled crate... + let krate = state.krate.unwrap(); + + // ...and walks the AST, collecting stats. + let mut visitor = StupidVisitor::new(); + visit::walk_crate(&mut visitor, krate); + + // And finally prints out the stupid stats that we collected. + let cratename = match attr::find_crate_name(&krate.attrs[]) { + Some(name) => name.to_string(), + None => String::from_str("unknown_crate"), + }; + println!("In crate: {},\n", cratename); + println!("Found {} uses of `println!`;", visitor.println_count); + + let (common, common_percent, four_percent) = visitor.compute_arg_stats(); + println!("The most common number of arguments is {} ({:.0}% of all functions);", + common, common_percent); + println!("{:.0}% of functions have four or more arguments.", four_percent); + }; + + control +} +``` + +That is all it takes to create your own drop-in compiler replacement or custom +compiler! For the sake of completeness I'll go over the rest of the stupid-stats +tool. + +``` +struct StupidVisitor { + println_count: usize, + arg_counts: Vec, +} +``` + +The `StupidVisitor` struct just keeps track of the number of `println!`s it has +seen and the count for each number of arguments. It implements +`syntax::visit::Visitor` to walk the AST. Mostly we just use the default +methods, these walk the AST taking no action. We override `visit_item` and +`visit_mac` to implement custom behaviour when we walk into items (items include +functions, modules, traits, structs, and so forth, we're only interested in +functions) and macros: + +``` +impl<'v> visit::Visitor<'v> for StupidVisitor { + fn visit_item(&mut self, i: &'v ast::Item) { + match i.node { + ast::Item_::ItemFn(ref decl, _, _, _, _) => { + // Record the number of args. + self.increment_args(decl.inputs.len()); + } + _ => {} + } + + // Keep walking. + visit::walk_item(self, i) + } + + fn visit_mac(&mut self, mac: &'v ast::Mac) { + // Find its name and check if it is "println". + let ast::Mac_::MacInvocTT(ref path, _, _) = mac.node; + if path_to_string(path) == "println" { + self.println_count += 1; + } + + // Keep walking. + visit::walk_mac(self, mac) + } +} +``` + +The `increment_args` method increments the correct count in +`StupidVisitor::arg_counts`. After we're done walking, `compute_arg_stats` does +some pretty basic maths to come up with the stats we want about arguments. + + +## What next? + +These APIs are pretty new and have a long way to go until they're really good. +If there are improvements you'd like to see or things you'd like to be able to +do, let me know in a comment or [GitHub issue](https://github.com/rust-lang/rust/issues). +In particular, it's not clear to me exactly what extra flexibility is required. +If you have an existing tool that would be suited to this setup, please try it +out and let me know if you have problems. + +It'd be great to see Rustdoc converted to using these APIs, if that is possible +(although long term, I'd prefer to see Rustdoc run on the output from save- +analysis, rather than doing its own analysis). Other parts of the compiler +(e.g., pretty printing, testing) could be refactored to use these APIs +internally (I already changed save-analysis to use `CompilerController`). I've +been experimenting with a prototype rustfmt which also uses these APIs. diff --git a/src/mir-borrowck.md b/src/mir-borrowck.md index 3c10191d4..ab99ac9dc 100644 --- a/src/mir-borrowck.md +++ b/src/mir-borrowck.md @@ -44,7 +44,7 @@ The overall flow of the borrow checker is as follows: Among other things, this function will replace all of the regions in the MIR with fresh [inference variables](glossary.html). - (More details can be found in [the regionck section](./mir-regionck.html).) -- Next, we perform a number of [dataflow analyses](./background.html#dataflow) +- Next, we perform a number of [dataflow analyses](./appendix-background.html#dataflow) that compute what data is moved and when. The results of these analyses are needed to do both borrow checking and region inference. - Using the move data, we can then compute the values of all the regions in the MIR. diff --git a/src/mir-regionck.md b/src/mir-regionck.md index e7b12405a..dbf740ea8 100644 --- a/src/mir-regionck.md +++ b/src/mir-regionck.md @@ -35,7 +35,7 @@ The MIR-based region analysis consists of two major functions: - More details to come, though the [NLL RFC] also includes fairly thorough (and hopefully readable) coverage. -[fvb]: background.html#free-vs-bound +[fvb]: appendix-background.html#free-vs-bound [NLL RFC]: http://rust-lang.github.io/rfcs/2094-nll.html ## Universal regions @@ -129,7 +129,7 @@ are going to wind up with a subtyping relationship like this one: We handle this sort of subtyping by taking the variables that are bound in the supertype and **skolemizing** them: this means that we replace them with -[universally quantified](background.html#quantified) +[universally quantified](appendix-background.html#quantified) representatives, written like `!1`. We call these regions "skolemized regions" -- they represent, basically, "some unknown region". @@ -144,7 +144,7 @@ what we wanted. So let's work through what happens next. To check if two functions are subtypes, we check if their arguments have the desired relationship -(fn arguments are [contravariant](./background.html#variance), so +(fn arguments are [contravariant](./appendix-background.html#variance), so we swap the left and right here): &'!1 u32 <: &'static u32 @@ -181,7 +181,7 @@ Here, the root universe would consist of the lifetimes `'static` and the same concept to types, in which case the types `Foo` and `T` would be in the root universe (along with other global types, like `i32`). Basically, the root universe contains all the names that -[appear free](./background.html#free-vs-bound) in the body of `bar`. +[appear free](./appendix-background.html#free-vs-bound) in the body of `bar`. Now let's extend `bar` a bit by adding a variable `x`: diff --git a/src/mir.md b/src/mir.md index 6e7ac0691..5c4b16310 100644 --- a/src/mir.md +++ b/src/mir.md @@ -26,7 +26,7 @@ Some of the key characteristics of MIR are: - It does not have nested expressions. - All types in MIR are fully explicit. -[cfg]: ./background.html#cfg +[cfg]: ./appendix-background.html#cfg ## Key MIR vocabulary From 53aa601f7fede5d7910a0ffa675f36fba18a0d87 Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Wed, 7 Mar 2018 22:32:45 +0800 Subject: [PATCH 145/648] Thank you link checker! --- src/mir-borrowck.md | 2 +- src/mir.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/mir-borrowck.md b/src/mir-borrowck.md index ab99ac9dc..6c4c99d61 100644 --- a/src/mir-borrowck.md +++ b/src/mir-borrowck.md @@ -42,7 +42,7 @@ The overall flow of the borrow checker is as follows: include references to the new regions that we are computing. - We then invoke `nll::replace_regions_in_mir` to modify this copy C. Among other things, this function will replace all of the regions in - the MIR with fresh [inference variables](glossary.html). + the MIR with fresh [inference variables](./appendix-glossary.html). - (More details can be found in [the regionck section](./mir-regionck.html).) - Next, we perform a number of [dataflow analyses](./appendix-background.html#dataflow) that compute what data is moved and when. The results of these analyses diff --git a/src/mir.md b/src/mir.md index 5c4b16310..688a8750c 100644 --- a/src/mir.md +++ b/src/mir.md @@ -239,4 +239,4 @@ but [you can read about those below](#promoted)). [mir]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir [mirmanip]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir [mir]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir -[newtype'd]: glossary.html +[newtype'd]: appendix-glossary.html From d28ae97736d019dfdddff74dcc8db03a9f19fa67 Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Wed, 7 Mar 2018 22:42:27 +0800 Subject: [PATCH 146/648] Added a couple definitions to the code index --- src/appendix-code-index.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/appendix-code-index.md b/src/appendix-code-index.md index 64a40f74d..2097d61ae 100644 --- a/src/appendix-code-index.md +++ b/src/appendix-code-index.md @@ -7,7 +7,11 @@ compiler. Item | Kind | Short description | Chapter | Declaration ----------------|----------|-----------------------------|--------------------|------------------- `CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser](the-parser.html) | [src/libsyntax/codemap.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs) +`CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver](rustc-driver.html) | [src/librustc_driver/driver.rs](https://github.com/rust-lang/rust/blob/master/src/librustc_driver/driver.rs) +`ast::Crate` | struct | Syntax-level representation of a parsed crate | | [src/librustc/hir/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/ast.rs) +`hir::Crate` | struct | Top-level data structure representing the crate being compiled | | [src/librustc/hir/mod.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/hir/mod.rs) `ParseSess` | struct | This struct contains information about a parsing session | [The parser](the-parser.html) | [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs) `StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser](the-parser.html) | [src/libsyntax/parse/lexer/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/lexer/mod.rs) +`Session` | struct | The data associated with a compilation session | [the Parser](the-parser.html), [The Rustc Driver](rustc-driver.html) | [src/librustc/session/mod.html](https://github.com/rust-lang/rust/blob/master/src/librustc/session/mod.rs) `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules](ty.html) | [src/librustc/ty/trait_def.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/trait_def.rs) `TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules](ty.html) | [src/librustc/ty/context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs) From 27225507d78c35f2b6d0a592bf617c6cb1e10e83 Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Mon, 12 Mar 2018 18:51:08 +0800 Subject: [PATCH 147/648] Addressed some of @nrc and @mark-i-m's comments --- src/appendix-code-index.md | 23 ++++++++++++++--------- src/rustc-driver.md | 13 +++++++++---- 2 files changed, 23 insertions(+), 13 deletions(-) diff --git a/src/appendix-code-index.md b/src/appendix-code-index.md index 2097d61ae..f6dcb9c37 100644 --- a/src/appendix-code-index.md +++ b/src/appendix-code-index.md @@ -6,12 +6,17 @@ compiler. Item | Kind | Short description | Chapter | Declaration ----------------|----------|-----------------------------|--------------------|------------------- -`CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser](the-parser.html) | [src/libsyntax/codemap.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs) -`CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver](rustc-driver.html) | [src/librustc_driver/driver.rs](https://github.com/rust-lang/rust/blob/master/src/librustc_driver/driver.rs) -`ast::Crate` | struct | Syntax-level representation of a parsed crate | | [src/librustc/hir/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/ast.rs) -`hir::Crate` | struct | Top-level data structure representing the crate being compiled | | [src/librustc/hir/mod.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/hir/mod.rs) -`ParseSess` | struct | This struct contains information about a parsing session | [The parser](the-parser.html) | [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs) -`StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser](the-parser.html) | [src/libsyntax/parse/lexer/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/lexer/mod.rs) -`Session` | struct | The data associated with a compilation session | [the Parser](the-parser.html), [The Rustc Driver](rustc-driver.html) | [src/librustc/session/mod.html](https://github.com/rust-lang/rust/blob/master/src/librustc/session/mod.rs) -`TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules](ty.html) | [src/librustc/ty/trait_def.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/trait_def.rs) -`TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules](ty.html) | [src/librustc/ty/context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs) +`CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser] | [src/libsyntax/codemap.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs) +`CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver] | [src/librustc_driver/driver.rs](https://github.com/rust-lang/rust/blob/master/src/librustc_driver/driver.rs) +`ast::Crate` | struct | Syntax-level representation of a parsed crate | [The parser] | [src/librustc/hir/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/ast.rs) +`hir::Crate` | struct | More abstract, compiler-friendly form of a crate's AST | [The Hir] | [src/librustc/hir/mod.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/hir/mod.rs) +`ParseSess` | struct | This struct contains information about a parsing session | [the Parser] | [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs) +`Session` | struct | The data associated with a compilation session | [the Parser], [The Rustc Driver] | [src/librustc/session/mod.html](https://github.com/rust-lang/rust/blob/master/src/librustc/session/mod.rs) +`StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser] | [src/libsyntax/parse/lexer/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/lexer/mod.rs) +`TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules] | [src/librustc/ty/trait_def.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/trait_def.rs) +`TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules] | [src/librustc/ty/context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs) + +[The HIR]: hir.html +[The parser]: the-parser.html +[The Rustc Driver]: rustc-driver.html +[The `ty` modules]: ty.html diff --git a/src/rustc-driver.md b/src/rustc-driver.md index b21cc3662..7e6eb7e2e 100644 --- a/src/rustc-driver.md +++ b/src/rustc-driver.md @@ -8,7 +8,7 @@ managing state such as the [`CodeMap`] \(maps AST nodes to source code), stuff). The `rustc_driver` crate also provides external users with a method for running code at particular times during the compilation process, allowing third parties to effectively use `rustc`'s internals as a library for -analysing a crate. +analysing a crate or emulating the compiler in-process (e.g. the RLS). For those using `rustc` as a library, the `run_compiler()` function is the main entrypoint to the compiler. Its main parameters are a list of command-line @@ -20,10 +20,12 @@ of each phase. From `rustc_driver`'s perspective, the main phases of the compiler are: 1. *Parse Input:* Initial crate parsing -2. *Configure and Expand:* Resolve `#[cfg]` attributes and expand macros -3. *Run Analysis Passes:* Run the resolution, typechecking, region checking +2. *Configure and Expand:* Resolve `#[cfg]` attributes, name resolution, and + expand macros +3. *Run Analysis Passes:* Run trait resolution, typechecking, region checking and other miscellaneous analysis passes on the crate -4. *Translate to LLVM:* Turn the analysed program into executable code +4. *Translate to LLVM:* Translate to the in-memory form of LLVM IR and turn it + into an executable/object files The `CompileController` then gives users the ability to inspect the ongoing compilation process @@ -37,6 +39,9 @@ compilation process The `CompileState`'s various `state_after_*()` constructors can be inspected to determine what bits of information are available to which callback. +> **Warning:** By its very nature, the internal compiler APIs are always going +> to be unstable. That said, we do try not to break things unnecessarily. + ## A Note On Lifetimes The Rust compiler is a fairly large program containing lots of big data From c919a89995e9bd46ebc08ae407ab498a3cde202c Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Mon, 12 Mar 2018 19:04:18 +0800 Subject: [PATCH 148/648] Updated stupid-stats cc: nrc/stupid-stats#8 --- src/appendix-stupid-stats.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/appendix-stupid-stats.md b/src/appendix-stupid-stats.md index 405577e3c..b1c9d2141 100644 --- a/src/appendix-stupid-stats.md +++ b/src/appendix-stupid-stats.md @@ -243,13 +243,15 @@ impl<'a> CompilerCalls<'a> for StupidCalls { } fn late_callback(&mut self, + t: &TransCrate, m: &getopts::Matches, s: &Session, + c: &CrateStore, i: &Input, - odir: &Option, - ofile: &Option) + odir: &Option, + ofile: &Option) -> Compilation { - self.default_calls.late_callback(m, s, i, odir, ofile); + self.default_calls.late_callback(t, m, s, c, i, odir, ofile); Compilation::Continue } @@ -393,4 +395,4 @@ It'd be great to see Rustdoc converted to using these APIs, if that is possible analysis, rather than doing its own analysis). Other parts of the compiler (e.g., pretty printing, testing) could be refactored to use these APIs internally (I already changed save-analysis to use `CompilerController`). I've -been experimenting with a prototype rustfmt which also uses these APIs. +been experimenting with a prototype rustfmt which also uses these APIs. \ No newline at end of file From 746753138a708ee0585018db13d8b2fb6e1f5e62 Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Mon, 12 Mar 2018 19:15:06 +0800 Subject: [PATCH 149/648] Fixed a broken link --- src/traits-canonicalization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/traits-canonicalization.md b/src/traits-canonicalization.md index fc55fac0d..6ff61fdda 100644 --- a/src/traits-canonicalization.md +++ b/src/traits-canonicalization.md @@ -41,7 +41,7 @@ trait query: `?A: Foo<'static, ?B>`, where `?A` and `?B` are unbound. This query contains two unbound variables, but it also contains the lifetime `'static`. The trait system generally ignores all lifetimes and treats them equally, so when canonicalizing, we will *also* -replace any [free lifetime](./background.html#free-vs-bound) with a +replace any [free lifetime](./appendix-background.html#free-vs-bound) with a canonical variable. Therefore, we get the following result: ?0: Foo<'?1, ?2> From 9874533b6e5a3b33b0d6205d62948a6c216d0014 Mon Sep 17 00:00:00 2001 From: Michael Bryan Date: Mon, 12 Mar 2018 19:15:26 +0800 Subject: [PATCH 150/648] Added links back to nrc's stupid-stats --- src/appendix-stupid-stats.md | 9 ++++++++- src/rustc-driver.md | 7 ++++++- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/src/appendix-stupid-stats.md b/src/appendix-stupid-stats.md index b1c9d2141..20d5aaf9b 100644 --- a/src/appendix-stupid-stats.md +++ b/src/appendix-stupid-stats.md @@ -1,5 +1,10 @@ # Appendix A: A tutorial on creating a drop-in replacement for rustc +> **Note:** This is a copy of `@nrc`'s amazing [stupid-stats]. You should find +> a copy of the code on the GitHub repository although due to the compiler's +> constantly evolving nature, there is no guarantee it'll compile on the first +> go. + Many tools benefit from being a drop-in replacement for a compiler. By this, I mean that any user of the tool can use `mytool` in all the ways they would normally use `rustc` - whether manually compiling a single file or as part of a @@ -395,4 +400,6 @@ It'd be great to see Rustdoc converted to using these APIs, if that is possible analysis, rather than doing its own analysis). Other parts of the compiler (e.g., pretty printing, testing) could be refactored to use these APIs internally (I already changed save-analysis to use `CompilerController`). I've -been experimenting with a prototype rustfmt which also uses these APIs. \ No newline at end of file +been experimenting with a prototype rustfmt which also uses these APIs. + +[stupid-stats]: https://github.com/nrc/stupid-stats \ No newline at end of file diff --git a/src/rustc-driver.md b/src/rustc-driver.md index 7e6eb7e2e..23a036e73 100644 --- a/src/rustc-driver.md +++ b/src/rustc-driver.md @@ -39,6 +39,9 @@ compilation process The `CompileState`'s various `state_after_*()` constructors can be inspected to determine what bits of information are available to which callback. +For a more detailed explanation on using `rustc_driver`, check out the +[stupid-stats] guide by `@nrc` (attached as [Appendix A]). + > **Warning:** By its very nature, the internal compiler APIs are always going > to be unstable. That said, we do try not to break things unnecessarily. @@ -68,4 +71,6 @@ thread-locals, although you should rarely need to touch it. [`CompileState`]: https://github.com/rust-lang/rust/blob/master/src/librustc_driver/driver.rs [`Session`]: https://github.com/rust-lang/rust/blob/master/src/librustc/session/mod.rs [`TyCtxt`]: https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs -[`CodeMap`]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs \ No newline at end of file +[`CodeMap`]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs +[stupid-stats]: https://github.com/nrc/stupid-stats +[Appendix A]: appendix-stupid-stats.html \ No newline at end of file From ed04741208d43863b8fc694a49f680fc473d15d9 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 12 Mar 2018 12:54:19 -0500 Subject: [PATCH 151/648] fix typo --- src/traits-canonicalization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/traits-canonicalization.md b/src/traits-canonicalization.md index 6ff61fdda..bef858af2 100644 --- a/src/traits-canonicalization.md +++ b/src/traits-canonicalization.md @@ -218,7 +218,7 @@ for later verification. than eagerly instantiating all of the canonical values in the result with variables, we instead walk the vector of values, looking for cases where the value is just a canonical variable. In our example, -`values[2]` is `?C`, so that we means we can deduce that `?C := ?B and +`values[2]` is `?C`, so that means we can deduce that `?C := ?B and `'?D := 'static`. This gives us a partial set of values. Anything for which we do not find a value, we create an inference variable.) From e74567479d6f3592ae322d418b8fa13e4dc7e593 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 12 Mar 2018 19:39:56 -0500 Subject: [PATCH 152/648] Add the contents of the typeck READMEs --- src/SUMMARY.md | 2 + src/appendix-background.md | 3 + src/appendix-code-index.md | 2 + src/appendix-glossary.md | 3 +- src/method-lookup.md | 119 +++++++++++++++ src/type-checking.md | 43 ++++++ src/variance.md | 296 +++++++++++++++++++++++++++++++++++++ 7 files changed, 467 insertions(+), 1 deletion(-) create mode 100644 src/method-lookup.md create mode 100644 src/variance.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index eb889d36b..9215f8afa 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -35,6 +35,8 @@ - [The SLG solver](./traits-slg.md) - [Bibliography](./traits-bibliography.md) - [Type checking](./type-checking.md) + - [Method Lookup](./method-lookup.md) + - [Variance](./variance.md) - [The MIR (Mid-level IR)](./mir.md) - [MIR construction](./mir-construction.md) - [MIR visitor and traversal](./mir-visitor.md) diff --git a/src/appendix-background.md b/src/appendix-background.md index c69e7d93d..b49ad6d52 100644 --- a/src/appendix-background.md +++ b/src/appendix-background.md @@ -91,6 +91,9 @@ cycle. Check out the subtyping chapter from the [Rust Nomicon](https://doc.rust-lang.org/nomicon/subtyping.html). +See the [variance](./variance.html) chapter of this guide for more info on how +the type checker handles variance. + ## What is a "free region" or a "free variable"? What about "bound region"? diff --git a/src/appendix-code-index.md b/src/appendix-code-index.md index f6dcb9c37..3ca9bfb76 100644 --- a/src/appendix-code-index.md +++ b/src/appendix-code-index.md @@ -14,9 +14,11 @@ Item | Kind | Short description | Chapter | `Session` | struct | The data associated with a compilation session | [the Parser], [The Rustc Driver] | [src/librustc/session/mod.html](https://github.com/rust-lang/rust/blob/master/src/librustc/session/mod.rs) `StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser] | [src/libsyntax/parse/lexer/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/lexer/mod.rs) `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules] | [src/librustc/ty/trait_def.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/trait_def.rs) +`Ty<'tcx>` | struct | This is the internal representation of a type used for type checking | [Type checking] | [src/librustc/ty/mod.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/mod.rs) `TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules] | [src/librustc/ty/context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs) [The HIR]: hir.html [The parser]: the-parser.html [The Rustc Driver]: rustc-driver.html +[Type checking]: type-checking.html [The `ty` modules]: ty.html diff --git a/src/appendix-glossary.md b/src/appendix-glossary.md index 1914adec6..8a3df24e8 100644 --- a/src/appendix-glossary.md +++ b/src/appendix-glossary.md @@ -56,7 +56,8 @@ token | the smallest unit of parsing. Tokens are produced aft trans | the code to translate MIR into LLVM IR. trait reference | a trait and values for its type parameters ([see more](ty.html)). ty | the internal representation of a type ([see more](ty.html)). -variance | variance determines how changes to a generic type/lifetime parameter affect subtyping; for example, if `T` is a subtype of `U`, then `Vec` is a subtype `Vec` because `Vec` is *covariant* in its generic parameter. See [the background chapter for more](./appendix-background.html#variance). +UFCS | Universal Function Call Syntax. An unambiguous syntax for calling a method ([see more](type-checking.html)). +variance | variance determines how changes to a generic type/lifetime parameter affect subtyping; for example, if `T` is a subtype of `U`, then `Vec` is a subtype `Vec` because `Vec` is *covariant* in its generic parameter. See [the background chapter](./appendix-background.html#variance) for a more general explanation. See the [variance chapter](./variance.html) for an explanation of how type checking handles variance. [LLVM]: https://llvm.org/ [lto]: https://llvm.org/docs/LinkTimeOptimization.html diff --git a/src/method-lookup.md b/src/method-lookup.md new file mode 100644 index 000000000..543428d62 --- /dev/null +++ b/src/method-lookup.md @@ -0,0 +1,119 @@ +# Method lookup + +Method lookup can be rather complex due to the interaction of a number +of factors, such as self types, autoderef, trait lookup, etc. This +file provides an overview of the process. More detailed notes are in +the code itself, naturally. + +One way to think of method lookup is that we convert an expression of +the form: + +```rust +receiver.method(...) +``` + +into a more explicit UFCS form: + +```rust +Trait::method(ADJ(receiver), ...) // for a trait call +ReceiverType::method(ADJ(receiver), ...) // for an inherent method call +``` + +Here `ADJ` is some kind of adjustment, which is typically a series of +autoderefs and then possibly an autoref (e.g., `&**receiver`). However +we sometimes do other adjustments and coercions along the way, in +particular unsizing (e.g., converting from `[T; n]` to `[T]`). + +Method lookup is divided into two major phases: + +1. Probing ([`probe.rs`][probe]). The probe phase is when we decide what method + to call and how to adjust the receiver. +2. Confirmation ([`confirm.rs`][confirm]). The confirmation phase "applies" + this selection, updating the side-tables, unifying type variables, and + otherwise doing side-effectful things. + +One reason for this division is to be more amenable to caching. The +probe phase produces a "pick" (`probe::Pick`), which is designed to be +cacheable across method-call sites. Therefore, it does not include +inference variables or other information. + +[probe]: https://github.com/rust-lang/rust/blob/master/src/librustc_typeck/check/method/probe.rs +[confirm]: https://github.com/rust-lang/rust/blob/master/src/librustc_typeck/check/method/confirm.rs + +## The Probe phase + +### Steps + +The first thing that the probe phase does is to create a series of +*steps*. This is done by progressively dereferencing the receiver type +until it cannot be deref'd anymore, as well as applying an optional +"unsize" step. So if the receiver has type `Rc>`, this +might yield: + +```rust +Rc> +Box<[T; 3]> +[T; 3] +[T] +``` + +### Candidate assembly + +We then search along those steps to create a list of *candidates*. A +`Candidate` is a method item that might plausibly be the method being +invoked. For each candidate, we'll derive a "transformed self type" +that takes into account explicit self. + +Candidates are grouped into two kinds, inherent and extension. + +**Inherent candidates** are those that are derived from the +type of the receiver itself. So, if you have a receiver of some +nominal type `Foo` (e.g., a struct), any methods defined within an +impl like `impl Foo` are inherent methods. Nothing needs to be +imported to use an inherent method, they are associated with the type +itself (note that inherent impls can only be defined in the same +module as the type itself). + +FIXME: Inherent candidates are not always derived from impls. If you +have a trait object, such as a value of type `Box`, then the +trait methods (`to_string()`, in this case) are inherently associated +with it. Another case is type parameters, in which case the methods of +their bounds are inherent. However, this part of the rules is subject +to change: when DST's "impl Trait for Trait" is complete, trait object +dispatch could be subsumed into trait matching, and the type parameter +behavior should be reconsidered in light of where clauses. + +TODO: Is this FIXME still accurate? + +**Extension candidates** are derived from imported traits. If I have +the trait `ToString` imported, and I call `to_string()` on a value of +type `T`, then we will go off to find out whether there is an impl of +`ToString` for `T`. These kinds of method calls are called "extension +methods". They can be defined in any module, not only the one that +defined `T`. Furthermore, you must import the trait to call such a +method. + +So, let's continue our example. Imagine that we were calling a method +`foo` with the receiver `Rc>` and there is a trait `Foo` +that defines it with `&self` for the type `Rc` as well as a method +on the type `Box` that defines `Foo` but with `&mut self`. Then we +might have two candidates: + + &Rc> from the impl of `Foo` for `Rc` where `U=Box + &mut Box<[T; 3]>> from the inherent impl on `Box` where `U=[T; 3]` + +### Candidate search + +Finally, to actually pick the method, we will search down the steps, +trying to match the receiver type against the candidate types. At +each step, we also consider an auto-ref and auto-mut-ref to see whether +that makes any of the candidates match. We pick the first step where +we find a match. + +In the case of our example, the first step is `Rc>`, +which does not itself match any candidate. But when we autoref it, we +get the type `&Rc>` which does match. We would then +recursively consider all where-clauses that appear on the impl: if +those match (or we cannot rule out that they do), then this is the +method we would pick. Otherwise, we would continue down the series of +steps. diff --git a/src/type-checking.md b/src/type-checking.md index c559c1283..8148ef627 100644 --- a/src/type-checking.md +++ b/src/type-checking.md @@ -1 +1,44 @@ # Type checking + +The [`rustc_typeck`][typeck] crate contains the source for "type collection" +and "type checking", as well as a few other bits of related functionality. (It +draws heavily on the [type inference] and [trait solving].) + +[typeck]: https://github.com/rust-lang/rust/tree/master/src/librustc_typeck +[type inference]: type-inference.html +[trait solving]: trait-resolution.html + +## Type collection + +Type "collection" is the process of converting the types found in the HIR +(`hir::Ty`), which represent the syntactic things that the user wrote, into the +**internal representation** used by the compiler (`Ty<'tcx>`) -- we also do +similar conversions for where-clauses and other bits of the function signature. + +To try and get a sense for the difference, consider this function: + +```rust +struct Foo { } +fn foo(x: Foo, y: self::Foo) { .. } +// ^^^ ^^^^^^^^^ +``` + +Those two parameters `x` and `y` each have the same type: but they will have +distinct `hir::Ty` nodes. Those nodes will have different spans, and of course +they encode the path somewhat differently. But once they are "collected" into +`Ty<'tcx>` nodes, they will be represented by the exact same internal type. + +Collection is defined as a bundle of [queries] for computing information about +the various functions, traits, and other items in the crate being compiled. +Note that each of these queries is concerned with *interprocedural* things -- +for example, for a function definition, collection will figure out the type and +signature of the function, but it will not visit the *body* of the function in +any way, nor examine type annotations on local variables (that's the job of +type *checking*). + +For more details, see the [`collect`][collect] module. + +[queries]: query.html +[collect]: https://github.com/rust-lang/rust/blob/master/src/librustc_typeck/collect.rs + +**TODO**: actually talk about type checking... diff --git a/src/variance.md b/src/variance.md new file mode 100644 index 000000000..16b4a7518 --- /dev/null +++ b/src/variance.md @@ -0,0 +1,296 @@ +# Variance of type and lifetime parameters + +For a more general background on variance, see the [background] appendix. + +[background]: ./appendix-background.html + +During type checking we must infer the variance of type and lifetime +parameters. The algorithm is taken from Section 4 of the paper ["Taming the +Wildcards: Combining Definition- and Use-Site Variance"][pldi11] published in +PLDI'11 and written by Altidor et al., and hereafter referred to as The Paper. + +[pldi11]: https://people.cs.umass.edu/~yannis/variance-extended2011.pdf + +This inference is explicitly designed *not* to consider the uses of +types within code. To determine the variance of type parameters +defined on type `X`, we only consider the definition of the type `X` +and the definitions of any types it references. + +We only infer variance for type parameters found on *data types* +like structs and enums. In these cases, there is a fairly straightforward +explanation for what variance means. The variance of the type +or lifetime parameters defines whether `T` is a subtype of `T` +(resp. `T<'a>` and `T<'b>`) based on the relationship of `A` and `B` +(resp. `'a` and `'b`). + +We do not infer variance for type parameters found on traits, functions, +or impls. Variance on trait parameters can indeed make sense +(and we used to compute it) but it is actually rather subtle in +meaning and not that useful in practice, so we removed it. See the +[addendum] for some details. Variances on function/impl parameters, on the +other hand, doesn't make sense because these parameters are instantiated and +then forgotten, they don't persist in types or compiled byproducts. + +[addendum]: #addendum + +> **Notation** +> +> We use the notation of The Paper throughout this chapter: +> +> - `+` is _covariance_. +> - `-` is _contravariance_. +> - `*` is _bivariance_. +> - `o` is _invariance_. + +## The algorithm + +The basic idea is quite straightforward. We iterate over the types +defined and, for each use of a type parameter `X`, accumulate a +constraint indicating that the variance of `X` must be valid for the +variance of that use site. We then iteratively refine the variance of +`X` until all constraints are met. There is *always* a solution, because at +the limit we can declare all type parameters to be invariant and all +constraints will be satisfied. + +As a simple example, consider: + +```rust +enum Option { Some(A), None } +enum OptionalFn { Some(|B|), None } +enum OptionalMap { Some(|C| -> C), None } +``` + +Here, we will generate the constraints: + + 1. V(A) <= + + 2. V(B) <= - + 3. V(C) <= + + 4. V(C) <= - + +These indicate that (1) the variance of A must be at most covariant; +(2) the variance of B must be at most contravariant; and (3, 4) the +variance of C must be at most covariant *and* contravariant. All of these +results are based on a variance lattice defined as follows: + + * Top (bivariant) + - + + o Bottom (invariant) + +Based on this lattice, the solution `V(A)=+`, `V(B)=-`, `V(C)=o` is the +optimal solution. Note that there is always a naive solution which +just declares all variables to be invariant. + +You may be wondering why fixed-point iteration is required. The reason +is that the variance of a use site may itself be a function of the +variance of other type parameters. In full generality, our constraints +take the form: + + V(X) <= Term + Term := + | - | * | o | V(X) | Term x Term + +Here the notation `V(X)` indicates the variance of a type/region +parameter `X` with respect to its defining class. `Term x Term` +represents the "variance transform" as defined in the paper: + +> If the variance of a type variable `X` in type expression `E` is `V2` + and the definition-site variance of the [corresponding] type parameter + of a class `C` is `V1`, then the variance of `X` in the type expression + `C` is `V3 = V1.xform(V2)`. + +## Constraints + +If I have a struct or enum with where clauses: + +```rust +struct Foo { ... } +``` + +you might wonder whether the variance of `T` with respect to `Bar` affects the +variance `T` with respect to `Foo`. I claim no. The reason: assume that `T` is +invariant with respect to `Bar` but covariant with respect to `Foo`. And then +we have a `Foo` that is upcast to `Foo`, where `X <: Y`. However, while +`X : Bar`, `Y : Bar` does not hold. In that case, the upcast will be illegal, +but not because of a variance failure, but rather because the target type +`Foo` is itself just not well-formed. Basically we get to assume +well-formedness of all types involved before considering variance. + +### Dependency graph management + +Because variance is a whole-crate inference, its dependency graph +can become quite muddled if we are not careful. To resolve this, we refactor +into two queries: + +- `crate_variances` computes the variance for all items in the current crate. +- `variances_of` accesses the variance for an individual reading; it + works by requesting `crate_variances` and extracting the relevant data. + +If you limit yourself to reading `variances_of`, your code will only +depend then on the inference of that particular item. + +Ultimately, this setup relies on the [red-green algorithm][rga]. In particular, +every variance query effectively depends on all type definitions in the entire +crate (through `crate_variances`), but since most changes will not result in a +change to the actual results from variance inference, the `variances_of` query +will wind up being considered green after it is re-evaluated. + +[rga]: ./incremental-compilation.html + + + +## Addendum: Variance on traits + +As mentioned above, we used to permit variance on traits. This was +computed based on the appearance of trait type parameters in +method signatures and was used to represent the compatibility of +vtables in trait objects (and also "virtual" vtables or dictionary +in trait bounds). One complication was that variance for +associated types is less obvious, since they can be projected out +and put to myriad uses, so it's not clear when it is safe to allow +`X::Bar` to vary (or indeed just what that means). Moreover (as +covered below) all inputs on any trait with an associated type had +to be invariant, limiting the applicability. Finally, the +annotations (`MarkerTrait`, `PhantomFn`) needed to ensure that all +trait type parameters had a variance were confusing and annoying +for little benefit. + +Just for historical reference, I am going to preserve some text indicating how +one could interpret variance and trait matching. + +### Variance and object types + +Just as with structs and enums, we can decide the subtyping +relationship between two object types `&Trait` and `&Trait` +based on the relationship of `A` and `B`. Note that for object +types we ignore the `Self` type parameter -- it is unknown, and +the nature of dynamic dispatch ensures that we will always call a +function that is expected the appropriate `Self` type. However, we +must be careful with the other type parameters, or else we could +end up calling a function that is expecting one type but provided +another. + +To see what I mean, consider a trait like so: + + trait ConvertTo { + fn convertTo(&self) -> A; + } + +Intuitively, If we had one object `O=&ConvertTo` and another +`S=&ConvertTo`, then `S <: O` because `String <: Object` +(presuming Java-like "string" and "object" types, my go to examples +for subtyping). The actual algorithm would be to compare the +(explicit) type parameters pairwise respecting their variance: here, +the type parameter A is covariant (it appears only in a return +position), and hence we require that `String <: Object`. + +You'll note though that we did not consider the binding for the +(implicit) `Self` type parameter: in fact, it is unknown, so that's +good. The reason we can ignore that parameter is precisely because we +don't need to know its value until a call occurs, and at that time (as +you said) the dynamic nature of virtual dispatch means the code we run +will be correct for whatever value `Self` happens to be bound to for +the particular object whose method we called. `Self` is thus different +from `A`, because the caller requires that `A` be known in order to +know the return type of the method `convertTo()`. (As an aside, we +have rules preventing methods where `Self` appears outside of the +receiver position from being called via an object.) + +### Trait variance and vtable resolution + +But traits aren't only used with objects. They're also used when +deciding whether a given impl satisfies a given trait bound. To set the +scene here, imagine I had a function: + + fn convertAll>(v: &[T]) { + ... + } + +Now imagine that I have an implementation of `ConvertTo` for `Object`: + + impl ConvertTo for Object { ... } + +And I want to call `convertAll` on an array of strings. Suppose +further that for whatever reason I specifically supply the value of +`String` for the type parameter `T`: + + let mut vector = vec!["string", ...]; + convertAll::(vector); + +Is this legal? To put another way, can we apply the `impl` for +`Object` to the type `String`? The answer is yes, but to see why +we have to expand out what will happen: + +- `convertAll` will create a pointer to one of the entries in the + vector, which will have type `&String` +- It will then call the impl of `convertTo()` that is intended + for use with objects. This has the type: + + fn(self: &Object) -> i32 + + It is ok to provide a value for `self` of type `&String` because + `&String <: &Object`. + +OK, so intuitively we want this to be legal, so let's bring this back +to variance and see whether we are computing the correct result. We +must first figure out how to phrase the question "is an impl for +`Object,i32` usable where an impl for `String,i32` is expected?" + +Maybe it's helpful to think of a dictionary-passing implementation of +type classes. In that case, `convertAll()` takes an implicit parameter +representing the impl. In short, we *have* an impl of type: + + V_O = ConvertTo for Object + +and the function prototype expects an impl of type: + + V_S = ConvertTo for String + +As with any argument, this is legal if the type of the value given +(`V_O`) is a subtype of the type expected (`V_S`). So is `V_O <: V_S`? +The answer will depend on the variance of the various parameters. In +this case, because the `Self` parameter is contravariant and `A` is +covariant, it means that: + + V_O <: V_S iff + i32 <: i32 + String <: Object + +These conditions are satisfied and so we are happy. + +### Variance and associated types + +Traits with associated types -- or at minimum projection +expressions -- must be invariant with respect to all of their +inputs. To see why this makes sense, consider what subtyping for a +trait reference means: + + <: + +means that if I know that `T as Trait`, I also know that `U as +Trait`. Moreover, if you think of it as dictionary passing style, +it means that a dictionary for `` is safe to use where +a dictionary for `` is expected. + +The problem is that when you can project types out from ``, the relationship to types projected out of `` +is completely unknown unless `T==U` (see #21726 for more +details). Making `Trait` invariant ensures that this is true. + +Another related reason is that if we didn't make traits with +associated types invariant, then projection is no longer a +function with a single result. Consider: + +``` +trait Identity { type Out; fn foo(&self); } +impl Identity for T { type Out = T; ... } +``` + +Now if I have `<&'static () as Identity>::Out`, this can be +validly derived as `&'a ()` for any `'a`: + + <&'a () as Identity> <: <&'static () as Identity> + if &'static () < : &'a () -- Identity is contravariant in Self + if 'static : 'a -- Subtyping rules for relations + +This change otoh means that `<'static () as Identity>::Out` is +always `&'static ()` (which might then be upcast to `'a ()`, +separately). This was helpful in solving #21750. From a19cdc7a7174fe00f308983673dce572cc1d02d7 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 12 Mar 2018 19:50:43 -0500 Subject: [PATCH 153/648] Add the rustdoc readme --- src/SUMMARY.md | 1 + src/rustdoc.md | 177 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+) create mode 100644 src/rustdoc.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 9215f8afa..3079c0b48 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -10,6 +10,7 @@ - [Walkthrough: a typical contribution](./walkthrough.md) - [High-level overview of the compiler source](./high-level-overview.md) - [The Rustc Driver](./rustc-driver.md) + - [Rustdoc](./rustdoc.md) - [Queries: demand-driven compilation](./query.md) - [Incremental compilation](./incremental-compilation.md) - [The parser](./the-parser.md) diff --git a/src/rustdoc.md b/src/rustdoc.md new file mode 100644 index 000000000..4236ad6ff --- /dev/null +++ b/src/rustdoc.md @@ -0,0 +1,177 @@ +# The walking tour of rustdoc + +Rustdoc actually uses the rustc internals directly. It lives in-tree with the compiler and standard +library. This chapter is about how it works. (A new implementation is also [under way], though). + +[under way]: https://github.com/steveklabnik/rustdoc + +Rustdoc is implemented entirely within the crate `librustdoc`. After partially compiling a crate to +get its AST (technically the HIR map) from rustc, librustdoc performs two major steps past that to +render a set of documentation: + +* "Clean" the AST into a form that's more suited to creating documentation (and slightly more + resistant to churn in the compiler). +* Use this cleaned AST to render a crate's documentation, one page at a time. + +Naturally, there's more than just this, and those descriptions simplify out lots of details, but +that's the high-level overview. + +(Side note: this is a library crate! The `rustdoc` binary is crated using the project in +`src/tools/rustdoc`. Note that literally all that does is call the `main()` that's in this crate's +`lib.rs`, though.) + +## Cheat sheet + +* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable rustdoc you can run on + other projects. + * Add `src/libtest` to be able to use `rustdoc --test`. + * If you've used `rustup toolchain link local /path/to/build/$TARGET/stage1` previously, then + after the previous build command, `cargo +local doc` will Just Work. +* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the standard library docs. + * The completed docs will be available in `build/$TARGET/doc/std`, though the bundle is meant to + be used as though you would copy out the `doc` folder to a web server, since that's where the + CSS/JS and landing page are. +* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`. It's in a bunch of + `fmt::Display` implementations and supplementary functions. +* The types that got `Display` impls above are defined in `clean/mod.rs`, right next to the custom + `Clean` trait used to process them out of the rustc HIR. +* The bits specific to using rustdoc as a test harness are in `test.rs`. +* The Markdown renderer is loaded up in `html/markdown.rs`, including functions for extracting + doctests from a given block of Markdown. +* The tests on rustdoc *output* are located in `src/test/rustdoc`, where they're handled by the test + runner of rustbuild and the supplementary script `src/etc/htmldocck.py`. +* Tests on search index generation are located in `src/test/rustdoc-js`, as a series of JavaScript + files that encode queries on the standard library search index and expected results. + +## From crate to clean + +In `core.rs` are two central items: the `DocContext` struct, and the `run_core` function. The latter +is where rustdoc calls out to rustc to compile a crate to the point where rustdoc can take over. The +former is a state container used when crawling through a crate to gather its documentation. + +The main process of crate crawling is done in `clean/mod.rs` through several implementations of the +`Clean` trait defined within. This is a conversion trait, which defines one method: + +```rust +pub trait Clean { + fn clean(&self, cx: &DocContext) -> T; +} +``` + +`clean/mod.rs` also defines the types for the "cleaned" AST used later on to render documentation +pages. Each usually accompanies an implementation of `Clean` that takes some AST or HIR type from +rustc and converts it into the appropriate "cleaned" type. "Big" items like modules or associated +items may have some extra processing in its `Clean` implementation, but for the most part these +impls are straightforward conversions. The "entry point" to this module is the `impl Clean +for visit_ast::RustdocVisitor`, which is called by `run_core` above. + +You see, I actually lied a little earlier: There's another AST transformation that happens before +the events in `clean/mod.rs`. In `visit_ast.rs` is the type `RustdocVisitor`, which *actually* +crawls a `hir::Crate` to get the first intermediate representation, defined in `doctree.rs`. This +pass is mainly to get a few intermediate wrappers around the HIR types and to process visibility +and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and `#[doc(hidden)]` are +processed, as well as the logic for whether a `pub use` should get the full page or a "Reexport" +line in the module page. + +The other major thing that happens in `clean/mod.rs` is the collection of doc comments and +`#[doc=""]` attributes into a separate field of the Attributes struct, present on anything that gets +hand-written documentation. This makes it easier to collect this documentation later in the process. + +The primary output of this process is a clean::Crate with a tree of Items which describe the +publicly-documentable items in the target crate. + +### Hot potato + +Before moving on to the next major step, a few important "passes" occur over the documentation. +These do things like combine the separate "attributes" into a single string and strip leading +whitespace to make the document easier on the markdown parser, or drop items that are not public or +deliberately hidden with `#[doc(hidden)]`. These are all implemented in the `passes/` directory, one +file per pass. By default, all of these passes are run on a crate, but the ones regarding dropping +private/hidden items can be bypassed by passing `--document-private-items` to rustdoc. + +(Strictly speaking, you can fine-tune the passes run and even add your own, but [we're trying to +deprecate that][44136]. If you need finer-grain control over these passes, please let us know!) + +[44136]: https://github.com/rust-lang/rust/issues/44136 + +## From clean to crate + +This is where the "second phase" in rustdoc begins. This phase primarily lives in the `html/` +folder, and it all starts with `run()` in `html/render.rs`. This code is responsible for setting up +the `Context`, `SharedContext`, and `Cache` which are used during rendering, copying out the static +files which live in every rendered set of documentation (things like the fonts, CSS, and JavaScript +that live in `html/static/`), creating the search index, and printing out the source code rendering, +before beginning the process of rendering all the documentation for the crate. + +Several functions implemented directly on `Context` take the `clean::Crate` and set up some state +between rendering items or recursing on a module's child items. From here the "page rendering" +begins, via an enormous `write!()` call in `html/layout.rs`. The parts that actually generate HTML +from the items and documentation occurs within a series of `std::fmt::Display` implementations and +functions that pass around a `&mut std::fmt::Formatter`. The top-level implementation that writes +out the page body is the `impl<'a> fmt::Display for Item<'a>` in `html/render.rs`, which switches +out to one of several `item_*` functions based on the kind of `Item` being rendered. + +Depending on what kind of rendering code you're looking for, you'll probably find it either in +`html/render.rs` for major items like "what sections should I print for a struct page" or +`html/format.rs` for smaller component pieces like "how should I print a where clause as part of +some other item". + +Whenever rustdoc comes across an item that should print hand-written documentation alongside, it +calls out to `html/markdown.rs` which interfaces with the Markdown parser. This is exposed as a +series of types that wrap a string of Markdown, and implement `fmt::Display` to emit HTML text. It +takes special care to enable certain features like footnotes and tables and add syntax highlighting +to Rust code blocks (via `html/highlight.rs`) before running the Markdown parser. There's also a +function in here (`find_testable_code`) that specifically scans for Rust code blocks so the +test-runner code can find all the doctests in the crate. + +### From soup to nuts + +(alternate title: ["An unbroken thread that stretches from those first `Cell`s to us"][video]) + +[video]: https://www.youtube.com/watch?v=hOLAGYmUQV0 + +It's important to note that the AST cleaning can ask the compiler for information (crucially, +`DocContext` contains a `TyCtxt`), but page rendering cannot. The `clean::Crate` created within +`run_core` is passed outside the compiler context before being handed to `html::render::run`. This +means that a lot of the "supplementary data" that isn't immediately available inside an item's +definition, like which trait is the `Deref` trait used by the language, needs to be collected during +cleaning, stored in the `DocContext`, and passed along to the `SharedContext` during HTML rendering. +This manifests as a bunch of shared state, context variables, and `RefCell`s. + +Also of note is that some items that come from "asking the compiler" don't go directly into the +`DocContext` - for example, when loading items from a foreign crate, rustdoc will ask about trait +implementations and generate new `Item`s for the impls based on that information. This goes directly +into the returned `Crate` rather than roundabout through the `DocContext`. This way, these +implementations can be collected alongside the others, right before rendering the HTML. + +## Other tricks up its sleeve + +All this describes the process for generating HTML documentation from a Rust crate, but there are +couple other major modes that rustdoc runs in. It can also be run on a standalone Markdown file, or +it can run doctests on Rust code or standalone Markdown files. For the former, it shortcuts straight +to `html/markdown.rs`, optionally including a mode which inserts a Table of Contents to the output +HTML. + +For the latter, rustdoc runs a similar partial-compilation to get relevant documentation in +`test.rs`, but instead of going through the full clean and render process, it runs a much simpler +crate walk to grab *just* the hand-written documentation. Combined with the aforementioned +"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of tests to run before +handing them off to the libtest test runner. One notable location in `test.rs` is the function +`make_test`, which is where hand-written doctests get transformed into something that can be +executed. + +## Dotting i's and crossing t's + +So that's rustdoc's code in a nutshell, but there's more things in the repo that deal with it. Since +we have the full `compiletest` suite at hand, there's a set of tests in `src/test/rustdoc` that make +sure the final HTML is what we expect in various situations. These tests also use a supplementary +script, `src/etc/htmldocck.py`, that allows it to look through the final HTML using XPath notation +to get a precise look at the output. The full description of all the commands available to rustdoc +tests is in `htmldocck.py`. + +In addition, there are separate tests for the search index and rustdoc's ability to query it. The +files in `src/test/rustdoc-js` each contain a different search query and the expected results, +broken out by search tab. These files are processed by a script in `src/tools/rustdoc-js` and the +Node.js runtime. These tests don't have as thorough of a writeup, but a broad example that features +results in all tabs can be found in `basic.js`. The basic idea is that you match a given `QUERY` +with a set of `EXPECTED` results, complete with the full item path of each item. From 21ed6e6ce41667034e25c7ffee04d628e510af40 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 14 Mar 2018 22:08:09 -0500 Subject: [PATCH 154/648] Address review comments --- src/appendix-code-index.md | 1 + src/rustdoc.md | 44 ++++++++++++++++++++++++++++++++------ 2 files changed, 39 insertions(+), 6 deletions(-) diff --git a/src/appendix-code-index.md b/src/appendix-code-index.md index 3ca9bfb76..c5be7e336 100644 --- a/src/appendix-code-index.md +++ b/src/appendix-code-index.md @@ -8,6 +8,7 @@ Item | Kind | Short description | Chapter | ----------------|----------|-----------------------------|--------------------|------------------- `CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser] | [src/libsyntax/codemap.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs) `CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver] | [src/librustc_driver/driver.rs](https://github.com/rust-lang/rust/blob/master/src/librustc_driver/driver.rs) +`DocContext` | struct | A state container used by rustdoc when crawling through a crate to gather its documentation | [Rustdoc] | [src/librustdoc/core.rs](https://github.com/rust-lang/rust/blob/master/src/librustdoc/core.rs) `ast::Crate` | struct | Syntax-level representation of a parsed crate | [The parser] | [src/librustc/hir/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/ast.rs) `hir::Crate` | struct | More abstract, compiler-friendly form of a crate's AST | [The Hir] | [src/librustc/hir/mod.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/hir/mod.rs) `ParseSess` | struct | This struct contains information about a parsing session | [the Parser] | [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs) diff --git a/src/rustdoc.md b/src/rustdoc.md index 4236ad6ff..2cacc80fd 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -5,9 +5,17 @@ library. This chapter is about how it works. (A new implementation is also [unde [under way]: https://github.com/steveklabnik/rustdoc -Rustdoc is implemented entirely within the crate `librustdoc`. After partially compiling a crate to -get its AST (technically the HIR map) from rustc, librustdoc performs two major steps past that to -render a set of documentation: +Rustdoc is implemented entirely within the crate [`librustdoc`][rd]. It runs +the compiler up to the point where we have an internal representation of a +crate (HIR) and the ability to run some queries about the types of items. [HIR] +and [queries] are discussed in the linked chapters. + +[HIR]: ./hir.html +[queries]: ./query.html +[rd]: https://github.com/rust-lang/rust/tree/master/src/librustdoc + +`librustdoc` performs two major steps after that to render a set of +documentation: * "Clean" the AST into a form that's more suited to creating documentation (and slightly more resistant to churn in the compiler). @@ -16,10 +24,12 @@ render a set of documentation: Naturally, there's more than just this, and those descriptions simplify out lots of details, but that's the high-level overview. -(Side note: this is a library crate! The `rustdoc` binary is crated using the project in -`src/tools/rustdoc`. Note that literally all that does is call the `main()` that's in this crate's +(Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated using the project in +[`src/tools/rustdoc`][bin]. Note that literally all that does is call the `main()` that's in this crate's `lib.rs`, though.) +[bin]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc + ## Cheat sheet * Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable rustdoc you can run on @@ -87,13 +97,32 @@ These do things like combine the separate "attributes" into a single string and whitespace to make the document easier on the markdown parser, or drop items that are not public or deliberately hidden with `#[doc(hidden)]`. These are all implemented in the `passes/` directory, one file per pass. By default, all of these passes are run on a crate, but the ones regarding dropping -private/hidden items can be bypassed by passing `--document-private-items` to rustdoc. +private/hidden items can be bypassed by passing `--document-private-items` to rustdoc. Note that +unlike the previous set of AST transformations, the passes happen on the _cleaned_ crate. (Strictly speaking, you can fine-tune the passes run and even add your own, but [we're trying to deprecate that][44136]. If you need finer-grain control over these passes, please let us know!) [44136]: https://github.com/rust-lang/rust/issues/44136 +Here is current (as of this writing) list of passes: + +- `collapse-docs` is necessary because each line of a doc comment is given as a + separate doc attribute, and this will combine them into a single string with + line breaks between each attribute. +- `unindent-comments` is necessary because the convention for writing + documentation is to provide a space between the `///` or `//!` marker and the + text, and stripping that leading space will make the text easier to parse by + the Markdown parser. (In my experience it's less necessary now that we have a + Commonmark-compliant parser, since it doesn't have a problem with headers + that have a space before the `##` that marks the heading.) +- `strip-priv-imports` is necessary because rustdoc will handle *public* + imports by either inlining the item's documentation to the module or creating + a "Reexports" section with the import in it. The pass ensures that all of + these imports are actually relevant to documentation. +- `strip-hidden` and `strip-private` also remove items that are not relevant + for public documentation. + ## From clean to crate This is where the "second phase" in rustdoc begins. This phase primarily lives in the `html/` @@ -160,6 +189,9 @@ handing them off to the libtest test runner. One notable location in `test.rs` i `make_test`, which is where hand-written doctests get transformed into something that can be executed. +Some extra reading about `make_test` can be found +[here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/). + ## Dotting i's and crossing t's So that's rustdoc's code in a nutshell, but there's more things in the repo that deal with it. Since From 7c2af89682e0811ac7be825de128b220d389750a Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Thu, 15 Mar 2018 09:38:50 -0500 Subject: [PATCH 155/648] address review comments --- src/appendix-code-index.md | 1 + src/rustdoc.md | 17 +++++++++++------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/src/appendix-code-index.md b/src/appendix-code-index.md index c5be7e336..49fe08ee3 100644 --- a/src/appendix-code-index.md +++ b/src/appendix-code-index.md @@ -23,3 +23,4 @@ Item | Kind | Short description | Chapter | [The Rustc Driver]: rustc-driver.html [Type checking]: type-checking.html [The `ty` modules]: ty.html +[Rustdoc]: rustdoc.html diff --git a/src/rustdoc.md b/src/rustdoc.md index 2cacc80fd..bb4122e8d 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -87,7 +87,7 @@ The other major thing that happens in `clean/mod.rs` is the collection of doc co `#[doc=""]` attributes into a separate field of the Attributes struct, present on anything that gets hand-written documentation. This makes it easier to collect this documentation later in the process. -The primary output of this process is a clean::Crate with a tree of Items which describe the +The primary output of this process is a `clean::Crate` with a tree of Items which describe the publicly-documentable items in the target crate. ### Hot potato @@ -107,21 +107,26 @@ deprecate that][44136]. If you need finer-grain control over these passes, pleas Here is current (as of this writing) list of passes: -- `collapse-docs` is necessary because each line of a doc comment is given as a +- `propagate-doc-cfg` - propagates `#[doc(cfg(...))]` to child items. +- `collapse-docs` concatenates all document attributes into one document + attribute. This is necessary because each line of a doc comment is given as a separate doc attribute, and this will combine them into a single string with line breaks between each attribute. -- `unindent-comments` is necessary because the convention for writing +- `unindent-comments` removes excess indentation on comments in order for + markdown to like it. This is necessary because the convention for writing documentation is to provide a space between the `///` or `//!` marker and the text, and stripping that leading space will make the text easier to parse by the Markdown parser. (In my experience it's less necessary now that we have a Commonmark-compliant parser, since it doesn't have a problem with headers that have a space before the `##` that marks the heading.) -- `strip-priv-imports` is necessary because rustdoc will handle *public* +- `strip-priv-imports` strips all private import statements (`use`, `extern + crate`) from a crate. This is necessary because rustdoc will handle *public* imports by either inlining the item's documentation to the module or creating a "Reexports" section with the import in it. The pass ensures that all of these imports are actually relevant to documentation. -- `strip-hidden` and `strip-private` also remove items that are not relevant - for public documentation. +- `strip-hidden` and `strip-private` strip all `doc(hidden)` and private items + from the output. `strip-private` implies `strip-priv-imports`. Basically, the + goal is to remove items that are not relevant for public documentation. ## From clean to crate From 81071c68384c3c49499ac37de214e6317551a86d Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 15 Mar 2018 12:46:59 -0500 Subject: [PATCH 156/648] Updated parenthetical --- src/rustdoc.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/rustdoc.md b/src/rustdoc.md index bb4122e8d..ae640d110 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -116,9 +116,9 @@ Here is current (as of this writing) list of passes: markdown to like it. This is necessary because the convention for writing documentation is to provide a space between the `///` or `//!` marker and the text, and stripping that leading space will make the text easier to parse by - the Markdown parser. (In my experience it's less necessary now that we have a - Commonmark-compliant parser, since it doesn't have a problem with headers - that have a space before the `##` that marks the heading.) + the Markdown parser. (In the past, the markdown parser used was not Commonmark- + compliant, which caused annoyances with extra whitespace but this seems to be + less of an issue today.) - `strip-priv-imports` strips all private import statements (`use`, `extern crate`) from a crate. This is necessary because rustdoc will handle *public* imports by either inlining the item's documentation to the module or creating From 52787a6ec4f1ae782b459d9c1b9c71a134f8d851 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Thu, 15 Mar 2018 14:26:09 -0400 Subject: [PATCH 157/648] add names to the trait lowering rules This allows cross-references from the code. --- src/traits-lowering-rules.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/src/traits-lowering-rules.md b/src/traits-lowering-rules.md index 46827dce5..8de20f1d8 100644 --- a/src/traits-lowering-rules.md +++ b/src/traits-lowering-rules.md @@ -20,6 +20,16 @@ the [notation given in this section](./traits-goals-and-clauses.html). We sometimes insert "macros" like `LowerWhereClause!` into these definitions; these macros reference other sections within this chapter. +## Rule names and cross-references + +Each of these lowering rules is given a name, documented with a +comment like so: + + // Rule Foo-Bar-Baz + +you can also search through the `librustc_traits` crate in rustc +to find the corresponding rules from the implementation. + ## Lowering where clauses When used in a goal position, where clauses can be mapped directly to @@ -76,6 +86,7 @@ relationships between different kinds of domain goals. The first such rule from the trait header creates the mapping between the `FromEnv` and `Implemented` predicates: + // Rule Implemented-From-Env forall { Implemented(Self: Trait) :- FromEnv(Self: Trait) } @@ -89,6 +100,8 @@ The next few clauses have to do with implied bounds (see also [RFC 2089]: https://rust-lang.github.io/rfcs/2089-implied-bounds.html + // Rule Implied-Bound-From-Trait + // // For each where clause WC: forall { FromEnv(WC) :- FromEnv(Self: Trait { WellFormed(Self: Trait) :- Implemented(Self: Trait) && WellFormed(WC) @@ -175,12 +190,16 @@ the rules by which `ProjectionEq` can succeed; these two clauses are discussed in detail in the [section on associated types](./traits-associated-types.html),, but reproduced here for reference: + // Rule ProjectionEq-Normalize + // // ProjectionEq can succeed by normalizing: forall { ProjectionEq(>::AssocType = U) :- Normalize(>::AssocType -> U) } + // Rule ProjectionEq-Skolemize + // // ProjectionEq can succeed by skolemizing, see "associated type" // chapter for more: forall { @@ -223,6 +242,7 @@ where WC Let `TraitRef` be the trait reference `A0: Trait`. Then we will create the following rules: + // Rule Implemented-From-Impl forall { Implemented(TraitRef) :- WC } @@ -245,6 +265,7 @@ where WC We produce the following rule: + // Rule Normalize-From-Impl forall { forall { Normalize(>::AssocType -> T) :- From 1131ff4bbaa95bdb18a105cf4b79967766b982ae Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 16 Mar 2018 11:45:09 -0500 Subject: [PATCH 158/648] Add incremental compilation debugging subchapter --- src/SUMMARY.md | 1 + src/incrcomp-debugging.md | 113 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 114 insertions(+) create mode 100644 src/incrcomp-debugging.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 3079c0b48..47775e29c 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -13,6 +13,7 @@ - [Rustdoc](./rustdoc.md) - [Queries: demand-driven compilation](./query.md) - [Incremental compilation](./incremental-compilation.md) + - [Debugging and Testing](./incrcomp-debugging.md) - [The parser](./the-parser.md) - [Macro expansion](./macro-expansion.md) - [Name resolution](./name-resolution.md) diff --git a/src/incrcomp-debugging.md b/src/incrcomp-debugging.md new file mode 100644 index 000000000..92a8e10ea --- /dev/null +++ b/src/incrcomp-debugging.md @@ -0,0 +1,113 @@ +# Debugging and Testing Dependencies + +## Testing the dependency graph + +There are various ways to write tests against the dependency graph. +The simplest mechanisms are the `#[rustc_if_this_changed]` and +`#[rustc_then_this_would_need]` annotations. These are used in compile-fail +tests to test whether the expected set of paths exist in the dependency graph. +As an example, see `src/test/compile-fail/dep-graph-caller-callee.rs`. + +The idea is that you can annotate a test like: + +```rust +#[rustc_if_this_changed] +fn foo() { } + +#[rustc_then_this_would_need(TypeckTables)] //~ ERROR OK +fn bar() { foo(); } + +#[rustc_then_this_would_need(TypeckTables)] //~ ERROR no path +fn baz() { } +``` + +This will check whether there is a path in the dependency graph from `Hir(foo)` +to `TypeckTables(bar)`. An error is reported for each +`#[rustc_then_this_would_need]` annotation that indicates whether a path +exists. `//~ ERROR` annotations can then be used to test if a path is found (as +demonstrated above). + +## Debugging the dependency graph + +### Dumping the graph + +The compiler is also capable of dumping the dependency graph for your +debugging pleasure. To do so, pass the `-Z dump-dep-graph` flag. The +graph will be dumped to `dep_graph.{txt,dot}` in the current +directory. You can override the filename with the `RUST_DEP_GRAPH` +environment variable. + +Frequently, though, the full dep graph is quite overwhelming and not +particularly helpful. Therefore, the compiler also allows you to filter +the graph. You can filter in three ways: + +1. All edges originating in a particular set of nodes (usually a single node). +2. All edges reaching a particular set of nodes. +3. All edges that lie between given start and end nodes. + +To filter, use the `RUST_DEP_GRAPH_FILTER` environment variable, which should +look like one of the following: + +``` +source_filter // nodes originating from source_filter +-> target_filter // nodes that can reach target_filter +source_filter -> target_filter // nodes in between source_filter and target_filter +``` + +`source_filter` and `target_filter` are a `&`-separated list of strings. +A node is considered to match a filter if all of those strings appear in its +label. So, for example: + +``` +RUST_DEP_GRAPH_FILTER='-> TypeckTables' +``` + +would select the predecessors of all `TypeckTables` nodes. Usually though you +want the `TypeckTables` node for some particular fn, so you might write: + +``` +RUST_DEP_GRAPH_FILTER='-> TypeckTables & bar' +``` + +This will select only the predecessors of `TypeckTables` nodes for functions with +`bar` in their name. + +Perhaps you are finding that when you change `foo` you need to re-type-check `bar`, +but you don't think you should have to. In that case, you might do: + +``` +RUST_DEP_GRAPH_FILTER='Hir & foo -> TypeckTables & bar' +``` + +This will dump out all the nodes that lead from `Hir(foo)` to +`TypeckTables(bar)`, from which you can (hopefully) see the source +of the erroneous edge. + +### Tracking down incorrect edges + +Sometimes, after you dump the dependency graph, you will find some +path that should not exist, but you will not be quite sure how it came +to be. **When the compiler is built with debug assertions,** it can +help you track that down. Simply set the `RUST_FORBID_DEP_GRAPH_EDGE` +environment variable to a filter. Every edge created in the dep-graph +will be tested against that filter -- if it matches, a `bug!` is +reported, so you can easily see the backtrace (`RUST_BACKTRACE=1`). + +The syntax for these filters is the same as described in the previous +section. However, note that this filter is applied to every **edge** +and doesn't handle longer paths in the graph, unlike the previous +section. + +Example: + +You find that there is a path from the `Hir` of `foo` to the type +check of `bar` and you don't think there should be. You dump the +dep-graph as described in the previous section and open `dep-graph.txt` +to see something like: + + Hir(foo) -> Collect(bar) + Collect(bar) -> TypeckTables(bar) + +That first edge looks suspicious to you. So you set +`RUST_FORBID_DEP_GRAPH_EDGE` to `Hir&foo -> Collect&bar`, re-run, and +then observe the backtrace. Voila, bug fixed! From 28da7a02acc591c65834232f1a86e82d8be6c711 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 19 Mar 2018 10:50:43 -0400 Subject: [PATCH 159/648] describe how the lowering code works --- src/SUMMARY.md | 1 + src/traits-lowering-module.md | 66 +++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+) create mode 100644 src/traits-lowering-module.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 47775e29c..da84350ea 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -33,6 +33,7 @@ - [Canonical queries](./traits-canonical-queries.md) - [Canonicalization](./traits-canonicalization.md) - [Lowering rules](./traits-lowering-rules.md) + - [The lowering module in rustc](./traits-lowering-module.md) - [Well-formedness checking](./traits-wf.md) - [The SLG solver](./traits-slg.md) - [Bibliography](./traits-bibliography.md) diff --git a/src/traits-lowering-module.md b/src/traits-lowering-module.md new file mode 100644 index 000000000..c47b8fbe8 --- /dev/null +++ b/src/traits-lowering-module.md @@ -0,0 +1,66 @@ +# The lowering module in rustc + +The program clauses described in the +[lowering rules](./traits-lowering-rules.html) section are actually +created in the [`rustc_traits::lowering`][lowering] module. + +[lowering]: https://github.com/rust-lang/rust/tree/master/src/librustc_traits/lowering.rs + +## The `program_clauses_for` query + +The main entry point is the `program_clauses_for` [query], which -- +given a def-id -- produces a set of Chalk program clauses. These +queries are tested using a +[dedicated unit-testing mechanism, described below](#unit-tests). The +query is invoked on a `DefId` that identifies something like a trait, +an impl, or an associated item definition. It then produces and +returns a vector of program clauses. + +[query]: ./query.html + + + +## Unit tests + +Unit tests are located in [`src/test/ui/chalkify`][chalkify]. A good +example test is [the `lower_impl` test][lower_impl]. At the time of +this writing, it looked like this: + +```rust +#![feature(rustc_attrs)] + +trait Foo { } + +#[rustc_dump_program_clauses] //~ ERROR Implemented(T: Foo) :- +impl Foo for T where T: Iterator { } + +fn main() { + println!("hello"); +} +``` + +The `#[rustc_dump_program_clauses]` annotation can be attached to +anything with a def-id. (It requires the `rustc_attrs` feature.) The +compiler will then invoke the `program_clauses_for` query on that +item, and emit compiler errors that dump the clauses produced. These +errors just exist for unit-testing, as we can then leverage the +standard [ui test] mechanisms to check them. In this case, there is a +`//~ ERROR Implemented` annotation which is intentionally minimal (it +need only be a prefix of the error), but [the stderr file] contains +the full details: + +``` +error: Implemented(T: Foo) :- ProjectionEq(::Item == i32), TypeOutlives(T \ +: 'static), Implemented(T: std::iter::Iterator), Implemented(T: std::marker::Sized). + --> $DIR/lower_impl.rs:15:1 + | +LL | #[rustc_dump_program_clauses] //~ ERROR Implemented(T: Foo) :- + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +error: aborting due to previous error +``` + +[chalkify]: https://github.com/rust-lang/rust/tree/master/src/test/ui/chalkify +[lower_impl]: https://github.com/rust-lang/rust/tree/master/src/test/ui/chalkify/lower_impl.rs +[the stderr file]: https://github.com/rust-lang/rust/tree/master/src/test/ui/chalkify/lower_impl.stderr +[ui test]: https://rust-lang-nursery.github.io/rustc-guide/tests/adding.html#guide-to-the-ui-tests From df5401ca7a0375aacb28d2f2822984f693f147c3 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 19 Mar 2018 10:56:51 -0400 Subject: [PATCH 160/648] bump mdbook to 0.1.5 --- ci/install.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install.sh b/ci/install.sh index a5d87a6bc..0a36ff1f9 100644 --- a/ci/install.sh +++ b/ci/install.sh @@ -19,5 +19,5 @@ function cargo_install() { fi } -cargo_install mdbook 0.1.1 +cargo_install mdbook 0.1.5 cargo_install mdbook-linkcheck 0.1.0 From d16bde7fb85d254950b33e53bcb8882ea4a788d5 Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Fri, 16 Mar 2018 10:39:12 -0700 Subject: [PATCH 161/648] Add mdbook search. --- book.toml | 2 ++ ci/install.sh | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/book.toml b/book.toml index 3c4c537e8..f3a192c67 100644 --- a/book.toml +++ b/book.toml @@ -6,3 +6,5 @@ description = "A guide to developing rustc " [output.html] [output.linkcheck] + +[output.html.search] diff --git a/ci/install.sh b/ci/install.sh index 0a36ff1f9..975857a95 100644 --- a/ci/install.sh +++ b/ci/install.sh @@ -20,4 +20,4 @@ function cargo_install() { } cargo_install mdbook 0.1.5 -cargo_install mdbook-linkcheck 0.1.0 +cargo_install mdbook-linkcheck 0.1.2 From 6136199bf5ec210c9c1aa21910a32ceaeac95171 Mon Sep 17 00:00:00 2001 From: Dan Robertson Date: Sun, 25 Mar 2018 13:50:58 +0000 Subject: [PATCH 162/648] Minor grammar and syntax fixes Minor grammar and syntax fixes found while reading. --- src/trait-caching.md | 4 +++- src/traits-associated-types.md | 10 ++++++---- src/type-inference.md | 2 +- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/src/trait-caching.md b/src/trait-caching.md index ee92814cd..4c728d521 100644 --- a/src/trait-caching.md +++ b/src/trait-caching.md @@ -54,7 +54,7 @@ use is done by the method `pick_candidate_cache` in `select.rs`. At the moment, we use a very simple, conservative rule: if there are any where-clauses in scope, then we use the local cache. We used to try and draw finer-grained distinctions, but that led to a serious of -annoying and weird bugs like #22019 and #18290. This simple rule seems +annoying and weird bugs like [#22019] and [#18290]. This simple rule seems to be pretty clearly safe and also still retains a very high hit rate (~95% when compiling rustc). @@ -63,3 +63,5 @@ general, is this section still accurate at all? [`ParamEnv`]: ./param_env.html [`tcx`]: ./ty.html +[#18290]: https://github.com/rust-lang/rust/issues/18290 +[#22019]: https://github.com/rust-lang/rust/issues/22019 diff --git a/src/traits-associated-types.md b/src/traits-associated-types.md index 6ac3fc512..c91dc255f 100644 --- a/src/traits-associated-types.md +++ b/src/traits-associated-types.md @@ -20,6 +20,8 @@ that syntax is expanded during ["type collection"](./type-checking.html) into the explicit form, though that is something we may want to change in the future.) +[intoiter-item]: https://doc.rust-lang.org/nightly/core/iter/trait.IntoIterator.html#associatedtype.Item + In some cases, associated type projections can be **normalized** -- @@ -51,8 +53,8 @@ we saw above would be lowered to a program clause like so: forall { Normalize( as IntoIterator>::Item -> T) - } - + } + (An aside: since we do not permit quantification over traits, this is really more like a family of predicates, one for each associated type.) @@ -98,7 +100,7 @@ We now introduce the `ProjectionEq` predicate to bring those two cases together. The `ProjectionEq` predicate looks like so: ProjectionEq(::Item = U) - + and we will see that it can be proven *either* via normalization or skolemization. As part of lowering an associated type declaration from some trait, we create two program clauses for `ProjectionEq`: @@ -123,7 +125,7 @@ with unification. As described in the basically a procedure with a signature like this: Unify(A, B) = Result<(Subgoals, RegionConstraints), NoSolution> - + In other words, we try to unify two things A and B. That procedure might just fail, in which case we get back `Err(NoSolution)`. This would happen, for example, if we tried to unify `u32` and `i32`. diff --git a/src/type-inference.md b/src/type-inference.md index 3795ea70e..c7a9c2b63 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -84,7 +84,7 @@ below in a separate section. The most basic operations you can perform in the type inferencer is **equality**, which forces two types `T` and `U` to be the same. The -recommended way to add an equality constraint is using the `at` +recommended way to add an equality constraint is to use the `at` method, roughly like so: ```rust From dbc65d25202298984c42b575d37e10f6669bc927 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Fri, 23 Feb 2018 21:07:36 +0000 Subject: [PATCH 163/648] Added EditorConfig file for `src`. --- .editorconfig | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 .editorconfig diff --git a/.editorconfig b/.editorconfig new file mode 100644 index 000000000..cb3312517 --- /dev/null +++ b/.editorconfig @@ -0,0 +1,9 @@ +root = true + +[src/*] +end_of_line = lf +insert_final_newline = true + +[ci/*.sh] +indent_style = space +indent_size = 2 From 61a3f536d1f0e6793d1d9a984a79ac28a4b736f1 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Fri, 23 Feb 2018 21:51:29 +0000 Subject: [PATCH 164/648] Added check for all source files to ensure they have no lines longer than 80 chars. --- .travis.yml | 2 ++ ci/check_line_lengths.sh | 19 +++++++++++++++++++ ci/install.sh | 1 + 3 files changed, 22 insertions(+) create mode 100755 ci/check_line_lengths.sh mode change 100644 => 100755 ci/install.sh diff --git a/.travis.yml b/.travis.yml index 5ca97ba79..da9c1ed0a 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,6 +1,8 @@ language: rust cache: - cargo +before_install: +- MAX_LINE_LENGTH=80 bash ci/check_line_lengths.sh src/**/*.md install: - source ~/.cargo/env || true - bash ci/install.sh diff --git a/ci/check_line_lengths.sh b/ci/check_line_lengths.sh new file mode 100755 index 000000000..8001ea9a7 --- /dev/null +++ b/ci/check_line_lengths.sh @@ -0,0 +1,19 @@ +#!/bin/bash + +echo "Checking line lengths in all source files <= $MAX_LINE_LENGTH chars..." + +echo "Offending files and lines:" +(( success = 1 )) +for file in "$@" ; do + echo "$file" + (( line_no = 0 )) + while IFS="" read -r line || [[ -n "$line" ]] ; do + (( line_no++ )) + if (( "${#line}" > $MAX_LINE_LENGTH )) ; then + (( success = 0 )) + echo -e "\t$line_no : $line" + fi + done < "$file" +done + +(( $success )) && echo "No offending lines found." diff --git a/ci/install.sh b/ci/install.sh old mode 100644 new mode 100755 index 975857a95..ea90e853d --- a/ci/install.sh +++ b/ci/install.sh @@ -1,4 +1,5 @@ #!/bin/bash + set -ex function cargo_install() { From ba057b34ad6ab2cdaae5dd72488b150b8888bc4b Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Sat, 24 Feb 2018 01:43:36 +0000 Subject: [PATCH 165/648] Ignore line check on certain types of lines or in certain blocks. --- ci/check_line_lengths.sh | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/ci/check_line_lengths.sh b/ci/check_line_lengths.sh index 8001ea9a7..27711806a 100755 --- a/ci/check_line_lengths.sh +++ b/ci/check_line_lengths.sh @@ -3,17 +3,23 @@ echo "Checking line lengths in all source files <= $MAX_LINE_LENGTH chars..." echo "Offending files and lines:" -(( success = 1 )) +(( bad_lines = 0 )) +(( inside_block = 0 )) for file in "$@" ; do echo "$file" (( line_no = 0 )) while IFS="" read -r line || [[ -n "$line" ]] ; do (( line_no++ )) - if (( "${#line}" > $MAX_LINE_LENGTH )) ; then - (( success = 0 )) + if [[ "$line" =~ ^'```' ]] ; then + (( inside_block = !$inside_block )) + continue + fi + if ! (( $inside_block )) && ! [[ "$line" =~ " | "|"://"|\[\^[^\ ]+\]: ]] && (( "${#line}" > $MAX_LINE_LENGTH )) ; then + (( bad_lines++ )) echo -e "\t$line_no : $line" fi done < "$file" done -(( $success )) && echo "No offending lines found." +echo "$bad_lines offending lines found." +(( $bad_lines == 0 )) From e6acd30581b57c6e95579ce48c912dd889ee9353 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Sat, 24 Feb 2018 01:43:53 +0000 Subject: [PATCH 166/648] Hard-wrapped lines that are too long. --- ci/check_line_lengths.sh | 2 +- src/SUMMARY.md | 3 +- src/compiletest.md | 30 ++++++++-------- src/high-level-overview.md | 29 ++++++++-------- src/hir.md | 38 +++++++++++---------- src/how-to-build-and-run.md | 34 +++++++++++-------- src/incremental-compilation.md | 39 +++++++++++---------- src/query.md | 12 ++++--- src/tests/adding.md | 62 +++++++++++++++++++++------------- src/tests/intro.md | 39 +++++++++++---------- src/tests/running.md | 9 ++--- src/trait-hrtb.md | 8 +++-- src/trait-resolution.md | 28 ++++++++------- src/ty.md | 17 ++++++---- src/type-inference.md | 17 +++++----- 15 files changed, 206 insertions(+), 161 deletions(-) diff --git a/ci/check_line_lengths.sh b/ci/check_line_lengths.sh index 27711806a..b32906da6 100755 --- a/ci/check_line_lengths.sh +++ b/ci/check_line_lengths.sh @@ -14,7 +14,7 @@ for file in "$@" ; do (( inside_block = !$inside_block )) continue fi - if ! (( $inside_block )) && ! [[ "$line" =~ " | "|"://"|\[\^[^\ ]+\]: ]] && (( "${#line}" > $MAX_LINE_LENGTH )) ; then + if ! (( $inside_block )) && ! [[ "$line" =~ " | "|"-|-"|"://"|\[\^[^\ ]+\]: ]] && (( "${#line}" > $MAX_LINE_LENGTH )) ; then (( bad_lines++ )) echo -e "\t$line_no : $line" fi diff --git a/src/SUMMARY.md b/src/SUMMARY.md index da84350ea..3460d949d 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -6,7 +6,8 @@ - [The compiler testing framework](./tests/intro.md) - [Running tests](./tests/running.md) - [Adding new tests](./tests/adding.md) - - [Using `compiletest` + commands to control test execution](./compiletest.md) + - [Using `compiletest` + commands to control test + execution](./compiletest.md) - [Walkthrough: a typical contribution](./walkthrough.md) - [High-level overview of the compiler source](./high-level-overview.md) - [The Rustc Driver](./rustc-driver.md) diff --git a/src/compiletest.md b/src/compiletest.md index f07d3065a..0cfc6b39d 100644 --- a/src/compiletest.md +++ b/src/compiletest.md @@ -1,8 +1,10 @@ # `compiletest` ## Introduction -`compiletest` is the main test harness of the Rust test suite. It allows test authors to organize large numbers of tests (the -Rust compiler has many thousands), efficient test execution (parallel execution is supported), and allows the test author to -configure behavior and expected results of both individual and groups of tests. +`compiletest` is the main test harness of the Rust test suite. It allows +test authors to organize large numbers of tests (the Rust compiler has many +thousands), efficient test execution (parallel execution is supported), and +allows the test author to configure behavior and expected results of both +individual and groups of tests. `compiletest` tests may check test code for success, for failure or in some cases, even failure to compile. Tests are typically organized as a Rust source file with annotations in comments before and/or within the test code, which serve to @@ -12,17 +14,17 @@ testing framework, see [`this chapter`](./tests/intro.html) for additional backg The tests themselves are typically (but not always) organized into "suites"--for example, `run-pass`, a folder representing tests that should succeed, `run-fail`, a folder holding tests that should compile successfully, but return a failure (non-zero status), `compile-fail`, a folder holding tests that should fail to compile, and many more. The various -suites are defined in [src/tools/compiletest/src/common.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs) in the `pub struct Config` declaration. And a very good +suites are defined in [src/tools/compiletest/src/common.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs) in the `pub struct Config` declaration. And a very good introduction to the different suites of compiler tests along with details about them can be found in [`Adding new tests`](./tests/adding.html). ## Adding a new test file -Briefly, simply create your new test in the appropriate location under [src/test](https://github.com/rust-lang/rust/tree/master/src/test). No registration of test files is necessary as +Briefly, simply create your new test in the appropriate location under [src/test](https://github.com/rust-lang/rust/tree/master/src/test). No registration of test files is necessary as `compiletest` will scan the [src/test](https://github.com/rust-lang/rust/tree/master/src/test) subfolder recursively, and will execute any Rust source files it finds as tests. -See [`Adding new tests`](./tests/adding.html) for a complete guide on how to adding new tests. +See [`Adding new tests`](./tests/adding.html) for a complete guide on how to adding new tests. ## Header Commands Source file annotations which appear in comments near the top of the source file *before* any test code are known as header -commands. These commands can instruct `compiletest` to ignore this test, set expectations on whether it is expected to +commands. These commands can instruct `compiletest` to ignore this test, set expectations on whether it is expected to succeed at compiling, or what the test's return code is expected to be. Header commands (and their inline counterparts, Error Info commands) are described more fully [here](./tests/adding.html#header-commands-configuring-rustc). @@ -96,7 +98,7 @@ As a concrete example, here is the implementation for the `parse_failure_status( pub normalize_stderr: Vec<(String, String)>, + pub failure_status: i32, } - + impl TestProps { @@ -260,6 +261,7 @@ impl TestProps { run_pass: false, @@ -105,7 +107,7 @@ As a concrete example, here is the implementation for the `parse_failure_status( + failure_status: 101, } } - + @@ -383,6 +385,10 @@ impl TestProps { if let Some(rule) = config.parse_custom_normalization(ln, "normalize-stderr") { self.normalize_stderr.push(rule); @@ -115,12 +117,12 @@ As a concrete example, here is the implementation for the `parse_failure_status( + self.failure_status = code; + } }); - + for key in &["RUST_TEST_NOCAPTURE", "RUST_TEST_THREADS"] { @@ -488,6 +494,13 @@ impl Config { self.parse_name_directive(line, "pretty-compare-only") } - + + fn parse_failure_status(&self, line: &str) -> Option { + match self.parse_name_value_directive(line, "failure-status") { + Some(code) => code.trim().parse::().ok(), @@ -141,7 +143,7 @@ located in [src/tools/compiletest/src/runtest.rs](https://github.com/rust-lang/r ```diff @@ -295,11 +295,14 @@ impl<'test> TestCx<'test> { } - + fn check_correct_failure_status(&self, proc_res: &ProcRes) { - // The value the rust runtime returns on failure - const RUST_ERR: i32 = 101; @@ -160,7 +162,7 @@ located in [src/tools/compiletest/src/runtest.rs](https://github.com/rust-lang/r } @@ -320,7 +323,6 @@ impl<'test> TestCx<'test> { ); - + let proc_res = self.exec_compiled_test(); - if !proc_res.status.success() { @@ -172,7 +174,7 @@ located in [src/tools/compiletest/src/runtest.rs](https://github.com/rust-lang/r ); - panic!(); } - } + } ``` Note the use of `self.props.failure_status` to access the header command property. In tests which do not specify the failure status header command, `self.props.failure_status` will evaluate to the default value of 101 at the time of this writing. diff --git a/src/high-level-overview.md b/src/high-level-overview.md index 493656713..041136548 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -94,20 +94,21 @@ order to compile a Rust crate, these are the general steps that we take: 1. **Parsing input** - - this processes the `.rs` files and produces the AST ("abstract syntax tree") + - this processes the `.rs` files and produces the AST + ("abstract syntax tree") - the AST is defined in `syntax/ast.rs`. It is intended to match the lexical syntax of the Rust language quite closely. 2. **Name resolution, macro expansion, and configuration** - - once parsing is complete, we process the AST recursively, resolving paths - and expanding macros. This same process also processes `#[cfg]` nodes, and hence - may strip things out of the AST as well. + - once parsing is complete, we process the AST recursively, resolving + paths and expanding macros. This same process also processes `#[cfg]` + nodes, and hence may strip things out of the AST as well. 3. **Lowering to HIR** - Once name resolution completes, we convert the AST into the HIR, - or "high-level IR". The HIR is defined in `src/librustc/hir/`; that module also includes - the lowering code. - - The HIR is a lightly desugared variant of the AST. It is more processed than the - AST and more suitable for the analyses that follow. It is **not** required to match - the syntax of the Rust language. + or "high-level IR". The HIR is defined in `src/librustc/hir/`; + that module also includes the lowering code. + - The HIR is a lightly desugared variant of the AST. It is more processed + than the AST and more suitable for the analyses that follow. + It is **not** required to match the syntax of the Rust language. - As a simple example, in the **AST**, we preserve the parentheses that the user wrote, so `((1 + 2) + 3)` and `1 + 2 + 3` parse into distinct trees, even though they are equivalent. In the @@ -125,13 +126,13 @@ take: the types of expressions, the way to resolve methods, and so forth. - After type-checking, we can do other analyses, such as privacy checking. 4. **Lowering to MIR and post-processing** - - Once type-checking is done, we can lower the HIR into MIR ("middle IR"), which - is a **very** desugared version of Rust, well suited to the borrowck but also - certain high-level optimizations. + - Once type-checking is done, we can lower the HIR into MIR ("middle IR"), + which is a **very** desugared version of Rust, well suited to borrowck + but also to certain high-level optimizations. 5. **Translation to LLVM and LLVM optimizations** - From MIR, we can produce LLVM IR. - - LLVM then runs its various optimizations, which produces a number of `.o` files - (one for each "codegen unit"). + - LLVM then runs its various optimizations, which produces a number of + `.o` files (one for each "codegen unit"). 6. **Linking** - Finally, those `.o` files are linked together. diff --git a/src/hir.md b/src/hir.md index 41ed0ab20..f66468ffc 100644 --- a/src/hir.md +++ b/src/hir.md @@ -63,23 +63,25 @@ carry around references into the HIR, but rather to carry around *identifier numbers* (or just "ids"). Right now, you will find four sorts of identifiers in active use: -- `DefId` – primarily names "definitions" or top-level items. - - You can think of a `DefId` as shorthand for a very explicit and complete - path, like `std::collections::HashMap`. However, these paths are able to - name things that are not nameable in normal Rust (e.g. `impl`s), and they - also include extra information about the crate (such as its version number, - since two versions of the same crate can co-exist). - - A `DefId` really consists of two parts, a `CrateNum` (which identifies the - crate) and a `DefIndex` (which indexes into a list of items that is - maintained per crate). -- `HirId` – combines the index of a particular item with an offset within - that item. - - The key point of an `HirId` is that it is *relative* to some item (which is - named via a `DefId`). -- `BodyId` – an absolute identifier that refers to a specific body (definition - of a function or constant) in the crate. It is currently effectively a - "newtype'd" `NodeId`. -- `NodeId` – an absolute ID that identifies a single node in the HIR tree. +- `DefId`, which primarily names "definitions" or top-level items. + - You can think of a `DefId` as being shorthand for a very explicit + and complete path, like `std::collections::HashMap`. However, + these paths are able to name things that are not nameable in + normal Rust (e.g. impls), and they also include extra information + about the crate (such as its version number, as two versions of + the same crate can co-exist). + - A `DefId` really consists of two parts, a `CrateNum` (which + identifies the crate) and a `DefIndex` (which indixes into a list + of items that is maintained per crate). +- `HirId`, which combines the index of a particular item with an + offset within that item. + - the key point of a `HirId` is that it is *relative* to some item + (which is named via a `DefId`). +- `BodyId`, this is an absolute identifier that refers to a specific + body (definition of a function or constant) in the crate. It is currently + effectively a "newtype'd" `NodeId`. +- `NodeId`, which is an absolute id that identifies a single node in the HIR + tree. - While these are still in common use, **they are being slowly phased out**. - Since they are absolute within the crate, adding a new node anywhere in the tree causes the `NodeId`s of all subsequent code in the crate to change. @@ -119,5 +121,5 @@ of a function/closure or the definition of a constant. Bodies are associated with an **owner**, which is typically some kind of item (e.g. an `fn()` or `const`), but could also be a closure expression (e.g. `|x, y| x + y`). You can use the HIR map to find the body -associated with a given `DefId` (`maybe_body_owned_by()`) or to find +associated with a given def-id (`maybe_body_owned_by()`) or to find the owner of a body (`body_owner_def_id()`). diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 7b6088c7c..6e292934b 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -2,8 +2,9 @@ The compiler is built using a tool called `x.py`. You will need to have Python installed to run it. But before we get to that, if you're going to -be hacking on rustc, you'll want to tweak the configuration of the compiler. The default -configuration is oriented towards running the compiler as a user, not a developer. +be hacking on rustc, you'll want to tweak the configuration of the compiler. +The default configuration is oriented towards running the compiler as a user, +not a developer. ### Create a config.toml @@ -84,15 +85,16 @@ What this command will do is the following: - Using this stage 1 compiler, it will build the standard library. (this is what the `src/libstd`) means. -This is just a subset of the full rustc build. The **full** rustc build (what you -get if you just say `./x.py build`) has quite a few more steps: +This is just a subset of the full rustc build. The **full** rustc build +(what you get if you just say `./x.py build`) has quite a few more steps: -- Build stage1 rustc with stage0 compiler -- Build libstd with stage1 compiler (up to here is the same) -- Build rustc from `src` again, this time with the stage1 compiler (this part is new) - - The resulting compiler here is called the "stage2" compiler -- Build libstd with stage2 compiler -- Build librustdoc and a bunch of other things +- Build stage1 rustc with stage0 compiler. +- Build libstd with stage1 compiler (up to here is the same). +- Build rustc from `src` again, this time with the stage1 compiler + (this part is new). + - The resulting compiler here is called the "stage2" compiler. +- Build libstd with stage2 compiler. +- Build librustdoc and a bunch of other things. ### Creating a rustup toolchain @@ -126,12 +128,16 @@ LLVM version: 4.0 ### Other x.py commands -Here are a few other useful x.py commands. We'll cover some of them in detail in other sections: +Here are a few other useful x.py commands. We'll cover some of them in detail +in other sections: - Building things: - - `./x.py clean` – clean up the build directory (`rm -rf build` works too, but then you have to rebuild LLVM) - - `./x.py build --stage 1` – builds everything using the stage 1 compiler, not just up to libstd + - `./x.py clean` – clean up the build directory (`rm -rf build` works too, + but then you have to rebuild LLVM) + - `./x.py build --stage 1` – builds everything using the stage 1 compiler, + not just up to libstd - `./x.py build` – builds the stage2 compiler -- Running tests (see the [section on running tests](./tests/running.html) for more details): +- Running tests (see the [section on running tests](./tests/running.html) for + more details): - `./x.py test --stage 1 src/libstd` – runs the `#[test]` tests from libstd - `./x.py test --stage 1 src/test/run-pass` – runs the `run-pass` test suite diff --git a/src/incremental-compilation.md b/src/incremental-compilation.md index 6f5b48cca..a209207fc 100644 --- a/src/incremental-compilation.md +++ b/src/incremental-compilation.md @@ -2,8 +2,8 @@ The incremental compilation scheme is, in essence, a surprisingly simple extension to the overall query system. We'll start by describing -a slightly simplified variant of the real thing – the "basic algorithm" – and then describe -some possible improvements. +a slightly simplified variant of the real thing – the "basic algorithm" – +and then describe some possible improvements. ## The basic algorithm @@ -40,7 +40,7 @@ There are two key insights here: produced the same result as the previous time. **If it did,** we can still mark the query as green, and hence avoid re-executing dependent queries. - + ### The try-mark-green algorithm At the core of incremental compilation is an algorithm called @@ -66,13 +66,15 @@ Try-mark-green works as follows: - For each query R in `reads(Q)`, we recursively demand the color of R using try-mark-green. - Note: it is important that we visit each node in `reads(Q)` in same order - as they occurred in the original compilation. See [the section on the query DAG below](#dag). - - If **any** of the nodes in `reads(Q)` wind up colored **red**, then Q is dirty. - - We re-execute Q and compare the hash of its result to the hash of the result - from the previous compilation. + as they occurred in the original compilation. See [the section on the + query DAG below](#dag). + - If **any** of the nodes in `reads(Q)` wind up colored **red**, then Q is + dirty. + - We re-execute Q and compare the hash of its result to the hash of the + result from the previous compilation. - If the hash has not changed, we can mark Q as **green** and return. - - Otherwise, **all** of the nodes in `reads(Q)` must be **green**. In that case, - we can color Q as **green** and return. + - Otherwise, **all** of the nodes in `reads(Q)` must be **green**. In that + case, we can color Q as **green** and return. @@ -80,14 +82,14 @@ Try-mark-green works as follows: The query DAG code is stored in [`src/librustc/dep_graph`][dep_graph]. Construction of the DAG is done -by instrumenting the query execution. +by instrumenting the query execution. One key point is that the query DAG also tracks ordering; that is, for each query Q, we not only track the queries that Q reads, we track the **order** in which they were read. This allows try-mark-green to walk -those queries back in the same order. This is important because once a subquery comes back as red, -we can no longer be sure that Q will continue along the same path as before. -That is, imagine a query like this: +those queries back in the same order. This is important because once a +subquery comes back as red, we can no longer be sure that Q will continue +along the same path as before. That is, imagine a query like this: ```rust,ignore fn main_query(tcx) { @@ -105,9 +107,10 @@ query `main_query` executes will be `subquery2`, and `subquery3` will not be executed at all. But now imagine that in the **next** compilation, the input has -changed such that `subquery1` returns **false**. In this case, `subquery2` would never -execute. If try-mark-green were to visit `reads(main_query)` out of order, -however, it might visit `subquery2` before `subquery1`, and hence execute it. +changed such that `subquery1` returns **false**. In this case, `subquery2` +would never execute. If try-mark-green were to visit `reads(main_query)` out +of order, however, it might visit `subquery2` before `subquery1`, and hence +execute it. This can lead to ICEs and other problems in the compiler. [dep_graph]: https://github.com/rust-lang/rust/tree/master/src/librustc/dep_graph @@ -124,8 +127,8 @@ we **also** save the results. This is why the incremental algorithm separates computing the **color** of a node, which often does not require its value, from -computing the **result** of a node. Computing the result is done via a simple algorithm -like so: +computing the **result** of a node. Computing the result is done via a simple +algorithm like so: - Check if a saved result for Q is available. If so, compute the color of Q. If Q is green, deserialize and return the saved result. diff --git a/src/query.md b/src/query.md index 33d2b3bac..500b9dec8 100644 --- a/src/query.md +++ b/src/query.md @@ -74,10 +74,12 @@ match queries::type_of::try_get(tcx, DUMMY_SP, self.did) { } ``` -So, if you get back an `Err` from `try_get`, then a cycle *did* occur. This means that -you must ensure that a compiler error message is reported. You can do that in two ways: +So, if you get back an `Err` from `try_get`, then a cycle *did* occur. This +means that you must ensure that a compiler error message is reported. You can +do that in two ways: -The simplest is to invoke `err.emit()`. This will emit the cycle error to the user. +The simplest is to invoke `err.emit()`. This will emit the cycle error to the +user. However, often cycles happen because of an illegal program, and you know at that point that an error either already has been reported or @@ -192,8 +194,8 @@ fn fubar<'cx, 'tcx>(tcx: TyCtxt<'cx, 'tcx>, key: DefId) -> Fubar<'tcx> { .. } N.B. Most of the `rustc_*` crates only provide **local providers**. Almost all **extern providers** wind up going through the -[`rustc_metadata` crate][rustc_metadata], which loads the information from the crate -metadata. But in some cases there are crates that provide queries for +[`rustc_metadata` crate][rustc_metadata], which loads the information from the +crate metadata. But in some cases there are crates that provide queries for *both* local and external crates, in which case they define both a `provide` and a `provide_extern` function that `rustc_driver` can invoke. diff --git a/src/tests/adding.md b/src/tests/adding.md index 276189f59..f777a458c 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -12,8 +12,8 @@ structure: - They always begin with the [copyright notice](./conventions.html#copyright); - then they should have some kind of [comment explaining what the test is about](#explanatory_comment); -- next, they can have one or more [header commands](#header_commands), which are special - comments that the test interpreter knows how to interpret. +- next, they can have one or more [header commands](#header_commands), which + are special comments that the test interpreter knows how to interpret. - finally, they have the Rust source. This may have various [error annotations](#error_annotations) which indicate expected compilation errors or warnings. @@ -28,10 +28,12 @@ rough heuristics: - Some tests have specialized needs: - need to run gdb or lldb? use the `debuginfo` test suite - - need to inspect LLVM IR or MIR IR? use the `codegen` or `mir-opt` test suites + - need to inspect LLVM IR or MIR IR? use the `codegen` or `mir-opt` test + suites - need to run rustdoc? Prefer a `rustdoc` test - need to inspect the resulting binary in some way? Then use `run-make` -- For most other things, [a `ui` (or `ui-fulldeps`) test](#ui) is to be preferred: +- For most other things, [a `ui` (or `ui-fulldeps`) test](#ui) is to be + preferred: - `ui` tests subsume both run-pass, compile-fail, and parse-fail tests - in the case of warnings or errors, `ui` tests capture the full output, which makes it easier to review but also helps prevent "hidden" regressions @@ -59,8 +61,8 @@ When writing a new feature, **create a subdirectory to store your tests**. For example, if you are implementing RFC 1234 ("Widgets"), then it might make sense to put the tests in directories like: -- `src/test/ui/rfc1234-widgets/` -- `src/test/run-pass/rfc1234-widgets/` +- `src/test/ui/rfc1234-widgets/` +- `src/test/run-pass/rfc1234-widgets/` - etc In other cases, there may already be a suitable directory. (The proper @@ -118,16 +120,22 @@ fn main() { These are used to ignore the test in some situations, which means the test won't be compiled or run. -* `ignore-X` where `X` is a target detail or stage will ignore the test accordingly (see below) -* `ignore-pretty` will not compile the pretty-printed test (this is done to test the pretty-printer, but might not always work) +* `ignore-X` where `X` is a target detail or stage will ignore the + test accordingly (see below) +* `ignore-pretty` will not compile the pretty-printed test (this is + done to test the pretty-printer, but might not always work) * `ignore-test` always ignores the test -* `ignore-lldb` and `ignore-gdb` will skip a debuginfo test on that debugger. +* `ignore-lldb` and `ignore-gdb` will skip a debuginfo test on that + debugger. Some examples of `X` in `ignore-X`: -* Architecture: `aarch64`, `arm`, `asmjs`, `mips`, `wasm32`, `x86_64`, `x86`, ... -* OS: `android`, `emscripten`, `freebsd`, `ios`, `linux`, `macos`, `windows`, ... -* Environment (fourth word of the target triple): `gnu`, `msvc`, `musl`. +* Architecture: `aarch64`, `arm`, `asmjs`, `mips`, `wasm32`, `x86_64`, + `x86`, ... +* OS: `android`, `emscripten`, `freebsd`, `ios`, `linux`, `macos`, + `windows`, ... +* Environment (fourth word of the target triple): `gnu`, `msvc`, + `musl`. * Pointer width: `32bit`, `64bit`. * Stage: `stage0`, `stage1`, `stage2`. @@ -140,17 +148,20 @@ source. * `min-{gdb,lldb}-version` * `min-llvm-version` -* `must-compile-successfully` for UI tests, indicates that the test is supposed - to compile, as opposed to the default where the test is supposed to error out. +* `must-compile-successfully` for UI tests, indicates that the test is + supposed to compile, as opposed to the default where the test is + supposed to error out. * `compile-flags` passes extra command-line args to the compiler, e.g. `compile-flags -g` which forces debuginfo to be enabled. -* `should-fail` indicates that the test should fail; used for "meta testing", - where we test the compiletest program itself to check that it will generate - errors in appropriate scenarios. This header is ignored for pretty-printer tests. -* `gate-test-X` where `X` is a feature marks the test as "gate test" for feature X. - Such tests are supposed to ensure that the compiler errors when usage of a gated - feature is attempted without the proper `#![feature(X)]` tag. - Each unstable lang feature is required to have a gate test. +* `should-fail` indicates that the test should fail; used for "meta + testing", where we test the compiletest program itself to check that + it will generate errors in appropriate scenarios. This header is + ignored for pretty-printer tests. +* `gate-test-X` where `X` is a feature marks the test as "gate test" + for feature X. Such tests are supposed to ensure that the compiler + errors when usage of a gated feature is attempted without the proper + `#![feature(X)]` tag. Each unstable lang feature is required to + have a gate test. [`header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs @@ -245,8 +256,10 @@ can also make UI tests where compilation is expected to succeed, and you can even run the resulting program. Just add one of the following [header commands](#header_commands): -- `// must-compile-successfully` -- compilation should succeed but do not run the resulting binary -- `// run-pass` -- compilation should succeed and we should run the resulting binary +- `// must-compile-successfully` -- compilation should succeed but do + not run the resulting binary +- `// run-pass` -- compilation should succeed and we should run the + resulting binary ### Editing and updating the reference files @@ -293,7 +306,8 @@ The corresponding reference file will use the normalized output to test both ... ``` -Please see [`ui/transmute/main.rs`][mrs] and [`main.stderr`][] for a concrete usage example. +Please see [`ui/transmute/main.rs`][mrs] and [`main.stderr`][] for a +concrete usage example. [mrs]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.rs [`main.stderr`]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.stderr diff --git a/src/tests/intro.md b/src/tests/intro.md index eb7294c83..e8cf34aff 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -1,12 +1,12 @@ # The compiler testing framework -The Rust project runs a wide variety of different tests, orchestrated by the -build system (`x.py test`). The main test harness for testing the compiler -itself is a tool called compiletest (sources in the -[`src/tools/compiletest`]). This section gives a brief overview of how the -testing framework is setup, and then gets into some of the details on [how to -run tests](./tests/running.html#ui) as well as [how to add new -tests](./tests/adding.html). +The Rust project runs a wide variety of different tests, orchestrated +by the build system (`x.py test`). The main test harness for testing +the compiler itself is a tool called compiletest (sources in the +[`src/tools/compiletest`]). This section gives a brief overview of how +the testing framework is setup, and then gets into some of the details +on [how to run tests](./tests/running.html#ui) as well as +[how to add new tests](./tests/adding.html). [`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest @@ -24,11 +24,13 @@ Here is a brief summary of the test suites as of this writing and what they mean. In some cases, the test suites are linked to parts of the manual that give more details. -- [`ui`](./tests/adding.html#ui) -- tests that check the exact stdout/stderr from compilation - and/or running the test -- `run-pass` -- tests that are expected to compile and execute successfully (no panics) +- [`ui`](./tests/adding.html#ui) -- tests that check the exact + stdout/stderr from compilation and/or running the test +- `run-pass` -- tests that are expected to compile and execute + successfully (no panics) - `run-pass-valgrind` -- tests that ought to run with valgrind -- `run-fail` -- tests that are expected to compile but then panic during execution +- `run-fail` -- tests that are expected to compile but then panic + during execution - `compile-fail` -- tests that are expected to fail compilation. - `parse-fail` -- tests that are expected to fail to parse - `pretty` -- tests targeting the Rust "pretty printer", which @@ -44,19 +46,20 @@ that give more details. results from previous compilations. - `run-make` -- tests that basically just execute a `Makefile`; the ultimate in flexibility but quite annoying to write. -- `rustdoc` -- tests for rustdoc, making sure that the generated files contain - the expected documentation. -- `*-fulldeps` -- same as above, but indicates that the test depends on things other - than `libstd` (and hence those things must be built) +- `rustdoc` -- tests for rustdoc, making sure that the generated files + contain the expected documentation. +- `*-fulldeps` -- same as above, but indicates that the test depends + on things other than `libstd` (and hence those things must be built) ## Other Tests The Rust build system handles running tests for various other things, including: -- **Tidy** -- This is a custom tool used for validating source code style and - formatting conventions, such as rejecting long lines. There is more - information in the [section on coding conventions](./conventions.html#formatting). +- **Tidy** -- This is a custom tool used for validating source code + style and formatting conventions, such as rejecting long lines. + There is more information in the + [section on coding conventions](./conventions.html#formatting). Example: `./x.py test src/tools/tidy` diff --git a/src/tests/running.md b/src/tests/running.md index 98d590671..6bac0121a 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -24,10 +24,11 @@ generally working correctly would be the following: ./x.py test --stage 1 src/test/{ui,compile-fail,run-pass} ``` -This will run the `ui`, `compile-fail`, and `run-pass` test suites, and -only with the stage 1 build. Of course, the choice of test suites is somewhat -arbitrary, and may not suit the task you are doing. For example, if you are hacking -on debuginfo, you may be better off with the debuginfo test suite: +This will run the `ui`, `compile-fail`, and `run-pass` test suites, +and only with the stage 1 build. Of course, the choice of test suites +is somewhat arbitrary, and may not suit the task you are doing. For +example, if you are hacking on debuginfo, you may be better off with +the debuginfo test suite: ```bash ./x.py test --stage 1 src/test/debuginfo diff --git a/src/trait-hrtb.md b/src/trait-hrtb.md index 8c3a6f4e9..d677db2c5 100644 --- a/src/trait-hrtb.md +++ b/src/trait-hrtb.md @@ -32,7 +32,7 @@ impl<'a> Foo<&'a isize> for AnyInt { } And the question is, does `AnyInt : for<'a> Foo<&'a isize>`? We want the answer to be yes. The algorithm for figuring it out is closely related -to the subtyping for higher-ranked types (which is described in [here][hrsubtype] +to the subtyping for higher-ranked types (which is described [here][hrsubtype] and also in a [paper by SPJ]. If you wish to understand higher-ranked subtyping, we recommend you read the paper). There are a few parts: @@ -83,7 +83,8 @@ skolemized to `'0` and the impl trait reference is instantiated to like `'static == '0`. This means that the taint set for `'0` is `{'0, 'static}`, which fails the leak check. -**TODO**: This is because `'static` is not a region variable but is in the taint set, right? +**TODO**: This is because `'static` is not a region variable but is in the +taint set, right? ## Higher-ranked trait obligations @@ -122,4 +123,5 @@ from. (This is done in `higher_ranked::plug_leaks`). We know that the leak check passed, so this taint set consists solely of the skolemized region itself plus various intermediate region variables. We then walk the trait-reference and convert every region in that taint set back to -a late-bound region, so in this case we'd wind up with `Baz: for<'a> Bar<&'a isize>`. +a late-bound region, so in this case we'd wind up with +`Baz: for<'a> Bar<&'a isize>`. diff --git a/src/trait-resolution.md b/src/trait-resolution.md index d6edcf68a..7494d969a 100644 --- a/src/trait-resolution.md +++ b/src/trait-resolution.md @@ -58,20 +58,20 @@ will then be generated in the output binary. Trait resolution consists of three major parts: -- **Selection** is deciding how to resolve a specific obligation. For +- **Selection**: Deciding how to resolve a specific obligation. For example, selection might decide that a specific obligation can be - resolved by employing an impl which matches the `Self` type, or by - using a parameter bound (e.g. `T: Trait`). In the case of an impl, selecting one + resolved by employing an impl which matches the `Self` type, or by using a + parameter bound (e.g. `T: Trait`). In the case of an impl, selecting one obligation can create *nested obligations* because of where clauses on the impl itself. It may also require evaluating those nested obligations to resolve ambiguities. -- **Fulfillment** is keeping track of which obligations - are completely fulfilled. Basically, it is a worklist of obligations +- **Fulfillment**: The fulfillment code is what tracks that obligations + are completely fulfilled. Basically it is a worklist of obligations to be selected: once selection is successful, the obligation is removed from the worklist and any nested obligations are enqueued. -- **Coherence** checks are intended to ensure that there +- **Coherence**: The coherence checks are intended to ensure that there are never overlapping impls, where two impls could be used with equal precedence. @@ -174,8 +174,12 @@ select this impl, which will cause the type of `$Y` to be unified to `usize`. (Note that while assembling candidates, we do the initial unifications in a transaction, so that they don't affect one another.) -**TODO**: The example says we can "select" the impl, but this section is talking specifically about candidate assembly. Does this mean we can sometimes skip confirmation? Or is this poor wording? -**TODO**: Is the unification of `$Y` part of trait resolution or type inference? Or is this not the same type of "inference variable" as in type inference? +**TODO**: The example says we can "select" the impl, but this section is +talking specifically about candidate assembly. Does this mean we can sometimes +skip confirmation? Or is this poor wording? +**TODO**: Is the unification of `$Y` part of trait resolution or type +inference? Or is this not the same type of "inference variable" as in type +inference? #### Winnowing: Resolving ambiguities @@ -282,10 +286,10 @@ to a particular impl. One interesting twist has to do with nested obligations. In general, in trans, we only need to do a "shallow" selection for an obligation. That is, we wish to identify which impl applies, but we do not (yet) need to decide how to select -any nested obligations. Nonetheless, we *do* currently do a complete -resolution, and that is because it can sometimes inform the results of type -inference. That is, we do not have the full substitutions for the type -variables of the impl available to us, so we must run trait selection to figure +any nested obligations. Nonetheless, we *do* currently do a complete resolution, +and that is because it can sometimes inform the results of type inference. +That is, we do not have the full substitutions in terms of the type variables +of the impl available to us, so we must run trait selection to figure everything out. **TODO**: is this still talking about trans? diff --git a/src/ty.md b/src/ty.md index 26b8c1ca8..1fd86a6bd 100644 --- a/src/ty.md +++ b/src/ty.md @@ -56,8 +56,8 @@ fn not_in_inference<'a, 'tcx>(tcx: TyCtxt<'a, 'tcx, 'tcx>, def_id: DefId) { } ``` -In contrast, if we want to code that can be usable during type inference, then you -need to declare a distinct `'gcx` and `'tcx` lifetime parameter: +In contrast, if we want to code that can be usable during type inference, then +you need to declare a distinct `'gcx` and `'tcx` lifetime parameter: ```rust fn maybe_in_inference<'a, 'gcx, 'tcx>(tcx: TyCtxt<'a, 'gcx, 'tcx>, def_id: DefId) { @@ -141,19 +141,22 @@ In addition to types, there are a number of other arena-allocated data structures that you can allocate, and which are found in this module. Here are a few examples: -- `Substs`, allocated with `mk_substs` – this will intern a slice of types, often used to - specify the values to be substituted for generics (e.g. `HashMap` - would be represented as a slice `&'tcx [tcx.types.i32, tcx.types.u32]`). +- `Substs`, allocated with `mk_substs` – this will intern a slice of types, + often used to specify the values to be substituted for generics + (e.g. `HashMap` would be represented as a slice + `&'tcx [tcx.types.i32, tcx.types.u32]`). - `TraitRef`, typically passed by value – a **trait reference** consists of a reference to a trait along with its various type parameters (including `Self`), like `i32: Display` (here, the def-id would reference the `Display` trait, and the substs would contain `i32`). -- `Predicate` defines something the trait system has to prove (see `traits` module). +- `Predicate` defines something the trait system has to prove (see `traits` + module). ### Import conventions -Although there is no hard and fast rule, the `ty` module tends to be used like so: +Although there is no hard and fast rule, the `ty` module tends to be used like +so: ```rust use ty::{self, Ty, TyCtxt}; diff --git a/src/type-inference.md b/src/type-inference.md index c7a9c2b63..45f9df18a 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -59,13 +59,14 @@ inference works, or perhaps this blog post on [Unification in the Chalk project]: http://smallcultfollowing.com/babysteps/blog/2017/03/25/unification-in-chalk-part-1/ -All said, the inference context stores four kinds of inference variables as of -writing: +All told, the inference context stores four kinds of inference variables as of +this writing: - Type variables, which come in three varieties: - - General type variables (the most common). These can be unified with any type. - - Integral type variables, which can only be unified with an integral type, and - arise from an integer literal expression like `22`. + - General type variables (the most common). These can be unified with any + type. + - Integral type variables, which can only be unified with an integral type, + and arise from an integer literal expression like `22`. - Float type variables, which can only be unified with a float type, and arise from a float literal expression like `22.0`. - Region variables, which represent lifetimes, and arise all over the place. @@ -177,7 +178,7 @@ form of an "outlives" constraint: 'a: 'b -Actually, the code tends to view them as a subregion relation, but it's the same +Actually the code tends to view them as a subregion relation, but it's the same idea: 'b <= 'a @@ -185,8 +186,8 @@ idea: (There are various other kinds of constriants, such as "verifys"; see the `region_constraints` module for details.) -There is one case where we do some amount of eager unification. If you have an equality constraint -between two regions +There is one case where we do some amount of eager unification. If you have an +equality constraint between two regions 'a = 'b From aeef4cd9b33f5a948a3022266bee601923f93470 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 20 Mar 2018 09:48:17 -0400 Subject: [PATCH 167/648] make the script give more useful feedback --- ci/check_line_lengths.sh | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/ci/check_line_lengths.sh b/ci/check_line_lengths.sh index b32906da6..dc48091c8 100755 --- a/ci/check_line_lengths.sh +++ b/ci/check_line_lengths.sh @@ -1,5 +1,21 @@ #!/bin/bash +if [ "$1" == "--help" ]; then + echo 'Usage:' + echo ' MAX_LINE_LENGTH=80' "$0" 'src/**/*.md' + exit 1 +fi + +if [ "$MAX_LINE_LENGTH" == "" ]; then + echo '`MAX_LINE_LENGTH` environment variable not set. Try --help.' + exit 1 +fi + +if [ "$1" == "" ]; then + echo 'No files provided.' + exit 1 +fi + echo "Checking line lengths in all source files <= $MAX_LINE_LENGTH chars..." echo "Offending files and lines:" From 193a2bf42e1b872db09fd6d4f6c01c5b0195116c Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Thu, 22 Mar 2018 02:05:28 +0000 Subject: [PATCH 168/648] Travis CI script now sets `globstar` shell option. --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index da9c1ed0a..45ba4a890 100644 --- a/.travis.yml +++ b/.travis.yml @@ -2,6 +2,7 @@ language: rust cache: - cargo before_install: +- shopt -s globstar - MAX_LINE_LENGTH=80 bash ci/check_line_lengths.sh src/**/*.md install: - source ~/.cargo/env || true From cd053f3cb405d6127833fc097d1a25cb54b85f2a Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 23 Mar 2018 20:56:05 -0500 Subject: [PATCH 169/648] Fix long lines in glossary --- src/appendix-glossary.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/appendix-glossary.md b/src/appendix-glossary.md index 8a3df24e8..1d82686ba 100644 --- a/src/appendix-glossary.md +++ b/src/appendix-glossary.md @@ -1,6 +1,8 @@ # Appendix C: Glossary -The compiler uses a number of...idiosyncratic abbreviations and things. This glossary attempts to list them and give you a few pointers for understanding them better. +The compiler uses a number of...idiosyncratic abbreviations and things. This +glossary attempts to list them and give you a few pointers for understanding +them better. Term | Meaning ------------------------|-------- From 8615d9e9b0f1997af4398fa6798c180886b18b60 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 23 Mar 2018 20:57:17 -0500 Subject: [PATCH 170/648] Fix long lines in stupid stats appendix --- src/appendix-stupid-stats.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/appendix-stupid-stats.md b/src/appendix-stupid-stats.md index 20d5aaf9b..8e50b2c31 100644 --- a/src/appendix-stupid-stats.md +++ b/src/appendix-stupid-stats.md @@ -73,8 +73,8 @@ and a bunch of other crates with the 'librustc_' prefix. Next is translation, this translates the AST (and all those side tables) into LLVM IR (intermediate representation). We do this by calling into the LLVM -libraries, rather than actually writing IR directly to a file. The code for this is in -[librustc_trans](https://github.com/rust-lang/rust/tree/master/src/librustc_trans). +libraries, rather than actually writing IR directly to a file. The code for +this is in [librustc_trans](https://github.com/rust-lang/rust/tree/master/src/librustc_trans). The next phase is running the LLVM backend. This runs LLVM's optimisation passes on the generated IR and then generates machine code. The result is object files. @@ -117,8 +117,8 @@ I'll summarise the methods here. `early_callback` and `late_callback` let you call arbitrary code at different points - early is after command line arguments have been parsed, but before anything is done with them; late is pretty much the last thing before -compilation starts, i.e., after all processing of command line arguments, etc. is -done. Currently, you get to choose whether compilation stops or continues at +compilation starts, i.e., after all processing of command line arguments, etc. +is done. Currently, you get to choose whether compilation stops or continues at each point, but you don't get to change anything the driver has done. You can record some info for later, or perform other actions of your own. @@ -402,4 +402,4 @@ analysis, rather than doing its own analysis). Other parts of the compiler internally (I already changed save-analysis to use `CompilerController`). I've been experimenting with a prototype rustfmt which also uses these APIs. -[stupid-stats]: https://github.com/nrc/stupid-stats \ No newline at end of file +[stupid-stats]: https://github.com/nrc/stupid-stats From 94757096adefa565f96a9f8c02387ede917031c3 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 23 Mar 2018 21:01:59 -0500 Subject: [PATCH 171/648] Fix long lines in compiletest --- src/compiletest.md | 175 +++++++++++++++++++++++++++++---------------- 1 file changed, 114 insertions(+), 61 deletions(-) diff --git a/src/compiletest.md b/src/compiletest.md index 0cfc6b39d..011304d45 100644 --- a/src/compiletest.md +++ b/src/compiletest.md @@ -1,42 +1,66 @@ # `compiletest` + ## Introduction + `compiletest` is the main test harness of the Rust test suite. It allows test authors to organize large numbers of tests (the Rust compiler has many thousands), efficient test execution (parallel execution is supported), and allows the test author to configure behavior and expected results of both individual and groups of tests. -`compiletest` tests may check test code for success, for failure or in some cases, even failure to compile. Tests are -typically organized as a Rust source file with annotations in comments before and/or within the test code, which serve to -direct `compiletest` on if or how to run the test, what behavior to expect, and more. If you are unfamiliar with the compiler -testing framework, see [`this chapter`](./tests/intro.html) for additional background. - -The tests themselves are typically (but not always) organized into "suites"--for example, `run-pass`, a folder -representing tests that should succeed, `run-fail`, a folder holding tests that should compile successfully, but return -a failure (non-zero status), `compile-fail`, a folder holding tests that should fail to compile, and many more. The various -suites are defined in [src/tools/compiletest/src/common.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs) in the `pub struct Config` declaration. And a very good -introduction to the different suites of compiler tests along with details about them can be found in [`Adding new tests`](./tests/adding.html). +`compiletest` tests may check test code for success, for failure or in some +cases, even failure to compile. Tests are typically organized as a Rust source +file with annotations in comments before and/or within the test code, which +serve to direct `compiletest` on if or how to run the test, what behavior to +expect, and more. If you are unfamiliar with the compiler testing framework, +see [`this chapter`](./tests/intro.html) for additional background. + +The tests themselves are typically (but not always) organized into +"suites"--for example, `run-pass`, a folder representing tests that should +succeed, `run-fail`, a folder holding tests that should compile successfully, +but return a failure (non-zero status), `compile-fail`, a folder holding tests +that should fail to compile, and many more. The various suites are defined in +[src/tools/compiletest/src/common.rs][common] in the `pub struct Config` +declaration. And a very good introduction to the different suites of compiler +tests along with details about them can be found in [`Adding new +tests`](./tests/adding.html). ## Adding a new test file -Briefly, simply create your new test in the appropriate location under [src/test](https://github.com/rust-lang/rust/tree/master/src/test). No registration of test files is necessary as -`compiletest` will scan the [src/test](https://github.com/rust-lang/rust/tree/master/src/test) subfolder recursively, and will execute any Rust source files it finds as tests. -See [`Adding new tests`](./tests/adding.html) for a complete guide on how to adding new tests. + +Briefly, simply create your new test in the appropriate location under +[src/test][test]. No registration of test files is necessary as `compiletest` +will scan the [src/test][test] subfolder recursively, and will execute any Rust +source files it finds as tests. See [`Adding new tests`](./tests/adding.html) +for a complete guide on how to adding new tests. ## Header Commands -Source file annotations which appear in comments near the top of the source file *before* any test code are known as header -commands. These commands can instruct `compiletest` to ignore this test, set expectations on whether it is expected to -succeed at compiling, or what the test's return code is expected to be. Header commands (and their inline counterparts, -Error Info commands) are described more fully [here](./tests/adding.html#header-commands-configuring-rustc). + +Source file annotations which appear in comments near the top of the source +file *before* any test code are known as header commands. These commands can +instruct `compiletest` to ignore this test, set expectations on whether it is +expected to succeed at compiling, or what the test's return code is expected to +be. Header commands (and their inline counterparts, Error Info commands) are +described more fully +[here](./tests/adding.html#header-commands-configuring-rustc). ### Adding a new header command -Header commands are defined in the `TestProps` struct in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs). At a high level, there are dozens of test properties defined here, all set to default values in the `TestProp` struct's `impl` block. Any test can override this -default value by specifying the property in question as header command as a comment (`//`) in the test source file, before any source code. + +Header commands are defined in the `TestProps` struct in +[src/tools/compiletest/src/header.rs][header]. At a high level, there are +dozens of test properties defined here, all set to default values in the +`TestProp` struct's `impl` block. Any test can override this default value by +specifying the property in question as header command as a comment (`//`) in +the test source file, before any source code. #### Using a header command -Here is an example, specifying the `must-compile-successfully` header command, which takes no arguments, followed by the -`failure-status` header command, which takes a single argument (which, in this case is a value of 1). `failure-status` is -instructing `compiletest` to expect a failure status of 1 (rather than the current Rust default of 101 at the time of this -writing). The header command and the argument list (if present) are typically separated by a colon: + +Here is an example, specifying the `must-compile-successfully` header command, +which takes no arguments, followed by the `failure-status` header command, +which takes a single argument (which, in this case is a value of 1). +`failure-status` is instructing `compiletest` to expect a failure status of 1 +(rather than the current Rust default of 101 at the time of this writing). The +header command and the argument list (if present) are typically separated by a +colon: ``` // Copyright 2018 The Rust Project Developers. See the COPYRIGHT // file at the top-level directory of this distribution and at @@ -61,36 +85,51 @@ fn main() -> Result<(), Box> { ``` #### Adding a new header command property -One would add a new header command if there is a need to define some test property or behavior on an individual, test-by-test -basis. A header command property serves as the header command's backing store (holds the command's current value) at -runtime. + +One would add a new header command if there is a need to define some test +property or behavior on an individual, test-by-test basis. A header command +property serves as the header command's backing store (holds the command's +current value) at runtime. To add a new header command property: - 1. Look for the `pub struct TestProps` declaration in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs) and add -the new public property to the end of the declaration. - 2. Look for the `impl TestProps` implementation block immediately following the struct declaration and initialize the new -property to its default value. + 1. Look for the `pub struct TestProps` declaration in + [src/tools/compiletest/src/header.rs][header] and add the new public + property to the end of the declaration. + 2. Look for the `impl TestProps` implementation block immediately following + the struct declaration and initialize the new property to its default + value. #### Adding a new header command parser -When `compiletest` encounters a test file, it parses the file a line at a time by calling every parser defined in the -`Config` struct's implementation block, also in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs) (note the `Config` struct's declaration -block is found in [src/tools/compiletest/src/common.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs). `TestProps`'s `load_from()` method will try passing the current -line of text to each parser, which, in turn typically checks to see if the line begins with a particular commented (`//`) -header command such as `// must-compile-successfully` or `// failure-status`. Whitespace after the comment marker is -optional. - -Parsers will override a given header command property's default value merely by being specified in the test file as a header -command or by having a parameter value specified in the test file, depending on the header command. - -Parsers defined in `impl Config` are typically named `parse_` (note kebab-case `` transformed -to snake-case ``). `impl Config` also defines several 'low-level' parsers which make it simple to parse -common patterns like simple presence or not (`parse_name_directive()`), header-command:parameter(s) -(`parse_name_value_directive()`), optional parsing only if a particular `cfg` attribute is defined (`has_cfg_prefix()`) and -many more. The low-level parsers are found near the end of the `impl Config` block; be sure to look through them and their -associated parsers immediately above to see how they are used to avoid writing additional parsing code unneccessarily. - -As a concrete example, here is the implementation for the `parse_failure_status()` parser, in -[src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs): + +When `compiletest` encounters a test file, it parses the file a line at a time +by calling every parser defined in the `Config` struct's implementation block, +also in [src/tools/compiletest/src/header.rs][header] (note the `Config` +struct's declaration block is found in +[src/tools/compiletest/src/common.rs][common]. `TestProps`'s `load_from()` +method will try passing the current line of text to each parser, which, in turn +typically checks to see if the line begins with a particular commented (`//`) +header command such as `// must-compile-successfully` or `// failure-status`. +Whitespace after the comment marker is optional. + +Parsers will override a given header command property's default value merely by +being specified in the test file as a header command or by having a parameter +value specified in the test file, depending on the header command. + +Parsers defined in `impl Config` are typically named `parse_` +(note kebab-case `` transformed to snake-case +``). `impl Config` also defines several 'low-level' parsers +which make it simple to parse common patterns like simple presence or not +(`parse_name_directive()`), header-command:parameter(s) +(`parse_name_value_directive()`), optional parsing only if a particular `cfg` +attribute is defined (`has_cfg_prefix()`) and many more. The low-level parsers +are found near the end of the `impl Config` block; be sure to look through them +and their associated parsers immediately above to see how they are used to +avoid writing additional parsing code unneccessarily. + +As a concrete example, here is the implementation for the +`parse_failure_status()` parser, in +[src/tools/compiletest/src/header.rs][header]: + ```diff @@ -232,6 +232,7 @@ pub struct TestProps { // customized normalization rules @@ -132,14 +171,21 @@ As a concrete example, here is the implementation for the `parse_failure_status( ``` ## Implementing the behavior change -When a test invokes a particular header command, it is expected that some behavior will change as a result. What behavior, -obviously, will depend on the purpose of the header command. In the case of `failure-status`, the behavior that changes -is that `compiletest` expects the failure code defined by the header command invoked in the test, rather than the default -value. - -Although specific to `failure-status` (as every header command will have a different implementation in order to invoke -behavior change) perhaps it is helpful to see the behavior change implementation of one case, simply as an example. To implement `failure-status`, the `check_correct_failure_status()` function found in the `TestCx` implementation block, -located in [src/tools/compiletest/src/runtest.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs), was modified as per below: + +When a test invokes a particular header command, it is expected that some +behavior will change as a result. What behavior, obviously, will depend on the +purpose of the header command. In the case of `failure-status`, the behavior +that changes is that `compiletest` expects the failure code defined by the +header command invoked in the test, rather than the default value. + +Although specific to `failure-status` (as every header command will have a +different implementation in order to invoke behavior change) perhaps it is +helpful to see the behavior change implementation of one case, simply as an +example. To implement `failure-status`, the `check_correct_failure_status()` +function found in the `TestCx` implementation block, located in +[src/tools/compiletest/src/runtest.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs), +was modified as per below: + ```diff @@ -295,11 +295,14 @@ impl<'test> TestCx<'test> { } @@ -176,7 +222,14 @@ located in [src/tools/compiletest/src/runtest.rs](https://github.com/rust-lang/r } } ``` -Note the use of `self.props.failure_status` to access the header command property. In tests which do not specify the failure -status header command, `self.props.failure_status` will evaluate to the default value of 101 at the time of this writing. -But for a test which specifies a header command of, for example, `// failure-status: 1`, `self.props.failure_status` will -evaluate to 1, as `parse_failure_status()` will have overridden the `TestProps` default value, for that test specifically. +Note the use of `self.props.failure_status` to access the header command +property. In tests which do not specify the failure status header command, +`self.props.failure_status` will evaluate to the default value of 101 at the +time of this writing. But for a test which specifies a header command of, for +example, `// failure-status: 1`, `self.props.failure_status` will evaluate to +1, as `parse_failure_status()` will have overridden the `TestProps` default +value, for that test specifically. + +[test]: https://github.com/rust-lang/rust/tree/master/src/test +[header]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs +[common]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs From 80de3962fbc50f5030e9f68d6e2eabf51f850f12 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 23 Mar 2018 21:02:33 -0500 Subject: [PATCH 172/648] Fix long lines in const-eval --- src/const-eval.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/const-eval.md b/src/const-eval.md index 3fb65216a..4a05255c6 100644 --- a/src/const-eval.md +++ b/src/const-eval.md @@ -11,7 +11,8 @@ Prominent examples are * Array length * needs to be known to reserve stack or heap space * Enum variant discriminants - * needs to be known to prevent two variants from having the same discriminant + * needs to be known to prevent two variants from having the same + discriminant * Patterns * need to be known to check for overlapping patterns From 42ef2ad0b8e1fe47e380b5cb8dbabc0d57aad4fc Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 23 Mar 2018 21:03:13 -0500 Subject: [PATCH 173/648] Fix long lines in conventions --- src/conventions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/conventions.md b/src/conventions.md index db531c58c..96571301f 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -111,8 +111,8 @@ inspects the Cargo metadata to ensure this. # How to structure your PR -How you prepare the commits in your PR can make a big difference for the reviewer. -Here are some tips. +How you prepare the commits in your PR can make a big difference for the +reviewer. Here are some tips. **Isolate "pure refactorings" into their own commit.** For example, if you rename a method, then put that rename into its own commit, along From 2bb61a3f596b60460211d1d607bfdd63dd9dbb99 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 26 Mar 2018 10:41:14 -0500 Subject: [PATCH 174/648] More long lines --- src/incrcomp-debugging.md | 8 ++++---- src/mir-borrowck.md | 9 ++++++--- src/mir-passes.md | 22 +++++++++++++--------- src/mir-regionck.md | 35 +++++++++++++++++------------------ src/mir.md | 25 +++++++++++++++---------- 5 files changed, 55 insertions(+), 44 deletions(-) diff --git a/src/incrcomp-debugging.md b/src/incrcomp-debugging.md index 92a8e10ea..261b46eb0 100644 --- a/src/incrcomp-debugging.md +++ b/src/incrcomp-debugging.md @@ -69,11 +69,11 @@ want the `TypeckTables` node for some particular fn, so you might write: RUST_DEP_GRAPH_FILTER='-> TypeckTables & bar' ``` -This will select only the predecessors of `TypeckTables` nodes for functions with -`bar` in their name. +This will select only the predecessors of `TypeckTables` nodes for functions +with `bar` in their name. -Perhaps you are finding that when you change `foo` you need to re-type-check `bar`, -but you don't think you should have to. In that case, you might do: +Perhaps you are finding that when you change `foo` you need to re-type-check +`bar`, but you don't think you should have to. In that case, you might do: ``` RUST_DEP_GRAPH_FILTER='Hir & foo -> TypeckTables & bar' diff --git a/src/mir-borrowck.md b/src/mir-borrowck.md index 6c4c99d61..51a6c6cbf 100644 --- a/src/mir-borrowck.md +++ b/src/mir-borrowck.md @@ -6,7 +6,8 @@ enforcing a number of properties: - That all variables are initialized before they are used. - That you can't move the same value twice. - That you can't move a value while it is borrowed. -- That you can't access a place while it is mutably borrowed (except through the reference). +- That you can't access a place while it is mutably borrowed (except through + the reference). - That you can't mutate a place while it is shared borrowed. - etc @@ -44,10 +45,12 @@ The overall flow of the borrow checker is as follows: Among other things, this function will replace all of the regions in the MIR with fresh [inference variables](./appendix-glossary.html). - (More details can be found in [the regionck section](./mir-regionck.html).) -- Next, we perform a number of [dataflow analyses](./appendix-background.html#dataflow) +- Next, we perform a number of [dataflow + analyses](./appendix-background.html#dataflow) that compute what data is moved and when. The results of these analyses are needed to do both borrow checking and region inference. -- Using the move data, we can then compute the values of all the regions in the MIR. +- Using the move data, we can then compute the values of all the regions in the + MIR. - (More details can be found in [the NLL section](./mir-regionck.html).) - Finally, the borrow checker itself runs, taking as input (a) the results of move analysis and (b) the regions computed by the region diff --git a/src/mir-passes.md b/src/mir-passes.md index 6d657ae70..cd05edfb8 100644 --- a/src/mir-passes.md +++ b/src/mir-passes.md @@ -15,16 +15,19 @@ transformations. These suites represent useful intermediate points where we want to access the MIR for type checking or other purposes: - `mir_build(D)` – not a query, but this constructs the initial MIR -- `mir_const(D)` – applies some simple transformations to make MIR ready for constant evaluation; -- `mir_validated(D)` – applies some more transformations, making MIR ready for borrow checking; -- `optimized_mir(D)` – the final state, after all optimizations have been performed. +- `mir_const(D)` – applies some simple transformations to make MIR ready for + constant evaluation; +- `mir_validated(D)` – applies some more transformations, making MIR ready for + borrow checking; +- `optimized_mir(D)` – the final state, after all optimizations have been + performed. ### Seeing how the MIR changes as the compiler executes -`-Zdump-mir=F` is a handy compiler options that will let you view the MIR -for each function at each stage of compilation. `-Zdump-mir` takes a **filter** -`F` which allows you to control which functions and which passes you are interesting -in. For example: +`-Zdump-mir=F` is a handy compiler options that will let you view the MIR for +each function at each stage of compilation. `-Zdump-mir` takes a **filter** `F` +which allows you to control which functions and which passes you are +interesting in. For example: ```bash > rustc -Zdump-mir=foo ... @@ -58,8 +61,9 @@ rustc.main.000-000.CleanEndRegions.after.mir def-path to the function etc being dumped ``` -You can also make more selective filters. For example, `main & CleanEndRegions` will select -for things that reference *both* `main` and the pass `CleanEndRegions`: +You can also make more selective filters. For example, `main & CleanEndRegions` +will select for things that reference *both* `main` and the pass +`CleanEndRegions`: ```bash > rustc -Zdump-mir='main & CleanEndRegions' foo.rs diff --git a/src/mir-regionck.md b/src/mir-regionck.md index dbf740ea8..529b42395 100644 --- a/src/mir-regionck.md +++ b/src/mir-regionck.md @@ -418,16 +418,15 @@ variable `'?3`, but the variable `'?3` is not forced to outlive anything else. Therefore, it simply starts and ends as the empty set of elements, and hence the type-check succeeds here. -(This should surprise you a little. It surprised me when I first -realized it. We are saying that if we are a fn that **needs both of -its arguments to have the same region**, we can accept being called -with **arguments with two distinct regions**. That seems intuitively -unsound. But in fact, it's fine, as I -[tried to explain in this issue on the Rust issue tracker long ago][ohdeargoditsallbroken]. -The reason is that even if we get called with arguments of two -distinct lifetimes, those two lifetimes have some intersection (the -call itself), and that intersection can be our value of `'a` that we -use as the common lifetime of our arguments. -nmatsakis) +(This should surprise you a little. It surprised me when I first realized it. +We are saying that if we are a fn that **needs both of its arguments to have +the same region**, we can accept being called with **arguments with two +distinct regions**. That seems intuitively unsound. But in fact, it's fine, as +I tried to explain in [this issue][ohdeargoditsallbroken] on the Rust issue +tracker long ago. The reason is that even if we get called with arguments of +two distinct lifetimes, those two lifetimes have some intersection (the call +itself), and that intersection can be our value of `'a` that we use as the +common lifetime of our arguments. -nmatsakis) [ohdeargoditsallbroken]: https://github.com/rust-lang/rust/issues/32330#issuecomment-202536977 @@ -440,10 +439,10 @@ a return type: <: for<'b, 'c> fn(&'b u32, &'c u32) -> &'b u32 -Despite seeming very similar to the previous example, this case is -going to get an error. That's good: the problem is that we've gone -from a fn that promises to return one of its two arguments, to a fn -that is promising to return the first one. That is unsound. Let's see how it plays out. +Despite seeming very similar to the previous example, this case is going to get +an error. That's good: the problem is that we've gone from a fn that promises +to return one of its two arguments, to a fn that is promising to return the +first one. That is unsound. Let's see how it plays out. First, we skolemize the supertype: @@ -463,8 +462,8 @@ And now we create the subtyping relationships: &'!2 u32 <: &'?3 u32 // arg 2 &'?3 u32 <: &'!1 u32 // return type -And finally the outlives relationships. Here, let V1, V2, and V3 be the variables -we assign to `!1`, `!2`, and `?3` respectively: +And finally the outlives relationships. Here, let V1, V2, and V3 be the +variables we assign to `!1`, `!2`, and `?3` respectively: V1: V3 V2: V3 @@ -476,8 +475,8 @@ Those variables will have these initial values: V2 in U2 = {skol(2)} V3 in U2 = {} -Now because of the `V3: V1` constraint, we have to add `skol(1)` into `V3` (and indeed -it is visible from `V3`), so we get: +Now because of the `V3: V1` constraint, we have to add `skol(1)` into `V3` (and +indeed it is visible from `V3`), so we get: V3 in U2 = {skol(1)} diff --git a/src/mir.md b/src/mir.md index 688a8750c..58479cf96 100644 --- a/src/mir.md +++ b/src/mir.md @@ -34,14 +34,17 @@ This section introduces the key concepts of MIR, summarized here: - **Basic blocks**: units of the control-flow graph, consisting of: - **statements:** actions with one successor - - **terminators:** actions with potentially multiple successors; always at the end of a block - - (if you're not familiar with the term *basic block*, see the [background chapter][cfg]) + - **terminators:** actions with potentially multiple successors; always at + the end of a block + - (if you're not familiar with the term *basic block*, see the [background + chapter][cfg]) - **Locals:** Memory locations alloated on the stack (conceptually, at least), such as function arguments, local variables, and temporaries. These are identified by an index, written with a leading underscore, like `_1`. There is also a special "local" (`_0`) allocated to store the return value. -- **Places:** expressions that identify a location in memory, like `_1` or `_1.f`. +- **Places:** expressions that identify a location in memory, like `_1` or + `_1.f`. - **Rvalues:** expressions that produce a value. The "R" stands for the fact that these are the "right-hand side" of an assignment. - **Operands:** the arguments to an rvalue, which can either be a @@ -100,8 +103,9 @@ you their original name (`// "vec" in scope 1...`). The "scope" blocks (e.g., `scope 1 { .. }`) describe the lexical structure of the source program (which names were in scope when). -**Basic blocks.** Reading further, we see our first **basic block** (naturally it may look -slightly different when you view it, and I am ignoring some of the comments): +**Basic blocks.** Reading further, we see our first **basic block** (naturally +it may look slightly different when you view it, and I am ignoring some of the +comments): ``` bb0: { @@ -110,8 +114,8 @@ bb0: { } ``` -A basic block is defined by a series of **statements** and a final **terminator**. -In this case, there is one statement: +A basic block is defined by a series of **statements** and a final +**terminator**. In this case, there is one statement: ``` StorageLive(_1); @@ -146,8 +150,8 @@ bb2: { } ``` -Here there are two statements: another `StorageLive`, introducing the `_3` temporary, -and then an assignment: +Here there are two statements: another `StorageLive`, introducing the `_3` +temporary, and then an assignment: ``` _3 = &mut _1; @@ -189,7 +193,8 @@ TMP1 = a + b x = TMP1 + c ``` -([Try it and see, though you may want to do release mode to skip over the overflow checks.][play-abc]) +([Try it and see][play-abc], though you may want to do release mode to skip +over the overflow checks.) [play-abc]: https://play.rust-lang.org/?gist=1751196d63b2a71f8208119e59d8a5b6&version=stable From 9e6cdf3de3f6eb1db48ed13baacdcff867d09c01 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 26 Mar 2018 10:44:32 -0500 Subject: [PATCH 175/648] still more long lines --- src/traits-canonical-queries.md | 6 ++++-- src/traits-canonicalization.md | 4 ++-- src/traits-goals-and-clauses.md | 12 ++++++----- src/traits-lowering-rules.md | 37 ++++++++++++++++++--------------- 4 files changed, 33 insertions(+), 26 deletions(-) diff --git a/src/traits-canonical-queries.md b/src/traits-canonical-queries.md index 3fea5fbd9..b291a2898 100644 --- a/src/traits-canonical-queries.md +++ b/src/traits-canonical-queries.md @@ -99,7 +99,8 @@ that the query was false and had no answers (e.g., `Box: Copy`). Otherwise, the `QueryResult` gives back information about the possible answer(s) we did find. It consists of four parts: -- **Certainty:** tells you how sure we are of this answer. It can have two values: +- **Certainty:** tells you how sure we are of this answer. It can have two + values: - `Proven` means that the result is known to be true. - This might be the result for trying to prove `Vec: Clone`, say, or `Rc: Clone`. @@ -171,7 +172,8 @@ Therefore, the result we get back would be as follows (I'm going to ignore region constraints and the "value"): - Certainty: `Ambiguous` -- we're not sure yet if this holds -- Var values: `[?T = ?T, ?U = ?U]` -- we learned nothing about the values of the variables +- Var values: `[?T = ?T, ?U = ?U]` -- we learned nothing about the values of + the variables In short, the query result says that it is too soon to say much about whether this trait is proven. During type-checking, this is not an diff --git a/src/traits-canonicalization.md b/src/traits-canonicalization.md index bef858af2..9f3af36c9 100644 --- a/src/traits-canonicalization.md +++ b/src/traits-canonicalization.md @@ -129,8 +129,8 @@ for which we made a substutition S: S = [?A, '?B, ?C] -We then did some work which unified some of those variables with other things. If we -"refresh" S with the latest results, we get: +We then did some work which unified some of those variables with other things. +If we "refresh" S with the latest results, we get: S = [Vec, '?D, ?E] diff --git a/src/traits-goals-and-clauses.md b/src/traits-goals-and-clauses.md index 92e1b67d9..8e5e82d01 100644 --- a/src/traits-goals-and-clauses.md +++ b/src/traits-goals-and-clauses.md @@ -81,13 +81,15 @@ Given that, we can define a `DomainGoal` as follows: - as we'll see in the section on lowering, `FromEnv(X)` implies `Implemented(X)` but not vice versa. This distinction is crucial to [implied bounds]. -- `ProjectionEq(Projection = Type)` -- the given associated type `Projection` is equal - to `Type`; see [the section on associated types](./traits-associated-types.html) +- `ProjectionEq(Projection = Type)` -- the given associated type `Projection` + is equal to `Type`; see [the section on associated + types](./traits-associated-types.html) - in general, proving `ProjectionEq(TraitRef::Item = Type)` also requires proving `Implemented(TraitRef)` -- `Normalize(Projection -> Type)` -- the given associated type `Projection` can be [normalized][n] - to `Type` - - as discussed in [the section on associated types](./traits-associated-types.html), +- `Normalize(Projection -> Type)` -- the given associated type `Projection` can + be [normalized][n] to `Type` + - as discussed in [the section on associated + types](./traits-associated-types.html), `Normalize` implies `ProjectionEq` but not vice versa - `WellFormed(..)` -- these goals imply that the given item is *well-formed* diff --git a/src/traits-lowering-rules.md b/src/traits-lowering-rules.md index 8de20f1d8..544ae41b3 100644 --- a/src/traits-lowering-rules.md +++ b/src/traits-lowering-rules.md @@ -26,7 +26,7 @@ Each of these lowering rules is given a name, documented with a comment like so: // Rule Foo-Bar-Baz - + you can also search through the `librustc_traits` crate in rustc to find the corresponding rules from the implementation. @@ -36,7 +36,8 @@ When used in a goal position, where clauses can be mapped directly to [domain goals][dg], as follows: - `A0: Foo` maps to `Implemented(A0: Foo)`. -- `A0: Foo` maps to `ProjectionEq(>::Item = T)` +- `A0: Foo` maps to + `ProjectionEq(>::Item = T)` - `T: 'r` maps to `Outlives(T, 'r)` - `'a: 'b` maps to `Outlives('a, 'b)` @@ -58,7 +59,7 @@ on the lowered where clauses, as defined here: - `WellFormed(WC)` -- this indicates that: - `Implemented(TraitRef)` becomes `WellFormed(TraitRef)` - `ProjectionEq(Projection = Ty)` becomes `WellFormed(Projection = Ty)` - + *TODO*: I suspect that we want to alter the outlives relations too, but Chalk isn't modeling those right now. @@ -106,7 +107,7 @@ The next few clauses have to do with implied bounds (see also forall { FromEnv(WC) :- FromEnv(Self: Trait { - WellFormed(Self: Trait) :- Implemented(Self: Trait) && WellFormed(WC) - } +```txt +// Rule WellFormed-TraitRef +// +// For each where clause WC: +forall { + WellFormed(Self: Trait) :- Implemented(Self: Trait) && WellFormed(WC) +} +``` This `WellFormed` rule states that `T: Trait` is well-formed if (a) `T: Trait` is implemented and (b) all the where-clauses declared on @@ -149,9 +152,9 @@ trait A { } trait B { } ``` -Here, the transitive set of implications for `T: Foo` are `T: A`, `T: Bar`, and `T: B`. -And indeed if we were to try to prove `WellFormed(T: Foo)`, we would have to prove each -one of those: +Here, the transitive set of implications for `T: Foo` are `T: A`, `T: Bar`, and +`T: B`. And indeed if we were to try to prove `WellFormed(T: Foo)`, we would +have to prove each one of those: - `WellFormed(T: Foo)` - `Implemented(T: Foo)` @@ -212,7 +215,7 @@ but reproduced here for reference: Implemented(Self: Trait) && WC1 } - + The next rule covers implied bounds for the projection. In particular, the `Bounds` declared on the associated type must be proven to hold to show that the impl is well-formed, and hence we can rely on them @@ -223,8 +226,8 @@ elsewhere. ### Lowering function and constant declarations -Chalk didn't model functions and constants, but I would eventually -like to treat them exactly like normalization. See [the section on function/constant +Chalk didn't model functions and constants, but I would eventually like to +treat them exactly like normalization. See [the section on function/constant values below](#constant-vals) for more details. ## Lowering impls @@ -272,7 +275,7 @@ We produce the following rule: WC && WC1 } } - + Note that `WC` and `WC1` both encode where-clauses that the impl can rely on. From 066ba9ef8210703ddbe9c4b82caf3e1c8b912d35 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 26 Mar 2018 10:46:39 -0500 Subject: [PATCH 176/648] reformat basically the whole rustdoc.md --- src/rustdoc.md | 293 +++++++++++++++++++++++++++---------------------- 1 file changed, 162 insertions(+), 131 deletions(-) diff --git a/src/rustdoc.md b/src/rustdoc.md index ae640d110..7a9905d3b 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -1,7 +1,8 @@ # The walking tour of rustdoc -Rustdoc actually uses the rustc internals directly. It lives in-tree with the compiler and standard -library. This chapter is about how it works. (A new implementation is also [under way], though). +Rustdoc actually uses the rustc internals directly. It lives in-tree with the +compiler and standard library. This chapter is about how it works. (A new +implementation is also [under way], though). [under way]: https://github.com/steveklabnik/rustdoc @@ -17,50 +18,57 @@ and [queries] are discussed in the linked chapters. `librustdoc` performs two major steps after that to render a set of documentation: -* "Clean" the AST into a form that's more suited to creating documentation (and slightly more - resistant to churn in the compiler). +* "Clean" the AST into a form that's more suited to creating documentation (and + slightly more resistant to churn in the compiler). * Use this cleaned AST to render a crate's documentation, one page at a time. -Naturally, there's more than just this, and those descriptions simplify out lots of details, but -that's the high-level overview. +Naturally, there's more than just this, and those descriptions simplify out +lots of details, but that's the high-level overview. -(Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated using the project in -[`src/tools/rustdoc`][bin]. Note that literally all that does is call the `main()` that's in this crate's -`lib.rs`, though.) +(Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated +using the project in [`src/tools/rustdoc`][bin]. Note that literally all that +does is call the `main()` that's in this crate's `lib.rs`, though.) [bin]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc ## Cheat sheet -* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable rustdoc you can run on - other projects. +* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable + rustdoc you can run on other projects. * Add `src/libtest` to be able to use `rustdoc --test`. - * If you've used `rustup toolchain link local /path/to/build/$TARGET/stage1` previously, then - after the previous build command, `cargo +local doc` will Just Work. -* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the standard library docs. - * The completed docs will be available in `build/$TARGET/doc/std`, though the bundle is meant to - be used as though you would copy out the `doc` folder to a web server, since that's where the - CSS/JS and landing page are. -* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`. It's in a bunch of - `fmt::Display` implementations and supplementary functions. -* The types that got `Display` impls above are defined in `clean/mod.rs`, right next to the custom - `Clean` trait used to process them out of the rustc HIR. + * If you've used `rustup toolchain link local /path/to/build/$TARGET/stage1` + previously, then after the previous build command, `cargo +local doc` will + Just Work. +* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the + standard library docs. + * The completed docs will be available in `build/$TARGET/doc/std`, though the + bundle is meant to be used as though you would copy out the `doc` folder to + a web server, since that's where the CSS/JS and landing page are. +* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`. + It's in a bunch of `fmt::Display` implementations and supplementary + functions. +* The types that got `Display` impls above are defined in `clean/mod.rs`, right + next to the custom `Clean` trait used to process them out of the rustc HIR. * The bits specific to using rustdoc as a test harness are in `test.rs`. -* The Markdown renderer is loaded up in `html/markdown.rs`, including functions for extracting - doctests from a given block of Markdown. -* The tests on rustdoc *output* are located in `src/test/rustdoc`, where they're handled by the test - runner of rustbuild and the supplementary script `src/etc/htmldocck.py`. -* Tests on search index generation are located in `src/test/rustdoc-js`, as a series of JavaScript - files that encode queries on the standard library search index and expected results. +* The Markdown renderer is loaded up in `html/markdown.rs`, including functions + for extracting doctests from a given block of Markdown. +* The tests on rustdoc *output* are located in `src/test/rustdoc`, where + they're handled by the test runner of rustbuild and the supplementary script + `src/etc/htmldocck.py`. +* Tests on search index generation are located in `src/test/rustdoc-js`, as a + series of JavaScript files that encode queries on the standard library search + index and expected results. ## From crate to clean -In `core.rs` are two central items: the `DocContext` struct, and the `run_core` function. The latter -is where rustdoc calls out to rustc to compile a crate to the point where rustdoc can take over. The -former is a state container used when crawling through a crate to gather its documentation. +In `core.rs` are two central items: the `DocContext` struct, and the `run_core` +function. The latter is where rustdoc calls out to rustc to compile a crate to +the point where rustdoc can take over. The former is a state container used +when crawling through a crate to gather its documentation. -The main process of crate crawling is done in `clean/mod.rs` through several implementations of the -`Clean` trait defined within. This is a conversion trait, which defines one method: +The main process of crate crawling is done in `clean/mod.rs` through several +implementations of the `Clean` trait defined within. This is a conversion +trait, which defines one method: ```rust pub trait Clean { @@ -68,40 +76,47 @@ pub trait Clean { } ``` -`clean/mod.rs` also defines the types for the "cleaned" AST used later on to render documentation -pages. Each usually accompanies an implementation of `Clean` that takes some AST or HIR type from -rustc and converts it into the appropriate "cleaned" type. "Big" items like modules or associated -items may have some extra processing in its `Clean` implementation, but for the most part these -impls are straightforward conversions. The "entry point" to this module is the `impl Clean -for visit_ast::RustdocVisitor`, which is called by `run_core` above. - -You see, I actually lied a little earlier: There's another AST transformation that happens before -the events in `clean/mod.rs`. In `visit_ast.rs` is the type `RustdocVisitor`, which *actually* -crawls a `hir::Crate` to get the first intermediate representation, defined in `doctree.rs`. This -pass is mainly to get a few intermediate wrappers around the HIR types and to process visibility -and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and `#[doc(hidden)]` are -processed, as well as the logic for whether a `pub use` should get the full page or a "Reexport" -line in the module page. - -The other major thing that happens in `clean/mod.rs` is the collection of doc comments and -`#[doc=""]` attributes into a separate field of the Attributes struct, present on anything that gets -hand-written documentation. This makes it easier to collect this documentation later in the process. - -The primary output of this process is a `clean::Crate` with a tree of Items which describe the -publicly-documentable items in the target crate. +`clean/mod.rs` also defines the types for the "cleaned" AST used later on to +render documentation pages. Each usually accompanies an implementation of +`Clean` that takes some AST or HIR type from rustc and converts it into the +appropriate "cleaned" type. "Big" items like modules or associated items may +have some extra processing in its `Clean` implementation, but for the most part +these impls are straightforward conversions. The "entry point" to this module +is the `impl Clean for visit_ast::RustdocVisitor`, which is called by +`run_core` above. + +You see, I actually lied a little earlier: There's another AST transformation +that happens before the events in `clean/mod.rs`. In `visit_ast.rs` is the +type `RustdocVisitor`, which *actually* crawls a `hir::Crate` to get the first +intermediate representation, defined in `doctree.rs`. This pass is mainly to +get a few intermediate wrappers around the HIR types and to process visibility +and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and +`#[doc(hidden)]` are processed, as well as the logic for whether a `pub use` +should get the full page or a "Reexport" line in the module page. + +The other major thing that happens in `clean/mod.rs` is the collection of doc +comments and `#[doc=""]` attributes into a separate field of the Attributes +struct, present on anything that gets hand-written documentation. This makes it +easier to collect this documentation later in the process. + +The primary output of this process is a `clean::Crate` with a tree of Items +which describe the publicly-documentable items in the target crate. ### Hot potato -Before moving on to the next major step, a few important "passes" occur over the documentation. -These do things like combine the separate "attributes" into a single string and strip leading -whitespace to make the document easier on the markdown parser, or drop items that are not public or -deliberately hidden with `#[doc(hidden)]`. These are all implemented in the `passes/` directory, one -file per pass. By default, all of these passes are run on a crate, but the ones regarding dropping -private/hidden items can be bypassed by passing `--document-private-items` to rustdoc. Note that -unlike the previous set of AST transformations, the passes happen on the _cleaned_ crate. +Before moving on to the next major step, a few important "passes" occur over +the documentation. These do things like combine the separate "attributes" into +a single string and strip leading whitespace to make the document easier on the +markdown parser, or drop items that are not public or deliberately hidden with +`#[doc(hidden)]`. These are all implemented in the `passes/` directory, one +file per pass. By default, all of these passes are run on a crate, but the ones +regarding dropping private/hidden items can be bypassed by passing +`--document-private-items` to rustdoc. Note that unlike the previous set of AST +transformations, the passes happen on the _cleaned_ crate. -(Strictly speaking, you can fine-tune the passes run and even add your own, but [we're trying to -deprecate that][44136]. If you need finer-grain control over these passes, please let us know!) +(Strictly speaking, you can fine-tune the passes run and even add your own, but +[we're trying to deprecate that][44136]. If you need finer-grain control over +these passes, please let us know!) [44136]: https://github.com/rust-lang/rust/issues/44136 @@ -116,9 +131,9 @@ Here is current (as of this writing) list of passes: markdown to like it. This is necessary because the convention for writing documentation is to provide a space between the `///` or `//!` marker and the text, and stripping that leading space will make the text easier to parse by - the Markdown parser. (In the past, the markdown parser used was not Commonmark- - compliant, which caused annoyances with extra whitespace but this seems to be - less of an issue today.) + the Markdown parser. (In the past, the markdown parser used was not + Commonmark- compliant, which caused annoyances with extra whitespace but this + seems to be less of an issue today.) - `strip-priv-imports` strips all private import statements (`use`, `extern crate`) from a crate. This is necessary because rustdoc will handle *public* imports by either inlining the item's documentation to the module or creating @@ -130,85 +145,101 @@ Here is current (as of this writing) list of passes: ## From clean to crate -This is where the "second phase" in rustdoc begins. This phase primarily lives in the `html/` -folder, and it all starts with `run()` in `html/render.rs`. This code is responsible for setting up -the `Context`, `SharedContext`, and `Cache` which are used during rendering, copying out the static -files which live in every rendered set of documentation (things like the fonts, CSS, and JavaScript -that live in `html/static/`), creating the search index, and printing out the source code rendering, -before beginning the process of rendering all the documentation for the crate. - -Several functions implemented directly on `Context` take the `clean::Crate` and set up some state -between rendering items or recursing on a module's child items. From here the "page rendering" -begins, via an enormous `write!()` call in `html/layout.rs`. The parts that actually generate HTML -from the items and documentation occurs within a series of `std::fmt::Display` implementations and -functions that pass around a `&mut std::fmt::Formatter`. The top-level implementation that writes -out the page body is the `impl<'a> fmt::Display for Item<'a>` in `html/render.rs`, which switches -out to one of several `item_*` functions based on the kind of `Item` being rendered. - -Depending on what kind of rendering code you're looking for, you'll probably find it either in -`html/render.rs` for major items like "what sections should I print for a struct page" or -`html/format.rs` for smaller component pieces like "how should I print a where clause as part of -some other item". - -Whenever rustdoc comes across an item that should print hand-written documentation alongside, it -calls out to `html/markdown.rs` which interfaces with the Markdown parser. This is exposed as a -series of types that wrap a string of Markdown, and implement `fmt::Display` to emit HTML text. It -takes special care to enable certain features like footnotes and tables and add syntax highlighting -to Rust code blocks (via `html/highlight.rs`) before running the Markdown parser. There's also a -function in here (`find_testable_code`) that specifically scans for Rust code blocks so the +This is where the "second phase" in rustdoc begins. This phase primarily lives +in the `html/` folder, and it all starts with `run()` in `html/render.rs`. This +code is responsible for setting up the `Context`, `SharedContext`, and `Cache` +which are used during rendering, copying out the static files which live in +every rendered set of documentation (things like the fonts, CSS, and JavaScript +that live in `html/static/`), creating the search index, and printing out the +source code rendering, before beginning the process of rendering all the +documentation for the crate. + +Several functions implemented directly on `Context` take the `clean::Crate` and +set up some state between rendering items or recursing on a module's child +items. From here the "page rendering" begins, via an enormous `write!()` call +in `html/layout.rs`. The parts that actually generate HTML from the items and +documentation occurs within a series of `std::fmt::Display` implementations and +functions that pass around a `&mut std::fmt::Formatter`. The top-level +implementation that writes out the page body is the `impl<'a> fmt::Display for +Item<'a>` in `html/render.rs`, which switches out to one of several `item_*` +functions based on the kind of `Item` being rendered. + +Depending on what kind of rendering code you're looking for, you'll probably +find it either in `html/render.rs` for major items like "what sections should I +print for a struct page" or `html/format.rs` for smaller component pieces like +"how should I print a where clause as part of some other item". + +Whenever rustdoc comes across an item that should print hand-written +documentation alongside, it calls out to `html/markdown.rs` which interfaces +with the Markdown parser. This is exposed as a series of types that wrap a +string of Markdown, and implement `fmt::Display` to emit HTML text. It takes +special care to enable certain features like footnotes and tables and add +syntax highlighting to Rust code blocks (via `html/highlight.rs`) before +running the Markdown parser. There's also a function in here +(`find_testable_code`) that specifically scans for Rust code blocks so the test-runner code can find all the doctests in the crate. ### From soup to nuts -(alternate title: ["An unbroken thread that stretches from those first `Cell`s to us"][video]) +(alternate title: ["An unbroken thread that stretches from those first `Cell`s +to us"][video]) [video]: https://www.youtube.com/watch?v=hOLAGYmUQV0 -It's important to note that the AST cleaning can ask the compiler for information (crucially, -`DocContext` contains a `TyCtxt`), but page rendering cannot. The `clean::Crate` created within -`run_core` is passed outside the compiler context before being handed to `html::render::run`. This -means that a lot of the "supplementary data" that isn't immediately available inside an item's -definition, like which trait is the `Deref` trait used by the language, needs to be collected during -cleaning, stored in the `DocContext`, and passed along to the `SharedContext` during HTML rendering. -This manifests as a bunch of shared state, context variables, and `RefCell`s. - -Also of note is that some items that come from "asking the compiler" don't go directly into the -`DocContext` - for example, when loading items from a foreign crate, rustdoc will ask about trait -implementations and generate new `Item`s for the impls based on that information. This goes directly -into the returned `Crate` rather than roundabout through the `DocContext`. This way, these -implementations can be collected alongside the others, right before rendering the HTML. +It's important to note that the AST cleaning can ask the compiler for +information (crucially, `DocContext` contains a `TyCtxt`), but page rendering +cannot. The `clean::Crate` created within `run_core` is passed outside the +compiler context before being handed to `html::render::run`. This means that a +lot of the "supplementary data" that isn't immediately available inside an +item's definition, like which trait is the `Deref` trait used by the language, +needs to be collected during cleaning, stored in the `DocContext`, and passed +along to the `SharedContext` during HTML rendering. This manifests as a bunch +of shared state, context variables, and `RefCell`s. + +Also of note is that some items that come from "asking the compiler" don't go +directly into the `DocContext` - for example, when loading items from a foreign +crate, rustdoc will ask about trait implementations and generate new `Item`s +for the impls based on that information. This goes directly into the returned +`Crate` rather than roundabout through the `DocContext`. This way, these +implementations can be collected alongside the others, right before rendering +the HTML. ## Other tricks up its sleeve -All this describes the process for generating HTML documentation from a Rust crate, but there are -couple other major modes that rustdoc runs in. It can also be run on a standalone Markdown file, or -it can run doctests on Rust code or standalone Markdown files. For the former, it shortcuts straight -to `html/markdown.rs`, optionally including a mode which inserts a Table of Contents to the output -HTML. - -For the latter, rustdoc runs a similar partial-compilation to get relevant documentation in -`test.rs`, but instead of going through the full clean and render process, it runs a much simpler -crate walk to grab *just* the hand-written documentation. Combined with the aforementioned -"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of tests to run before -handing them off to the libtest test runner. One notable location in `test.rs` is the function -`make_test`, which is where hand-written doctests get transformed into something that can be -executed. +All this describes the process for generating HTML documentation from a Rust +crate, but there are couple other major modes that rustdoc runs in. It can also +be run on a standalone Markdown file, or it can run doctests on Rust code or +standalone Markdown files. For the former, it shortcuts straight to +`html/markdown.rs`, optionally including a mode which inserts a Table of +Contents to the output HTML. + +For the latter, rustdoc runs a similar partial-compilation to get relevant +documentation in `test.rs`, but instead of going through the full clean and +render process, it runs a much simpler crate walk to grab *just* the +hand-written documentation. Combined with the aforementioned +"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of +tests to run before handing them off to the libtest test runner. One notable +location in `test.rs` is the function `make_test`, which is where hand-written +doctests get transformed into something that can be executed. Some extra reading about `make_test` can be found [here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/). ## Dotting i's and crossing t's -So that's rustdoc's code in a nutshell, but there's more things in the repo that deal with it. Since -we have the full `compiletest` suite at hand, there's a set of tests in `src/test/rustdoc` that make -sure the final HTML is what we expect in various situations. These tests also use a supplementary -script, `src/etc/htmldocck.py`, that allows it to look through the final HTML using XPath notation -to get a precise look at the output. The full description of all the commands available to rustdoc -tests is in `htmldocck.py`. - -In addition, there are separate tests for the search index and rustdoc's ability to query it. The -files in `src/test/rustdoc-js` each contain a different search query and the expected results, -broken out by search tab. These files are processed by a script in `src/tools/rustdoc-js` and the -Node.js runtime. These tests don't have as thorough of a writeup, but a broad example that features -results in all tabs can be found in `basic.js`. The basic idea is that you match a given `QUERY` -with a set of `EXPECTED` results, complete with the full item path of each item. +So that's rustdoc's code in a nutshell, but there's more things in the repo +that deal with it. Since we have the full `compiletest` suite at hand, there's +a set of tests in `src/test/rustdoc` that make sure the final HTML is what we +expect in various situations. These tests also use a supplementary script, +`src/etc/htmldocck.py`, that allows it to look through the final HTML using +XPath notation to get a precise look at the output. The full description of all +the commands available to rustdoc tests is in `htmldocck.py`. + +In addition, there are separate tests for the search index and rustdoc's +ability to query it. The files in `src/test/rustdoc-js` each contain a +different search query and the expected results, broken out by search tab. +These files are processed by a script in `src/tools/rustdoc-js` and the Node.js +runtime. These tests don't have as thorough of a writeup, but a broad example +that features results in all tabs can be found in `basic.js`. The basic idea is +that you match a given `QUERY` with a set of `EXPECTED` results, complete with +the full item path of each item. From 36179fb7410cf28f89469bfc3badb01f67b34c41 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tom=C3=A1=C5=A1=20Gaven=C4=8Diak?= Date: Tue, 27 Mar 2018 14:32:45 +0200 Subject: [PATCH 177/648] Add info on test result caching --- src/tests/running.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/tests/running.md b/src/tests/running.md index 6bac0121a..84cead4bf 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -13,6 +13,12 @@ a very long time, and anyway bors / travis will do it for you. (Often, I will run this command in the background after opening a PR that I think is done, but rarely otherwise. -nmatsakis) +The test results are cached and previously successfull tests are +`ignored` during testing. The stdout/stderr contents as well as a +timestamp file for every test can be found under `build/ARCH/test/`. +To force-rerun a test (e.g. in case the test runner fails to notice +a change) you can simply remove the timestamp file. + ## Running a subset of the test suites When working on a specific PR, you will usually want to run a smaller From 4d5ad15ad49afdc5f231dd57f2bc0906b2b45fff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tom=C3=A1=C5=A1=20Gaven=C4=8Diak?= Date: Tue, 27 Mar 2018 20:03:56 +0200 Subject: [PATCH 178/648] Fix a typo --- src/tests/running.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/tests/running.md b/src/tests/running.md index 84cead4bf..4e8c71590 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -13,7 +13,7 @@ a very long time, and anyway bors / travis will do it for you. (Often, I will run this command in the background after opening a PR that I think is done, but rarely otherwise. -nmatsakis) -The test results are cached and previously successfull tests are +The test results are cached and previously successful tests are `ignored` during testing. The stdout/stderr contents as well as a timestamp file for every test can be found under `build/ARCH/test/`. To force-rerun a test (e.g. in case the test runner fails to notice From 3d4332cd4fef33c6371907ffe15d34b496ca8a8b Mon Sep 17 00:00:00 2001 From: Dan Robertson Date: Wed, 28 Mar 2018 03:23:51 +0000 Subject: [PATCH 179/648] Update links Add missing link and add link for UFCS. --- src/method-lookup.md | 3 ++- src/traits-bibliography.md | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/src/method-lookup.md b/src/method-lookup.md index 543428d62..ac0e427db 100644 --- a/src/method-lookup.md +++ b/src/method-lookup.md @@ -12,7 +12,7 @@ the form: receiver.method(...) ``` -into a more explicit UFCS form: +into a more explicit [UFCS] form: ```rust Trait::method(ADJ(receiver), ...) // for a trait call @@ -37,6 +37,7 @@ probe phase produces a "pick" (`probe::Pick`), which is designed to be cacheable across method-call sites. Therefore, it does not include inference variables or other information. +[UFCS]: https://github.com/rust-lang/rfcs/blob/master/text/0132-ufcs.md [probe]: https://github.com/rust-lang/rust/blob/master/src/librustc_typeck/check/method/probe.rs [confirm]: https://github.com/rust-lang/rust/blob/master/src/librustc_typeck/check/method/confirm.rs diff --git a/src/traits-bibliography.md b/src/traits-bibliography.md index d9ff912c7..02607b525 100644 --- a/src/traits-bibliography.md +++ b/src/traits-bibliography.md @@ -26,3 +26,4 @@ SLG formulation that is the basis for our on-demand solver. [nftrd]: https://dl.acm.org/citation.cfm?id=651202 [ts]: http://www3.cs.stonybrook.edu/~tswift/ +[Theresa Swift]: http://www3.cs.stonybrook.edu/~tswift/ From 3c189daf2c14bd2e3bfb51446c6564dcc0a01f12 Mon Sep 17 00:00:00 2001 From: Philip Munksgaard Date: Tue, 3 Apr 2018 16:27:29 +0200 Subject: [PATCH 180/648] Fix typo in rustdoc.md --- src/rustdoc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/rustdoc.md b/src/rustdoc.md index 7a9905d3b..e075087fc 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -25,7 +25,7 @@ documentation: Naturally, there's more than just this, and those descriptions simplify out lots of details, but that's the high-level overview. -(Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated +(Side note: `librustdoc` is a library crate! The `rustdoc` binary is created using the project in [`src/tools/rustdoc`][bin]. Note that literally all that does is call the `main()` that's in this crate's `lib.rs`, though.) From f5ff6d03be39a72bbca282fe866e481f24471139 Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Wed, 4 Apr 2018 13:41:23 -0400 Subject: [PATCH 181/648] Link to internal rustdocs#100 --- src/appendix-stupid-stats.md | 2 +- src/macro-expansion.md | 6 +++--- src/mir-passes.md | 4 ++-- src/mir-visitor.md | 6 +++--- src/query.md | 4 ++-- src/rustc-driver.md | 10 +++++----- src/the-parser.md | 10 +++++----- src/traits-lowering-module.md | 2 +- 8 files changed, 22 insertions(+), 22 deletions(-) diff --git a/src/appendix-stupid-stats.md b/src/appendix-stupid-stats.md index 8e50b2c31..7aef3a87b 100644 --- a/src/appendix-stupid-stats.md +++ b/src/appendix-stupid-stats.md @@ -87,7 +87,7 @@ in [librustc_back](https://github.com/rust-lang/rust/tree/master/src/librustc_ba (which also contains some things used primarily during translation). All these phases are coordinated by the driver. To see the exact sequence, look -at the `compile_input` function in [librustc_driver/driver.rs](https://github.com/rust-lang/rust/tree/master/src/librustc_driver/driver.rs). +at the `compile_input` function in [librustc_driver/driver.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/). The driver (which is found in [librust_driver](https://github.com/rust-lang/rust/tree/master/src/librustc_driver)) handles all the highest level coordination of compilation - handling command line arguments, maintaining compilation state (primarily in the `Session`), and diff --git a/src/macro-expansion.md b/src/macro-expansion.md index 95ea64f19..c55fdd136 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -156,7 +156,7 @@ TODO [code_dir]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt -[code_mp]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt/macro_parser.rs -[code_mr]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt/macro_rules.rs -[code_parse_int]: https://github.com/rust-lang/rust/blob/a97cd17f5d71fb4ec362f4fbd79373a6e7ed7b82/src/libsyntax/ext/tt/macro_parser.rs#L421 +[code_mp]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ext/tt/macro_parser/ +[code_mr]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ext/tt/macro_rules/ +[code_parse_int]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ext/tt/macro_parser/fn.parse.html [parsing]: ./the-parser.html diff --git a/src/mir-passes.md b/src/mir-passes.md index cd05edfb8..1d1e27df4 100644 --- a/src/mir-passes.md +++ b/src/mir-passes.md @@ -172,6 +172,6 @@ This mechanism is a bit dodgy. There is a discussion of more elegant alternatives in [rust-lang/rust#41710]. [rust-lang/rust#41710]: https://github.com/rust-lang/rust/issues/41710 -[mirtransform]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir/transform/mod.rs -[`NoLandingPads`]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir/transform/no_landing_pads.rs +[mirtransform]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/transform/ +[`NoLandingPads`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/transform/no_landing_pads/struct.NoLandingPads.html [MIR visitor]: mir-visitor.html diff --git a/src/mir-visitor.md b/src/mir-visitor.md index 3a8b06c54..d8b3af497 100644 --- a/src/mir-visitor.md +++ b/src/mir-visitor.md @@ -7,7 +7,7 @@ them, generated via a single macro: `Visitor` (which operates on a `&Mir` and gives back shared references) and `MutVisitor` (which operates on a `&mut Mir` and gives back mutable references). -[m-v]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir/visit.rs +[m-v]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/visit/index.html To implement a visitor, you have to create a type that represents your visitor. Typically, this type wants to "hang on" to whatever @@ -41,7 +41,7 @@ A very simple example of a visitor can be found in [`NoLandingPads`]. That visitor doesn't even require any state: it just visits all terminators and removes their `unwind` successors. -[`NoLandingPads`]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir/transform/no_landing_pads.rs +[`NoLandingPads`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/transform/no_landing_pads/struct.NoLandingPads.html ## Traversal @@ -50,6 +50,6 @@ contains useful functions for walking the MIR CFG in [different standard orders][traversal] (e.g. pre-order, reverse post-order, and so forth). -[t]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir/traversal.rs +[t]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/traversal/index.html [traversal]: https://en.wikipedia.org/wiki/Tree_traversal diff --git a/src/query.md b/src/query.md index 500b9dec8..e3c554049 100644 --- a/src/query.md +++ b/src/query.md @@ -216,7 +216,7 @@ the big macro invocation in changed by the time you read this README, but at present it looks something like: -[maps-mod]: https://github.com/rust-lang/rust/blob/master/src/librustc/ty/maps/mod.rs +[maps-mod]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/maps/index.html ``` define_maps! { <'tcx> @@ -270,7 +270,7 @@ Let's go over them one by one: of `Steal` for more details. New uses of `Steal` should **not** be added without alerting `@rust-lang/compiler`. -[dep-node]: https://github.com/rust-lang/rust/blob/master/src/librustc/dep_graph/dep_node.rs +[dep-node]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/dep_graph/struct.DepNode.html So, to add a query: diff --git a/src/rustc-driver.md b/src/rustc-driver.md index 23a036e73..af3c3c099 100644 --- a/src/rustc-driver.md +++ b/src/rustc-driver.md @@ -67,10 +67,10 @@ pervasive lifetimes. The `rustc::ty::tls` module is used to access these thread-locals, although you should rarely need to touch it. -[`rustc_driver`]: https://github.com/rust-lang/rust/tree/master/src/librustc_driver -[`CompileState`]: https://github.com/rust-lang/rust/blob/master/src/librustc_driver/driver.rs -[`Session`]: https://github.com/rust-lang/rust/blob/master/src/librustc/session/mod.rs -[`TyCtxt`]: https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs -[`CodeMap`]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs +[`rustc_driver`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/ +[`CompileState`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/struct.CompileState.html +[`Session`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html +[`TyCtxt`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html +[`CodeMap`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html [stupid-stats]: https://github.com/nrc/stupid-stats [Appendix A]: appendix-stupid-stats.html \ No newline at end of file diff --git a/src/the-parser.md b/src/the-parser.md index 456f0a9ea..623a38e67 100644 --- a/src/the-parser.md +++ b/src/the-parser.md @@ -34,9 +34,9 @@ all the information needed while parsing, as well as the `CodeMap` itself. [libsyntax]: https://github.com/rust-lang/rust/tree/master/src/libsyntax [rustc_errors]: https://github.com/rust-lang/rust/tree/master/src/librustc_errors [ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree -[`CodeMap`]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs -[ast module]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/ast.rs +[`CodeMap`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html +[ast module]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/index.html [parser module]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/parse -[`Parser`]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/parser.rs -[`StringReader`]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/lexer/mod.rs -[visit module]: https://github.com/rust-lang/rust/blob/master/src/libsyntax/visit.rs +[`Parser`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/parser/struct.Parser.html +[`StringReader`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/lexer/struct.StringReader.html +[visit module]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/visit/index.html diff --git a/src/traits-lowering-module.md b/src/traits-lowering-module.md index c47b8fbe8..27c4f29bc 100644 --- a/src/traits-lowering-module.md +++ b/src/traits-lowering-module.md @@ -4,7 +4,7 @@ The program clauses described in the [lowering rules](./traits-lowering-rules.html) section are actually created in the [`rustc_traits::lowering`][lowering] module. -[lowering]: https://github.com/rust-lang/rust/tree/master/src/librustc_traits/lowering.rs +[lowering]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_traits/lowering/ ## The `program_clauses_for` query From 1aceb34508fd16e13fbb167e838d9952f902968e Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Thu, 5 Apr 2018 08:52:55 -0400 Subject: [PATCH 182/648] link directly to the compile-input function --- src/appendix-stupid-stats.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/appendix-stupid-stats.md b/src/appendix-stupid-stats.md index 7aef3a87b..87e2ba6dc 100644 --- a/src/appendix-stupid-stats.md +++ b/src/appendix-stupid-stats.md @@ -87,15 +87,16 @@ in [librustc_back](https://github.com/rust-lang/rust/tree/master/src/librustc_ba (which also contains some things used primarily during translation). All these phases are coordinated by the driver. To see the exact sequence, look -at the `compile_input` function in [librustc_driver/driver.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/). -The driver (which is found in [librust_driver](https://github.com/rust-lang/rust/tree/master/src/librustc_driver)) -handles all the highest level coordination of compilation - handling command -line arguments, maintaining compilation state (primarily in the `Session`), and -calling the appropriate code to run each phase of compilation. It also handles +at [the `compile_input` function in `librustc_driver`][compile-input]. +The driver handles all the highest level coordination of compilation - handling +command-line arguments, maintaining compilation state (primarily in the `Session`), +and calling the appropriate code to run each phase of compilation. It also handles high level coordination of pretty printing and testing. To create a drop-in compiler replacement or a compiler replacement, we leave most of compilation alone and customise the driver using its APIs. +[compile-input]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/fn.compile_input.html + ## The driver customisation APIs From f25f2989637c0df4985463aa83d41097340c8305 Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Thu, 5 Apr 2018 15:24:38 -0400 Subject: [PATCH 183/648] Fixing the long lines in appendix-stupid-status#100 --- src/appendix-code-index.md | 20 ++++++++++---------- src/appendix-stupid-stats.md | 13 +++++++------ 2 files changed, 17 insertions(+), 16 deletions(-) diff --git a/src/appendix-code-index.md b/src/appendix-code-index.md index 49fe08ee3..62edd0f5b 100644 --- a/src/appendix-code-index.md +++ b/src/appendix-code-index.md @@ -6,17 +6,17 @@ compiler. Item | Kind | Short description | Chapter | Declaration ----------------|----------|-----------------------------|--------------------|------------------- -`CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser] | [src/libsyntax/codemap.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs) -`CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver] | [src/librustc_driver/driver.rs](https://github.com/rust-lang/rust/blob/master/src/librustc_driver/driver.rs) +`CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser] | [src/libsyntax/codemap.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html) +`CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver] | [src/librustc_driver/driver.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/struct.CompileState.html) `DocContext` | struct | A state container used by rustdoc when crawling through a crate to gather its documentation | [Rustdoc] | [src/librustdoc/core.rs](https://github.com/rust-lang/rust/blob/master/src/librustdoc/core.rs) -`ast::Crate` | struct | Syntax-level representation of a parsed crate | [The parser] | [src/librustc/hir/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/ast.rs) -`hir::Crate` | struct | More abstract, compiler-friendly form of a crate's AST | [The Hir] | [src/librustc/hir/mod.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/hir/mod.rs) -`ParseSess` | struct | This struct contains information about a parsing session | [the Parser] | [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs) -`Session` | struct | The data associated with a compilation session | [the Parser], [The Rustc Driver] | [src/librustc/session/mod.html](https://github.com/rust-lang/rust/blob/master/src/librustc/session/mod.rs) -`StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser] | [src/libsyntax/parse/lexer/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/lexer/mod.rs) -`TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules] | [src/librustc/ty/trait_def.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/trait_def.rs) -`Ty<'tcx>` | struct | This is the internal representation of a type used for type checking | [Type checking] | [src/librustc/ty/mod.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/mod.rs) -`TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules] | [src/librustc/ty/context.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/ty/context.rs) +`ast::Crate` | struct | Syntax-level representation of a parsed crate | [The parser] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Crate.html) +`hir::Crate` | struct | More abstract, compiler-friendly form of a crate's AST | [The Hir] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Crate.html) +`ParseSess` | struct | This struct contains information about a parsing session | [the Parser] | [src/libsyntax/parse/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/struct.ParseSess.html) +`Session` | struct | The data associated with a compilation session | [the Parser], [The Rustc Driver] | [src/librustc/session/mod.html](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html) +`StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser] | [src/libsyntax/parse/lexer/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/lexer/struct.StringReader.html) +`TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules] | [src/librustc/ty/trait_def.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/trait_def/struct.TraitDef.html) +`Ty<'tcx>` | struct | This is the internal representation of a type used for type checking | [Type checking] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/type.Ty.html) +`TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules] | [src/librustc/ty/context.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html) [The HIR]: hir.html [The parser]: the-parser.html diff --git a/src/appendix-stupid-stats.md b/src/appendix-stupid-stats.md index 87e2ba6dc..91da6d2a8 100644 --- a/src/appendix-stupid-stats.md +++ b/src/appendix-stupid-stats.md @@ -88,12 +88,13 @@ in [librustc_back](https://github.com/rust-lang/rust/tree/master/src/librustc_ba All these phases are coordinated by the driver. To see the exact sequence, look at [the `compile_input` function in `librustc_driver`][compile-input]. -The driver handles all the highest level coordination of compilation - handling -command-line arguments, maintaining compilation state (primarily in the `Session`), -and calling the appropriate code to run each phase of compilation. It also handles -high level coordination of pretty printing and testing. To create a drop-in -compiler replacement or a compiler replacement, we leave most of compilation -alone and customise the driver using its APIs. +The driver handles all the highest level coordination of compilation - + 1. handling command-line arguments + 2. maintaining compilation state (primarily in the `Session`) + 3. calling the appropriate code to run each phase of compilation + 4. handles high level coordination of pretty printing and testing. +To create a drop-in compiler replacement or a compiler replacement, +we leave most of compilation alone and customise the driver using its APIs. [compile-input]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/fn.compile_input.html From 43341c8894d036a70ebcb782f93da124ba9ae525 Mon Sep 17 00:00:00 2001 From: varkor Date: Thu, 5 Apr 2018 13:24:08 +0100 Subject: [PATCH 184/648] Consolidate and fix code blocks --- README.md | 4 +- src/appendix-background.md | 6 +- src/appendix-stupid-stats.md | 16 +-- src/compiletest.md | 3 +- src/conventions.md | 6 +- src/high-level-overview.md | 2 +- src/hir.md | 4 +- src/how-to-build-and-run.md | 8 +- src/incrcomp-debugging.md | 16 +-- src/macro-expansion.md | 6 +- src/method-lookup.md | 15 +-- src/mir-passes.md | 6 +- src/mir-regionck.md | 206 +++++++++++++++++++------------- src/mir-visitor.md | 6 +- src/mir.md | 32 ++--- src/miri.md | 4 +- src/query.md | 24 ++-- src/rustdoc.md | 2 +- src/tests/adding.md | 10 +- src/tests/running.md | 18 +-- src/trait-caching.md | 2 +- src/trait-hrtb.md | 6 +- src/trait-resolution.md | 38 +++--- src/traits-associated-types.md | 38 +++--- src/traits-canonical-queries.md | 70 +++++++---- src/traits-canonicalization.md | 118 +++++++++++------- src/traits-goals-and-clauses.md | 76 +++++++----- src/traits-lowering-module.md | 2 +- src/traits-lowering-rules.md | 116 ++++++++++-------- src/traits-lowering-to-logic.md | 24 ++-- src/ty.md | 14 +-- src/type-checking.md | 4 +- src/type-inference.md | 20 +++- src/variance.md | 88 ++++++++------ 34 files changed, 584 insertions(+), 426 deletions(-) diff --git a/README.md b/README.md index 7e468fbb0..07d1749e8 100644 --- a/README.md +++ b/README.md @@ -30,8 +30,8 @@ To help prevent accidentally introducing broken links, we use the invoke this link checker, otherwise it will emit a warning saying it couldn't be found. -``` -$ cargo install mdbook-linkcheck +```bash +> cargo install mdbook-linkcheck ``` You will need `mdbook` version `>= 0.1`. `linkcheck` will be run automatically when you run `mdbook build`. diff --git a/src/appendix-background.md b/src/appendix-background.md index b49ad6d52..285d74477 100644 --- a/src/appendix-background.md +++ b/src/appendix-background.md @@ -21,7 +21,7 @@ all the remainder. Only at the end of the block is there the possibility of branching to more than one place (in MIR, we call that final statement the **terminator**): -``` +```mir bb0: { statement0; statement1; @@ -34,7 +34,7 @@ bb0: { Many expressions that you are used to in Rust compile down to multiple basic blocks. For example, consider an if statement: -```rust +```rust,ignore a = 1; if some_variable { b = 1; @@ -46,7 +46,7 @@ d = 1; This would compile into four basic blocks: -``` +```mir BB0: { a = 1; if some_variable { goto BB1 } else { goto BB2 } diff --git a/src/appendix-stupid-stats.md b/src/appendix-stupid-stats.md index 91da6d2a8..0c4de5c7a 100644 --- a/src/appendix-stupid-stats.md +++ b/src/appendix-stupid-stats.md @@ -3,7 +3,7 @@ > **Note:** This is a copy of `@nrc`'s amazing [stupid-stats]. You should find > a copy of the code on the GitHub repository although due to the compiler's > constantly evolving nature, there is no guarantee it'll compile on the first -> go. +> go. Many tools benefit from being a drop-in replacement for a compiler. By this, I mean that any user of the tool can use `mytool` in all the ways they would @@ -177,7 +177,7 @@ foo.rs` (assuming you have a Rust program called `foo.rs`. You can also pass any command line arguments that you would normally pass to rustc). When you run it you'll see output similar to -``` +```txt In crate: foo, Found 12 uses of `println!`; @@ -205,7 +205,7 @@ should dump stupid-stats' stdout to Cargo's stdout). Let's start with the `main` function for our tool, it is pretty simple: -``` +```rust,ignore fn main() { let args: Vec<_> = std::env::args().collect(); rustc_driver::run_compiler(&args, &mut StupidCalls::new()); @@ -223,7 +223,7 @@ this tool different from rustc. `StupidCalls` is a mostly empty struct: -``` +```rust,ignore struct StupidCalls { default_calls: RustcDefaultCalls, } @@ -238,7 +238,7 @@ to keep Cargo happy. Most of the rest of the impl of `CompilerCalls` is trivial: -``` +```rust,ignore impl<'a> CompilerCalls<'a> for StupidCalls { fn early_callback(&mut self, _: &getopts::Matches, @@ -300,7 +300,7 @@ tool does it's actual work by walking the AST. We do that by creating an AST visitor and making it walk the AST from the top (the crate root). Once we've walked the crate, we print the stats we've collected: -``` +```rust,ignore fn build_controller(&mut self, _: &Session) -> driver::CompileController<'a> { // We mostly want to do what rustc does, which is what basic() will return. let mut control = driver::CompileController::basic(); @@ -340,7 +340,7 @@ That is all it takes to create your own drop-in compiler replacement or custom compiler! For the sake of completeness I'll go over the rest of the stupid-stats tool. -``` +```rust struct StupidVisitor { println_count: usize, arg_counts: Vec, @@ -355,7 +355,7 @@ methods, these walk the AST taking no action. We override `visit_item` and functions, modules, traits, structs, and so forth, we're only interested in functions) and macros: -``` +```rust,ignore impl<'v> visit::Visitor<'v> for StupidVisitor { fn visit_item(&mut self, i: &'v ast::Item) { match i.node { diff --git a/src/compiletest.md b/src/compiletest.md index 011304d45..363c12d3b 100644 --- a/src/compiletest.md +++ b/src/compiletest.md @@ -61,7 +61,8 @@ which takes a single argument (which, in this case is a value of 1). (rather than the current Rust default of 101 at the time of this writing). The header command and the argument list (if present) are typically separated by a colon: -``` + +```rust,ignore // Copyright 2018 The Rust Project Developers. See the COPYRIGHT // file at the top-level directory of this distribution and at // http://rust-lang.org/COPYRIGHT. diff --git a/src/conventions.md b/src/conventions.md index 96571301f..89a986789 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -21,7 +21,7 @@ tidy script runs automatically when you do `./x.py test`. All files must begin with the following copyright notice: -``` +```rust // Copyright 2012-2013 The Rust Project Developers. See the COPYRIGHT // file at the top-level directory of this distribution and at // http://rust-lang.org/COPYRIGHT. @@ -48,7 +48,7 @@ tests -- it can be necessary to exempt yourself from this limit. In that case, you can add a comment towards the top of the file (after the copyright notice) like so: -``` +```rust // ignore-tidy-linelength ``` @@ -61,7 +61,7 @@ Prefer 4-space indent. # Coding for correctness Beyond formatting, there are a few other tips that are worth -following. +following. ## Prefer exhaustive matches diff --git a/src/high-level-overview.md b/src/high-level-overview.md index 041136548..e369db28c 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -19,7 +19,7 @@ compilation improves, that may change.) The dependency structure of these crates is roughly a diamond: -``` +```txt rustc_driver / | \ / | \ diff --git a/src/hir.md b/src/hir.md index f66468ffc..3d2fbede3 100644 --- a/src/hir.md +++ b/src/hir.md @@ -12,8 +12,8 @@ This chapter covers the main concepts of the HIR. You can view the HIR representation of your code by passing the `-Zunpretty=hir-tree` flag to rustc: -``` -cargo rustc -- -Zunpretty=hir-tree +```bash +> cargo rustc -- -Zunpretty=hir-tree ``` ### Out-of-band storage and the `Crate` type diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 6e292934b..535823dfd 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -70,8 +70,8 @@ Once you've created a config.toml, you are now ready to run `x.py`. There are a lot of options here, but let's start with what is probably the best "go to" command for building a local rust: -``` -./x.py build -i --stage 1 src/libstd +```bash +> ./x.py build -i --stage 1 src/libstd ``` What this command will do is the following: @@ -106,7 +106,7 @@ will execute the stage2 compiler (which we did not build, but which you will likely need to build at some point; for example, if you want to run the entire test suite). -``` +```bash > rustup toolchain link stage1 build//stage1 > rustup toolchain link stage2 build//stage2 ``` @@ -115,7 +115,7 @@ Now you can run the rustc you built with. If you run with `-vV`, you should see a version number ending in `-dev`, indicating a build from your local environment: -``` +```bash > rustc +stage1 -vV rustc 1.25.0-dev binary: rustc diff --git a/src/incrcomp-debugging.md b/src/incrcomp-debugging.md index 261b46eb0..3ad9f3a49 100644 --- a/src/incrcomp-debugging.md +++ b/src/incrcomp-debugging.md @@ -10,7 +10,7 @@ As an example, see `src/test/compile-fail/dep-graph-caller-callee.rs`. The idea is that you can annotate a test like: -```rust +```rust,ignore #[rustc_if_this_changed] fn foo() { } @@ -48,7 +48,7 @@ the graph. You can filter in three ways: To filter, use the `RUST_DEP_GRAPH_FILTER` environment variable, which should look like one of the following: -``` +```txt source_filter // nodes originating from source_filter -> target_filter // nodes that can reach target_filter source_filter -> target_filter // nodes in between source_filter and target_filter @@ -58,14 +58,14 @@ source_filter -> target_filter // nodes in between source_filter and target_filt A node is considered to match a filter if all of those strings appear in its label. So, for example: -``` +```txt RUST_DEP_GRAPH_FILTER='-> TypeckTables' ``` would select the predecessors of all `TypeckTables` nodes. Usually though you want the `TypeckTables` node for some particular fn, so you might write: -``` +```txt RUST_DEP_GRAPH_FILTER='-> TypeckTables & bar' ``` @@ -75,7 +75,7 @@ with `bar` in their name. Perhaps you are finding that when you change `foo` you need to re-type-check `bar`, but you don't think you should have to. In that case, you might do: -``` +```txt RUST_DEP_GRAPH_FILTER='Hir & foo -> TypeckTables & bar' ``` @@ -105,8 +105,10 @@ check of `bar` and you don't think there should be. You dump the dep-graph as described in the previous section and open `dep-graph.txt` to see something like: - Hir(foo) -> Collect(bar) - Collect(bar) -> TypeckTables(bar) +```txt +Hir(foo) -> Collect(bar) +Collect(bar) -> TypeckTables(bar) +``` That first edge looks suspicious to you. So you set `RUST_FORBID_DEP_GRAPH_EDGE` to `Hir&foo -> Collect&bar`, re-run, and diff --git a/src/macro-expansion.md b/src/macro-expansion.md index c55fdd136..ba807faf2 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -15,7 +15,7 @@ expansion works. It's helpful to have an example to refer to. For the remainder of this chapter, whenever we refer to the "example _definition_", we mean the following: -```rust +```rust,ignore macro_rules! printer { (print $mvar:ident) => { println!("{}", $mvar); @@ -45,7 +45,7 @@ worrying about _where_. For more information about tokens, see the Whenever we refer to the "example _invocation_", we mean the following snippet: -```rust +```rust,ignore printer!(print foo); // Assume `foo` is a variable defined somewhere else... ``` @@ -65,7 +65,7 @@ defined in [`src/libsyntax/ext/tt/macro_parser.rs`][code_mp]. The interface of the macro parser is as follows (this is slightly simplified): -```rust +```rust,ignore fn parse( sess: ParserSession, tts: TokenStream, diff --git a/src/method-lookup.md b/src/method-lookup.md index ac0e427db..dbbc62fa8 100644 --- a/src/method-lookup.md +++ b/src/method-lookup.md @@ -8,13 +8,13 @@ the code itself, naturally. One way to think of method lookup is that we convert an expression of the form: -```rust +```rust,ignore receiver.method(...) ``` into a more explicit [UFCS] form: -```rust +```rust,ignore Trait::method(ADJ(receiver), ...) // for a trait call ReceiverType::method(ADJ(receiver), ...) // for an inherent method call ``` @@ -24,7 +24,7 @@ autoderefs and then possibly an autoref (e.g., `&**receiver`). However we sometimes do other adjustments and coercions along the way, in particular unsizing (e.g., converting from `[T; n]` to `[T]`). -Method lookup is divided into two major phases: +Method lookup is divided into two major phases: 1. Probing ([`probe.rs`][probe]). The probe phase is when we decide what method to call and how to adjust the receiver. @@ -51,7 +51,7 @@ until it cannot be deref'd anymore, as well as applying an optional "unsize" step. So if the receiver has type `Rc>`, this might yield: -```rust +```rust,ignore Rc> Box<[T; 3]> [T; 3] @@ -99,9 +99,10 @@ So, let's continue our example. Imagine that we were calling a method that defines it with `&self` for the type `Rc` as well as a method on the type `Box` that defines `Foo` but with `&mut self`. Then we might have two candidates: - - &Rc> from the impl of `Foo` for `Rc` where `U=Box - &mut Box<[T; 3]>> from the inherent impl on `Box` where `U=[T; 3]` +```txt +&Rc> from the impl of `Foo` for `Rc` where `U=Box +&mut Box<[T; 3]>> from the inherent impl on `Box` where `U=[T; 3]` +``` ### Candidate search diff --git a/src/mir-passes.md b/src/mir-passes.md index 1d1e27df4..fd0c6cca1 100644 --- a/src/mir-passes.md +++ b/src/mir-passes.md @@ -52,13 +52,13 @@ fn main() { The files have names like `rustc.main.000-000.CleanEndRegions.after.mir`. These names have a number of parts: -``` +```txt rustc.main.000-000.CleanEndRegions.after.mir ---- --- --- --------------- ----- either before or after | | | name of the pass | | index of dump within the pass (usually 0, but some passes dump intermediate states) | index of the pass - def-path to the function etc being dumped + def-path to the function etc being dumped ``` You can also make more selective filters. For example, `main & CleanEndRegions` @@ -159,7 +159,7 @@ ensuring that the reads have already happened (remember that [queries are memoized](./query.html), so executing a query twice simply loads from a cache the second time): -``` +```txt mir_const(D) --read-by--> mir_const_qualif(D) | ^ stolen-by | diff --git a/src/mir-regionck.md b/src/mir-regionck.md index 529b42395..949b11d90 100644 --- a/src/mir-regionck.md +++ b/src/mir-regionck.md @@ -34,7 +34,7 @@ The MIR-based region analysis consists of two major functions: are used. - More details to come, though the [NLL RFC] also includes fairly thorough (and hopefully readable) coverage. - + [fvb]: appendix-background.html#free-vs-bound [NLL RFC]: http://rust-lang.github.io/rfcs/2094-nll.html @@ -82,7 +82,7 @@ The kinds of region elements are as follows: corresponds (intuitively) to some unknown set of other elements -- for details on skolemization, see the section [skolemization and universes](#skol). - + ## Causal tracking *to be written* -- describe how we can extend the values of a variable @@ -97,7 +97,7 @@ The kinds of region elements are as follows: From time to time we have to reason about regions that we can't concretely know. For example, consider this program: -```rust +```rust,ignore // A function that needs a static reference fn foo(x: &'static u32) { } @@ -122,10 +122,12 @@ stack, for example). But *how* do we reject it and *why*? When we type-check `main`, and in particular the call `bar(foo)`, we are going to wind up with a subtyping relationship like this one: - fn(&'static u32) <: for<'a> fn(&'a u32) - ---------------- ------------------- - the type of `foo` the type `bar` expects - +```txt +fn(&'static u32) <: for<'a> fn(&'a u32) +---------------- ------------------- +the type of `foo` the type `bar` expects +``` + We handle this sort of subtyping by taking the variables that are bound in the supertype and **skolemizing** them: this means that we replace them with @@ -135,8 +137,10 @@ regions" -- they represent, basically, "some unknown region". Once we've done that replacement, we have the following relation: - fn(&'static u32) <: fn(&'!1 u32) - +```txt +fn(&'static u32) <: fn(&'!1 u32) +``` + The key idea here is that this unknown region `'!1` is not related to any other regions. So if we can prove that the subtyping relationship is true for `'!1`, then it ought to be true for any region, which is @@ -147,7 +151,9 @@ subtypes, we check if their arguments have the desired relationship (fn arguments are [contravariant](./appendix-background.html#variance), so we swap the left and right here): - &'!1 u32 <: &'static u32 +```txt +&'!1 u32 <: &'static u32 +``` According to the basic subtyping rules for a reference, this will be true if `'!1: 'static`. That is -- if "some unknown region `!1`" lives @@ -168,7 +174,7 @@ put generic type parameters into this root universe (in this sense, there is not just one root universe, but one per item). So consider this function `bar`: -```rust +```rust,ignore struct Foo { } fn bar<'a, T>(t: &'a T) { @@ -185,7 +191,7 @@ Basically, the root universe contains all the names that Now let's extend `bar` a bit by adding a variable `x`: -```rust +```rust,ignore fn bar<'a, T>(t: &'a T) { let x: for<'b> fn(&'b u32) = ...; } @@ -195,7 +201,7 @@ Here, the name `'b` is not part of the root universe. Instead, when we "enter" into this `for<'b>` (e.g., by skolemizing it), we will create a child universe of the root, let's call it U1: -``` +```txt U0 (root universe) │ └─ U1 (child universe) @@ -207,7 +213,7 @@ with a new name, which we are identifying by its universe number: Now let's extend `bar` a bit by adding one more variable, `y`: -```rust +```rust,ignore fn bar<'a, T>(t: &'a T) { let x: for<'b> fn(&'b u32) = ...; let y: for<'c> fn(&'b u32) = ...; @@ -218,7 +224,7 @@ When we enter *this* type, we will again create a new universe, which we'll call `U2`. Its parent will be the root universe, and U1 will be its sibling: -``` +```txt U0 (root universe) │ ├─ U1 (child universe) @@ -257,11 +263,11 @@ children, that inference variable X would have to be in U0. And since X is in U0, it cannot name anything from U1 (or U2). This is perhaps easiest to see by using a kind of generic "logic" example: -``` +```txt exists { forall { ... /* Y is in U1 ... */ } forall { ... /* Z is in U2 ... */ } -} +} ``` Here, the only way for the two foralls to interact would be through X, @@ -290,8 +296,10 @@ does not say region elements **will** appear. In the region inference engine, outlives constraints have the form: - V1: V2 @ P - +```txt +V1: V2 @ P +``` + where `V1` and `V2` are region indices, and hence map to some region variable (which may be universally or existentially quantified). The `P` here is a "point" in the control-flow graph; it's not important @@ -338,8 +346,10 @@ for universal regions from the fn signature.) Put another way, the "universal regions" check can be considered to be checking constraints like: - {skol(1)}: V1 - +```txt +{skol(1)}: V1 +``` + where `{skol(1)}` is like a constant set, and V1 is the variable we made to represent the `!1` region. @@ -348,30 +358,40 @@ made to represent the `!1` region. OK, so far so good. Now let's walk through what would happen with our first example: - fn(&'static u32) <: fn(&'!1 u32) @ P // this point P is not imp't here +```txt +fn(&'static u32) <: fn(&'!1 u32) @ P // this point P is not imp't here +``` The region inference engine will create a region element domain like this: - { CFG; end('static); skol(1) } - --- ------------ ------- from the universe `!1` - | 'static is always in scope - all points in the CFG; not especially relevant here +```txt +{ CFG; end('static); skol(1) } + --- ------------ ------- from the universe `!1` + | 'static is always in scope + all points in the CFG; not especially relevant here +``` It will always create two universal variables, one representing `'static` and one representing `'!1`. Let's call them Vs and V1. They will have initial values like so: - Vs = { CFG; end('static) } // it is in U0, so can't name anything else - V1 = { skol(1) } - +```txt +Vs = { CFG; end('static) } // it is in U0, so can't name anything else +V1 = { skol(1) } +``` + From the subtyping constraint above, we would have an outlives constraint like - '!1: 'static @ P +```txt +'!1: 'static @ P +``` To process this, we would grow the value of V1 to include all of Vs: - Vs = { CFG; end('static) } - V1 = { CFG; end('static), skol(1) } +```txt +Vs = { CFG; end('static) } +V1 = { CFG; end('static), skol(1) } +``` At that point, constraint propagation is complete, because all the outlives relationships are satisfied. Then we would go to the "check @@ -385,34 +405,44 @@ In this case, `V1` *did* grow too large -- it is not known to outlive What about this subtyping relationship? - for<'a> fn(&'a u32, &'a u32) - <: - for<'b, 'c> fn(&'b u32, &'c u32) - -Here we would skolemize the supertype, as before, yielding: +```txt +for<'a> fn(&'a u32, &'a u32) + <: +for<'b, 'c> fn(&'b u32, &'c u32) +``` + +Here we would skolemize the supertype, as before, yielding: + +```txt +for<'a> fn(&'a u32, &'a u32) + <: +fn(&'!1 u32, &'!2 u32) +``` - for<'a> fn(&'a u32, &'a u32) - <: - fn(&'!1 u32, &'!2 u32) - then we instantiate the variable on the left-hand side with an existential in universe U2, yielding the following (`?n` is a notation for an existential variable): - fn(&'?3 u32, &'?3 u32) - <: - fn(&'!1 u32, &'!2 u32) - +```txt +fn(&'?3 u32, &'?3 u32) + <: +fn(&'!1 u32, &'!2 u32) +``` + Then we break this down further: - &'!1 u32 <: &'?3 u32 - &'!2 u32 <: &'?3 u32 - +```txt +&'!1 u32 <: &'?3 u32 +&'!2 u32 <: &'?3 u32 +``` + and even further, yield up our region constraints: - '!1: '?3 - '!2: '?3 - +```txt +'!1: '?3 +'!2: '?3 +``` + Note that, in this case, both `'!1` and `'!2` have to outlive the variable `'?3`, but the variable `'?3` is not forced to outlive anything else. Therefore, it simply starts and ends as the empty set @@ -430,15 +460,17 @@ common lifetime of our arguments. -nmatsakis) [ohdeargoditsallbroken]: https://github.com/rust-lang/rust/issues/32330#issuecomment-202536977 -## Final example +## Final example Let's look at one last example. We'll extend the previous one to have a return type: - for<'a> fn(&'a u32, &'a u32) -> &'a u32 - <: - for<'b, 'c> fn(&'b u32, &'c u32) -> &'b u32 - +```txt +for<'a> fn(&'a u32, &'a u32) -> &'a u32 + <: +for<'b, 'c> fn(&'b u32, &'c u32) -> &'b u32 +``` + Despite seeming very similar to the previous example, this case is going to get an error. That's good: the problem is that we've gone from a fn that promises to return one of its two arguments, to a fn that is promising to return the @@ -446,45 +478,59 @@ first one. That is unsound. Let's see how it plays out. First, we skolemize the supertype: - for<'a> fn(&'a u32, &'a u32) -> &'a u32 - <: - fn(&'!1 u32, &'!2 u32) -> &'!1 u32 - +```txt +for<'a> fn(&'a u32, &'a u32) -> &'a u32 + <: +fn(&'!1 u32, &'!2 u32) -> &'!1 u32 +``` + Then we instantiate the subtype with existentials (in U2): - fn(&'?3 u32, &'?3 u32) -> &'?3 u32 - <: - fn(&'!1 u32, &'!2 u32) -> &'!1 u32 - +```txt +fn(&'?3 u32, &'?3 u32) -> &'?3 u32 + <: +fn(&'!1 u32, &'!2 u32) -> &'!1 u32 +``` + And now we create the subtyping relationships: - &'!1 u32 <: &'?3 u32 // arg 1 - &'!2 u32 <: &'?3 u32 // arg 2 - &'?3 u32 <: &'!1 u32 // return type - +```txt +&'!1 u32 <: &'?3 u32 // arg 1 +&'!2 u32 <: &'?3 u32 // arg 2 +&'?3 u32 <: &'!1 u32 // return type +``` + And finally the outlives relationships. Here, let V1, V2, and V3 be the variables we assign to `!1`, `!2`, and `?3` respectively: - V1: V3 - V2: V3 - V3: V1 - +```txt +V1: V3 +V2: V3 +V3: V1 +``` + Those variables will have these initial values: - V1 in U1 = {skol(1)} - V2 in U2 = {skol(2)} - V3 in U2 = {} - +```txt +V1 in U1 = {skol(1)} +V2 in U2 = {skol(2)} +V3 in U2 = {} +``` + Now because of the `V3: V1` constraint, we have to add `skol(1)` into `V3` (and indeed it is visible from `V3`), so we get: - V3 in U2 = {skol(1)} - +```txt +V3 in U2 = {skol(1)} +``` + then we have this constraint `V2: V3`, so we wind up having to enlarge `V2` to include `skol(1)` (which it can also see): - V2 in U2 = {skol(1), skol(2)} - +```txt +V2 in U2 = {skol(1), skol(2)} +``` + Now contraint propagation is done, but when we check the outlives relationships, we find that `V2` includes this new element `skol(1)`, so we report an error. diff --git a/src/mir-visitor.md b/src/mir-visitor.md index d8b3af497..265769b34 100644 --- a/src/mir-visitor.md +++ b/src/mir-visitor.md @@ -13,7 +13,7 @@ To implement a visitor, you have to create a type that represents your visitor. Typically, this type wants to "hang on" to whatever state you will need while processing MIR: -```rust +```rust,ignore struct MyVisitor<...> { tcx: TyCtxt<'cx, 'tcx, 'tcx>, ... @@ -22,10 +22,10 @@ struct MyVisitor<...> { and you then implement the `Visitor` or `MutVisitor` trait for that type: -```rust +```rust,ignore impl<'tcx> MutVisitor<'tcx> for NoLandingPads { fn visit_foo(&mut self, ...) { - // ... + ... self.super_foo(...); } } diff --git a/src/mir.md b/src/mir.md index 58479cf96..8e45a62bc 100644 --- a/src/mir.md +++ b/src/mir.md @@ -69,12 +69,12 @@ fn main() { You should see something like: -``` +```mir // WARNING: This output format is intended for human consumers only // and is subject to change without notice. Knock yourself out. fn main() -> () { ... -} +} ``` This is the MIR format for the `main` function. @@ -82,7 +82,7 @@ This is the MIR format for the `main` function. **Variable declarations.** If we drill in a bit, we'll see it begins with a bunch of variable declarations. They look like this: -``` +```mir let mut _0: (); // return place scope 1 { let mut _1: std::vec::Vec; // "vec" in scope 1 at src/main.rs:2:9: 2:16 @@ -107,8 +107,8 @@ program (which names were in scope when). it may look slightly different when you view it, and I am ignoring some of the comments): -``` -bb0: { +```mir +bb0: { StorageLive(_1); _1 = const >::new() -> bb2; } @@ -117,7 +117,7 @@ bb0: { A basic block is defined by a series of **statements** and a final **terminator**. In this case, there is one statement: -``` +```mir StorageLive(_1); ``` @@ -129,7 +129,7 @@ allocate stack space. The **terminator** of the block `bb0` is the call to `Vec::new`: -``` +```mir _1 = const >::new() -> bb2; ``` @@ -142,8 +142,8 @@ possible, and hence we list only one succssor block, `bb2`. If we look ahead to `bb2`, we will see it looks like this: -``` -bb2: { +```mir +bb2: { StorageLive(_3); _3 = &mut _1; _2 = const >::push(move _3, const 1i32) -> [return: bb3, unwind: bb4]; @@ -153,13 +153,13 @@ bb2: { Here there are two statements: another `StorageLive`, introducing the `_3` temporary, and then an assignment: -``` +```mir _3 = &mut _1; ``` Assignments in general have the form: -``` +```txt = ``` @@ -169,7 +169,7 @@ value: in this case, the rvalue is a mutable borrow expression, which looks like `&mut `. So we can kind of define a grammar for rvalues like so: -``` +```txt = & (mut)? | + | - @@ -178,7 +178,7 @@ rvalues like so: = Constant | copy Place | move Place -``` +``` As you can see from this grammar, rvalues cannot be nested -- they can only reference places and constants. Moreover, when you use a place, @@ -188,7 +188,7 @@ for a place of any type). So, for example, if we had the expression `x = a + b + c` in Rust, that would get compile to two statements and a temporary: -``` +```mir TMP1 = a + b x = TMP1 + c ``` @@ -214,14 +214,14 @@ but [you can read about those below](#promoted)). we pass around `BasicBlock` values, which are [newtype'd] indices into this vector. - **Statements** are represented by the type `Statement`. -- **Terminators** are represented by the `Terminator`. +- **Terminators** are represented by the `Terminator`. - **Locals** are represented by a [newtype'd] index type `Local`. The data for a local variable is found in the `Mir` (the `local_decls` vector). There is also a special constant `RETURN_PLACE` identifying the special "local" representing the return value. - **Places** are identified by the enum `Place`. There are a few variants: - Local variables like `_1` - - Static variables `FOO` + - Static variables `FOO` - **Projections**, which are fields or other things that "project out" from a base place. So e.g. the place `_1.f` is a projection, with `f` being the "projection element and `_1` being the base diff --git a/src/miri.md b/src/miri.md index 4b0a600be..fb6675f50 100644 --- a/src/miri.md +++ b/src/miri.md @@ -14,7 +14,7 @@ placed into metadata. Once you have a use-site like -```rust +```rust,ignore type Foo = [u8; FOO - 42]; ``` @@ -24,7 +24,7 @@ create items that use the type (locals, constants, function arguments, ...). To obtain the (in this case empty) parameter environment, one can call `let param_env = tcx.param_env(length_def_id);`. The `GlobalId` needed is -```rust +```rust,ignore let gid = GlobalId { promoted: None, instance: Instance::mono(length_def_id), diff --git a/src/query.md b/src/query.md index e3c554049..2c518ee55 100644 --- a/src/query.md +++ b/src/query.md @@ -41,7 +41,7 @@ To invoke a query is simple. The tcx ("type context") offers a method for each defined query. So, for example, to invoke the `type_of` query, you would just do this: -```rust +```rust,ignore let ty = tcx.type_of(some_def_id); ``` @@ -59,7 +59,7 @@ better user experience. In order to recover from a cycle, you don't get to use the nice method-call-style syntax. Instead, you invoke using the `try_get` method, which looks roughly like this: -```rust +```rust,ignore use ty::maps::queries; ... match queries::type_of::try_get(tcx, DUMMY_SP, self.did) { @@ -87,7 +87,7 @@ will be reported due to this cycle by some other bit of code. In that case, you can invoke `err.cancel()` to not emit any error. It is traditional to then invoke: -``` +```rust,ignore tcx.sess.delay_span_bug(some_span, "some message") ``` @@ -126,7 +126,7 @@ on how that works). Providers always have the same signature: -```rust +```rust,ignore fn provider<'cx, 'tcx>(tcx: TyCtxt<'cx, 'tcx, 'tcx>, key: QUERY_KEY) -> QUERY_RESULT @@ -146,7 +146,7 @@ When the tcx is created, it is given the providers by its creator using the `Providers` struct. This struct is generated by the macros here, but it is basically a big list of function pointers: -```rust +```rust,ignore struct Providers { type_of: for<'cx, 'tcx> fn(TyCtxt<'cx, 'tcx, 'tcx>, DefId) -> Ty<'tcx>, ... @@ -163,7 +163,7 @@ throughout the other `rustc_*` crates. This is done by invoking various `provide` functions. These functions tend to look something like this: -```rust +```rust,ignore pub fn provide(providers: &mut Providers) { *providers = Providers { type_of, @@ -180,7 +180,7 @@ before.) So, if we want to add a provider for some other query, let's call it `fubar`, into the crate above, we might modify the `provide()` function like so: -```rust +```rust,ignore pub fn provide(providers: &mut Providers) { *providers = Providers { type_of, @@ -189,7 +189,7 @@ pub fn provide(providers: &mut Providers) { }; } -fn fubar<'cx, 'tcx>(tcx: TyCtxt<'cx, 'tcx>, key: DefId) -> Fubar<'tcx> { .. } +fn fubar<'cx, 'tcx>(tcx: TyCtxt<'cx, 'tcx>, key: DefId) -> Fubar<'tcx> { ... } ``` N.B. Most of the `rustc_*` crates only provide **local @@ -218,7 +218,7 @@ something like: [maps-mod]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/maps/index.html -``` +```rust,ignore define_maps! { <'tcx> /// Records the type of every item. [] fn type_of: TypeOfItem(DefId) -> Ty<'tcx>, @@ -229,7 +229,7 @@ define_maps! { <'tcx> Each line of the macro defines one query. The name is broken up like this: -``` +```rust,ignore [] fn type_of: TypeOfItem(DefId) -> Ty<'tcx>, ^^ ^^^^^^^ ^^^^^^^^^^ ^^^^^ ^^^^^^^^ | | | | | @@ -288,7 +288,7 @@ describing the query. Each such struct implements the key/value of that particular query. Basically the code generated looks something like this: -```rust +```rust,ignore // Dummy struct representing a particular kind of query: pub struct type_of<'tcx> { phantom: PhantomData<&'tcx ()> } @@ -306,7 +306,7 @@ this trait is optional if the query key is `DefId`, but if you *don't* implement it, you get a pretty generic error ("processing `foo`..."). You can put new impls into the `config` module. They look something like this: -```rust +```rust,ignore impl<'tcx> QueryDescription for queries::type_of<'tcx> { fn describe(tcx: TyCtxt, key: DefId) -> String { format!("computing the type of `{}`", tcx.item_path_str(key)) diff --git a/src/rustdoc.md b/src/rustdoc.md index e075087fc..36195c3a5 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -70,7 +70,7 @@ The main process of crate crawling is done in `clean/mod.rs` through several implementations of the `Clean` trait defined within. This is a conversion trait, which defines one method: -```rust +```rust,ignore pub trait Clean { fn clean(&self, cx: &DocContext) -> T; } diff --git a/src/tests/adding.md b/src/tests/adding.md index f777a458c..f97f4a57e 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -100,7 +100,7 @@ are normally put after the short comment that explains the point of this test. For example, this test uses the `// compile-flags` command to specify a custom flag to give to rustc when the test is compiled: -```rust +```rust,ignore // Copyright 2017 The Rust Project Developers. blah blah blah. // ... // except according to those terms. @@ -198,7 +198,7 @@ incremental, though incremental tests are somewhat different). Revisions allow a single test file to be used for multiple tests. This is done by adding a special header at the top of the file: -``` +```rust // revisions: foo bar baz ``` @@ -211,7 +211,7 @@ You can also customize headers and expected error messages to a particular revision. To do this, add `[foo]` (or `bar`, `baz`, etc) after the `//` comment, like so: -``` +```rust // A flag to pass in only for cfg `foo`: //[foo]compile-flags: -Z verbose @@ -284,7 +284,7 @@ between platforms, mainly about filenames: Sometimes these built-in normalizations are not enough. In such cases, you may provide custom normalization rules using the header commands, e.g. -``` +```rust // normalize-stdout-test: "foo" -> "bar" // normalize-stderr-32bit: "fn\(\) \(32 bits\)" -> "fn\(\) \($$PTR bits\)" // normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)" @@ -298,7 +298,7 @@ default regex flavor provided by `regex` crate. The corresponding reference file will use the normalized output to test both 32-bit and 64-bit platforms: -``` +```txt ... | = note: source type: fn() ($PTR bits) diff --git a/src/tests/running.md b/src/tests/running.md index 4e8c71590..03eb8ccc4 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -3,8 +3,8 @@ You can run the tests using `x.py`. The most basic command -- which you will almost never want to use! -- is as follows: -``` -./x.py test +```bash +> ./x.py test ``` This will build the full stage 2 compiler and then run the whole test @@ -17,7 +17,7 @@ The test results are cached and previously successful tests are `ignored` during testing. The stdout/stderr contents as well as a timestamp file for every test can be found under `build/ARCH/test/`. To force-rerun a test (e.g. in case the test runner fails to notice -a change) you can simply remove the timestamp file. +a change) you can simply remove the timestamp file. ## Running a subset of the test suites @@ -27,7 +27,7 @@ test" that can be used after modifying rustc to see if things are generally working correctly would be the following: ```bash -./x.py test --stage 1 src/test/{ui,compile-fail,run-pass} +> ./x.py test --stage 1 src/test/{ui,compile-fail,run-pass} ``` This will run the `ui`, `compile-fail`, and `run-pass` test suites, @@ -37,7 +37,7 @@ example, if you are hacking on debuginfo, you may be better off with the debuginfo test suite: ```bash -./x.py test --stage 1 src/test/debuginfo +> ./x.py test --stage 1 src/test/debuginfo ``` **Warning:** Note that bors only runs the tests with the full stage 2 @@ -51,8 +51,8 @@ Another common thing that people want to do is to run an **individual test**, often the test they are trying to fix. One way to do this is to invoke `x.py` with the `--test-args` option: -``` -./x.py test --stage 1 src/test/ui --test-args issue-1234 +```bash +> ./x.py test --stage 1 src/test/ui --test-args issue-1234 ``` Under the hood, the test runner invokes the standard rust test runner @@ -62,8 +62,8 @@ filtering for tests that include "issue-1234" in the name. Often, though, it's easier to just run the test by hand. Most tests are just `rs` files, so you can do something like -``` -rustc +stage1 src/test/ui/issue-1234.rs +```bash +> rustc +stage1 src/test/ui/issue-1234.rs ``` This is much faster, but doesn't always work. For example, some tests diff --git a/src/trait-caching.md b/src/trait-caching.md index 4c728d521..4b7d7e096 100644 --- a/src/trait-caching.md +++ b/src/trait-caching.md @@ -26,7 +26,7 @@ possible impl is this one, with def-id 22: [selection process]: ./trait-resolution.html#selection -```rust +```rust,ignore impl Foo for usize { ... } // Impl #22 ``` diff --git a/src/trait-hrtb.md b/src/trait-hrtb.md index d677db2c5..7f77f274c 100644 --- a/src/trait-hrtb.md +++ b/src/trait-hrtb.md @@ -18,14 +18,14 @@ trait Foo { Let's say we have a function `want_hrtb` that wants a type which implements `Foo<&'a isize>` for any `'a`: -```rust +```rust,ignore fn want_hrtb() where T : for<'a> Foo<&'a isize> { ... } ``` Now we have a struct `AnyInt` that implements `Foo<&'a isize>` for any `'a`: -```rust +```rust,ignore struct AnyInt; impl<'a> Foo<&'a isize> for AnyInt { } ``` @@ -71,7 +71,7 @@ set for `'0` is `{'0, '$a}`, and hence the check will succeed. Let's consider a failure case. Imagine we also have a struct -```rust +```rust,ignore struct StaticInt; impl Foo<&'static isize> for StaticInt; ``` diff --git a/src/trait-resolution.md b/src/trait-resolution.md index 7494d969a..5bf8f8716 100644 --- a/src/trait-resolution.md +++ b/src/trait-resolution.md @@ -13,13 +13,13 @@ see [*this* traits chapter](./traits.html). Trait resolution is the process of pairing up an impl with each reference to a trait. So, for example, if there is a generic function like: -```rust -fn clone_slice(x: &[T]) -> Vec { /*...*/ } +```rust,ignore +fn clone_slice(x: &[T]) -> Vec { ... } ``` and then a call to that function: -```rust +```rust,ignore let v: Vec = clone_slice(&[1, 2, 3]) ``` @@ -30,7 +30,7 @@ Note that in some cases, like generic functions, we may not be able to find a specific impl, but we can figure out that the caller must provide an impl. For example, consider the body of `clone_slice`: -```rust +```rust,ignore fn clone_slice(x: &[T]) -> Vec { let mut v = Vec::new(); for e in &x { @@ -143,7 +143,7 @@ otherwise the result is considered ambiguous. This process is easier if we work through some examples. Consider the following trait: -```rust +```rust,ignore trait Convert { fn convert(&self) -> Target; } @@ -154,14 +154,14 @@ converts from the (implicit) `Self` type to the `Target` type. If we wanted to permit conversion between `isize` and `usize`, we might implement `Convert` like so: -```rust -impl Convert for isize { /*...*/ } // isize -> usize -impl Convert for usize { /*...*/ } // usize -> isize +```rust,ignore +impl Convert for isize { ... } // isize -> usize +impl Convert for usize { ... } // usize -> isize ``` Now imagine there is some code like the following: -```rust +```rust,ignore let x: isize = ...; let y = x.convert(); ``` @@ -186,7 +186,7 @@ inference? But what happens if there are multiple impls where all the types unify? Consider this example: -```rust +```rust,ignore trait Get { fn get(&self) -> Self; } @@ -224,11 +224,11 @@ the same trait (or some subtrait) and which can match against the obligation. Consider this simple example: -```rust +```rust,ignore trait A1 { fn do_a1(&self); } -trait A2 : A1 { /*...*/ } +trait A2 : A1 { ... } trait B { fn do_b(&self); @@ -256,13 +256,13 @@ values found in the obligation, possibly yielding a type error. Suppose we have the following variation of the `Convert` example in the previous section: -```rust +```rust,ignore trait Convert { fn convert(&self) -> Target; } -impl Convert for isize { /*...*/ } // isize -> usize -impl Convert for usize { /*...*/ } // usize -> isize +impl Convert for isize { ... } // isize -> usize +impl Convert for usize { ... } // usize -> isize let x: isize = ...; let y: char = x.convert(); // NOTE: `y: char` now! @@ -296,11 +296,11 @@ everything out. Here is an example: -```rust -trait Foo { /*...*/ } -impl> Foo for Vec { /*...*/ } +```rust,ignore +trait Foo { ... } +impl> Foo for Vec { ... } -impl Bar for isize { /*...*/ } +impl Bar for isize { ... } ``` After one shallow round of selection for an obligation like `Vec diff --git a/src/traits-associated-types.md b/src/traits-associated-types.md index c91dc255f..bcf6b48ea 100644 --- a/src/traits-associated-types.md +++ b/src/traits-associated-types.md @@ -29,10 +29,10 @@ that is, simplified -- based on the types given in an impl. So, to continue with our example, the impl of `IntoIterator` for `Option` declares (among other things) that `Item = T`: -```rust +```rust,ignore impl IntoIterator for Option { type Item = T; - .. + ... } ``` @@ -51,9 +51,11 @@ In our logic, normalization is defined by a predicate impls. For example, the `impl` of `IntoIterator` for `Option` that we saw above would be lowered to a program clause like so: - forall { - Normalize( as IntoIterator>::Item -> T) - } +```txt +forall { + Normalize( as IntoIterator>::Item -> T) +} +``` (An aside: since we do not permit quantification over traits, this is really more like a family of predicates, one for each associated @@ -67,7 +69,7 @@ we've seen so far. Sometimes however we want to work with associated types that cannot be normalized. For example, consider this function: -```rust +```rust,ignore fn foo(...) { ... } ``` @@ -99,20 +101,24 @@ consider an associated type projection equal to another type?": We now introduce the `ProjectionEq` predicate to bring those two cases together. The `ProjectionEq` predicate looks like so: - ProjectionEq(::Item = U) +```txt +ProjectionEq(::Item = U) +``` and we will see that it can be proven *either* via normalization or skolemization. As part of lowering an associated type declaration from some trait, we create two program clauses for `ProjectionEq`: - forall { - ProjectionEq(::Item = U) :- - Normalize(::Item -> U) - } +```txt +forall { + ProjectionEq(::Item = U) :- + Normalize(::Item -> U) +} - forall { - ProjectionEq(::Item = (IntoIterator::Item)) - } +forall { + ProjectionEq(::Item = (IntoIterator::Item)) +} +``` These are the only two `ProjectionEq` program clauses we ever make for any given associated item. @@ -124,7 +130,9 @@ with unification. As described in the [type inference](./type-inference.html) section, unification is basically a procedure with a signature like this: - Unify(A, B) = Result<(Subgoals, RegionConstraints), NoSolution> +```txt +Unify(A, B) = Result<(Subgoals, RegionConstraints), NoSolution> +``` In other words, we try to unify two things A and B. That procedure might just fail, in which case we get back `Err(NoSolution)`. This diff --git a/src/traits-canonical-queries.md b/src/traits-canonical-queries.md index b291a2898..99916267d 100644 --- a/src/traits-canonical-queries.md +++ b/src/traits-canonical-queries.md @@ -19,12 +19,16 @@ In a traditional Prolog system, when you start a query, the solver will run off and start supplying you with every possible answer it can find. So given something like this: - ?- Vec: AsRef +```txt +?- Vec: AsRef +``` The solver might answer: - Vec: AsRef<[i32]> - continue? (y/n) +```txt +Vec: AsRef<[i32]> + continue? (y/n) +``` This `continue` bit is interesting. The idea in Prolog is that the solver is finding **all possible** instantiations of your query that @@ -35,34 +39,42 @@ response with our original query -- Rust's solver gives back a substitution instead). If we were to hit `y`, the solver might then give us another possible answer: - Vec: AsRef> - continue? (y/n) +```txt +Vec: AsRef> + continue? (y/n) +``` This answer derives from the fact that there is a reflexive impl (`impl AsRef for T`) for `AsRef`. If were to hit `y` again, then we might get back a negative response: - no +```txt +no +``` Naturally, in some cases, there may be no possible answers, and hence the solver will just give me back `no` right away: - ?- Box: Copy - no +```txt +?- Box: Copy + no +``` In some cases, there might be an infinite number of responses. So for example if I gave this query, and I kept hitting `y`, then the solver would never stop giving me back answers: - ?- Vec: Clone - Vec: Clone - continue? (y/n) - Vec>: Clone - continue? (y/n) - Vec>>: Clone - continue? (y/n) - Vec>>>: Clone - continue? (y/n) +```txt +?- Vec: Clone + Vec: Clone + continue? (y/n) + Vec>: Clone + continue? (y/n) + Vec>>: Clone + continue? (y/n) + Vec>>>: Clone + continue? (y/n) +``` As you can imagine, the solver will gleefully keep adding another layer of `Box` until we ask it to stop, or it runs out of memory. @@ -70,12 +82,16 @@ layer of `Box` until we ask it to stop, or it runs out of memory. Another interesting thing is that queries might still have variables in them. For example: - ?- Rc: Clone +```txt +?- Rc: Clone +``` might produce the answer: - Rc: Clone - continue? (y/n) +```txt +Rc: Clone + continue? (y/n) +``` After all, `Rc` is true **no matter what type `?T` is**. @@ -132,7 +148,7 @@ impls; among them, there are these two (for clarify, I've written the [borrow]: https://doc.rust-lang.org/std/borrow/trait.Borrow.html -```rust +```rust,ignore impl Borrow for T where T: ?Sized impl Borrow<[T]> for Vec where T: Sized ``` @@ -140,7 +156,7 @@ impl Borrow<[T]> for Vec where T: Sized **Example 1.** Imagine we are type-checking this (rather artificial) bit of code: -```rust +```rust,ignore fn foo(a: A, vec_b: Option) where A: Borrow { } fn main() { @@ -185,7 +201,7 @@ other sources, in which case we can try the trait query again. **Example 2.** We can now extend our previous example a bit, and assign a value to `u`: -```rust +```rust,ignore fn foo(a: A, vec_b: Option) where A: Borrow { } fn main() { @@ -210,11 +226,15 @@ Let's suppose that the type checker decides to revisit the Borrow`. `?U` is no longer an unbound inference variable; it now has a value, `Vec`. So, if we "refresh" the query with that value, we get: - Vec: Borrow> +```txt +Vec: Borrow> +``` This time, there is only one impl that applies, the reflexive impl: - impl Borrow for T where T: ?Sized +```txt +impl Borrow for T where T: ?Sized +``` Therefore, the trait checker will answer: diff --git a/src/traits-canonicalization.md b/src/traits-canonicalization.md index 9f3af36c9..576ca2055 100644 --- a/src/traits-canonicalization.md +++ b/src/traits-canonicalization.md @@ -42,14 +42,18 @@ This query contains two unbound variables, but it also contains the lifetime `'static`. The trait system generally ignores all lifetimes and treats them equally, so when canonicalizing, we will *also* replace any [free lifetime](./appendix-background.html#free-vs-bound) with a -canonical variable. Therefore, we get the following result: +canonical variable. Therefore, we get the following result: - ?0: Foo<'?1, ?2> - -Sometimes we write this differently, like so: +```txt +?0: Foo<'?1, ?2> +``` + +Sometimes we write this differently, like so: + +```txt +for { ?0: Foo<'?1, ?2> } +``` - for { ?0: Foo<'?1, ?2> } - This `for<>` gives some information about each of the canonical variables within. In this case, each `T` indicates a type variable, so `?0` and `?2` are types; the `L` indicates a lifetime varibale, so @@ -57,8 +61,10 @@ so `?0` and `?2` are types; the `L` indicates a lifetime varibale, so `CanonicalVarValues` array OV with the "original values" for each canonicalized variable: - [?A, 'static, ?B] - +```txt +[?A, 'static, ?B] +``` + We'll need this vector OV later, when we process the query response. ## Executing the query @@ -70,18 +76,24 @@ we create a substitution S from the canonical form containing a fresh inference variable (of suitable kind) for each canonical variable. So, for our example query: - for { ?0: Foo<'?1, ?2> } +```txt +for { ?0: Foo<'?1, ?2> } +``` the substitution S might be: - S = [?A, '?B, ?C] - +```txt +S = [?A, '?B, ?C] +``` + We can then replace the bound canonical variables (`?0`, etc) with these inference variables, yielding the following fully instantiated query: - ?A: Foo<'?B, ?C> - +```txt +?A: Foo<'?B, ?C> +``` + Remember that substitution S though! We're going to need it later. OK, now that we have a fresh inference context and an instantiated @@ -93,7 +105,7 @@ created. For example, if there were only one impl of `Foo`, like so: [cqqr]: ./traits-canonical-queries.html#query-response -``` +```rust,ignore impl<'a, X> Foo<'a, X> for Vec where X: 'a { ... } @@ -123,39 +135,49 @@ result substitution `var_values`, and some region constraints. To create this, we wind up re-using the substitution S that we created when first instantiating our query. To refresh your memory, we had a query - for { ?0: Foo<'?1, ?2> } +```txt +for { ?0: Foo<'?1, ?2> } +``` for which we made a substutition S: - S = [?A, '?B, ?C] - +```txt +S = [?A, '?B, ?C] +``` + We then did some work which unified some of those variables with other things. If we "refresh" S with the latest results, we get: - S = [Vec, '?D, ?E] - +```txt +S = [Vec, '?D, ?E] +``` + These are precisely the new values for the three input variables from our original query. Note though that they include some new variables (like `?E`). We can make those go away by canonicalizing again! We don't just canonicalize S, though, we canonicalize the whole query response QR: - QR = { - certainty: Proven, // or whatever - var_values: [Vec, '?D, ?E] // this is S - region_constraints: [?E: '?D], // from the impl - value: (), // for our purposes, just (), but - // in some cases this might have - // a type or other info - } +```txt +QR = { + certainty: Proven, // or whatever + var_values: [Vec, '?D, ?E] // this is S + region_constraints: [?E: '?D], // from the impl + value: (), // for our purposes, just (), but + // in some cases this might have + // a type or other info +} +``` The result would be as follows: - Canonical(QR) = for { - certainty: Proven, - var_values: [Vec, '?1, ?2] - region_constraints: [?2: '?1], - value: (), - } +```txt +Canonical(QR) = for { + certainty: Proven, + var_values: [Vec, '?1, ?2] + region_constraints: [?2: '?1], + value: (), +} +``` (One subtle point: when we canonicalize the query **result**, we do not use any special treatment for free lifetimes. Note that both @@ -172,20 +194,26 @@ In the previous section we produced a canonical query result. We now have to apply that result in our original context. If you recall, way back in the beginning, we were trying to prove this query: - ?A: Foo<'static, ?B> - +```txt +?A: Foo<'static, ?B> +``` + We canonicalized that into this: - for { ?0: Foo<'?1, ?2> } +```txt +for { ?0: Foo<'?1, ?2> } +``` and now we got back a canonical response: - for { - certainty: Proven, - var_values: [Vec, '?1, ?2] - region_constraints: [?2: '?1], - value: (), - } +```txt +for { + certainty: Proven, + var_values: [Vec, '?1, ?2] + region_constraints: [?2: '?1], + value: (), +} +``` We now want to apply that response to our context. Conceptually, how we do that is to (a) instantiate each of the canonical variables in @@ -193,19 +221,19 @@ the result with a fresh inference variable, (b) unify the values in the result with the original values, and then (c) record the region constraints for later. Doing step (a) would yield a result of -``` +```txt { certainty: Proven, var_values: [Vec, '?D, ?C] ^^ ^^^ fresh inference variables region_constraints: [?C: '?D], value: (), -} +} ``` Step (b) would then unify: -``` +```txt ?A with Vec 'static with '?D ?B with ?C diff --git a/src/traits-goals-and-clauses.md b/src/traits-goals-and-clauses.md index 8e5e82d01..97532195e 100644 --- a/src/traits-goals-and-clauses.md +++ b/src/traits-goals-and-clauses.md @@ -12,22 +12,24 @@ a few new superpowers. In Rust's solver, **goals** and **clauses** have the following forms (note that the two definitions reference one another): - Goal = DomainGoal // defined in the section below - | Goal && Goal - | Goal || Goal - | exists { Goal } // existential quantification - | forall { Goal } // universal quantification - | if (Clause) { Goal } // implication - | true // something that's trivially true - | ambiguous // something that's never provable - - Clause = DomainGoal - | Clause :- Goal // if can prove Goal, then Clause is true - | Clause && Clause - | forall { Clause } - - K = // a "kind" - | +```txt +Goal = DomainGoal // defined in the section below + | Goal && Goal + | Goal || Goal + | exists { Goal } // existential quantification + | forall { Goal } // universal quantification + | if (Clause) { Goal } // implication + | true // something that's trivially true + | ambiguous // something that's never provable + +Clause = DomainGoal + | Clause :- Goal // if can prove Goal, then Clause is true + | Clause && Clause + | forall { Clause } + +K = // a "kind" + | +``` The proof procedure for these sorts of goals is actually quite straightforward. Essentially, it's a form of depth-first search. The @@ -47,8 +49,10 @@ To define the set of *domain goals* in our system, we need to first introduce a few simple formulations. A **trait reference** consists of the name of a trait along with a suitable set of inputs P0..Pn: - TraitRef = P0: TraitName - +```txt +TraitRef = P0: TraitName +``` + So, for example, `u32: Display` is a trait reference, as is `Vec: IntoIterator`. Note that Rust surface syntax also permits some extra things, like associated type bindings (`Vec: IntoIterator`), that are not part of a trait reference. A **projection** consists of an associated item reference along with its inputs P0..Pm: - Projection = >::AssocItem +```txt +Projection = >::AssocItem +``` Given that, we can define a `DomainGoal` as follows: - DomainGoal = Implemented(TraitRef) - | ProjectionEq(Projection = Type) - | Normalize(Projection -> Type) - | FromEnv(TraitRef) - | FromEnv(Projection = Type) - | WellFormed(Type) - | WellFormed(TraitRef) - | WellFormed(Projection = Type) - | Outlives(Type, Region) - | Outlives(Region, Region) +```txt +DomainGoal = Implemented(TraitRef) + | ProjectionEq(Projection = Type) + | Normalize(Projection -> Type) + | FromEnv(TraitRef) + | FromEnv(Projection = Type) + | WellFormed(Type) + | WellFormed(TraitRef) + | WellFormed(Projection = Type) + | Outlives(Type, Region) + | Outlives(Region, Region) +``` - `Implemented(TraitRef)` -- true if the given trait is implemented for the given input types and lifetimes @@ -104,9 +112,11 @@ Given that, we can define a `DomainGoal` as follows: Most goals in our system are "inductive". In an inductive goal, circular reasoning is disallowed. Consider this example clause: +```txt Implemented(Foo: Bar) :- Implemented(Foo: Bar). - +``` + Considered inductively, this clause is useless: if we are trying to prove `Implemented(Foo: Bar)`, we would then recursively have to prove `Implemented(Foo: Bar)`, and that cycle would continue ad infinitum @@ -130,8 +140,10 @@ struct Foo { The default rules for auto traits say that `Foo` is `Send` if the types of its fields are `Send`. Therefore, we would have a rule like - Implemented(Foo: Send) :- - Implemented(Option>: Send). +```txt +Implemented(Foo: Send) :- + Implemented(Option>: Send). +``` As you can probably imagine, proving that `Option>: Send` is going to wind up circularly requiring us to prove that `Foo: Send` diff --git a/src/traits-lowering-module.md b/src/traits-lowering-module.md index 27c4f29bc..fa068f86a 100644 --- a/src/traits-lowering-module.md +++ b/src/traits-lowering-module.md @@ -49,7 +49,7 @@ standard [ui test] mechanisms to check them. In this case, there is a need only be a prefix of the error), but [the stderr file] contains the full details: -``` +```txt error: Implemented(T: Foo) :- ProjectionEq(::Item == i32), TypeOutlives(T \ : 'static), Implemented(T: std::iter::Iterator), Implemented(T: std::marker::Sized). --> $DIR/lower_impl.rs:15:1 diff --git a/src/traits-lowering-rules.md b/src/traits-lowering-rules.md index 544ae41b3..1a9717f9f 100644 --- a/src/traits-lowering-rules.md +++ b/src/traits-lowering-rules.md @@ -67,7 +67,7 @@ but Chalk isn't modeling those right now. Given a trait definition -```rust +```rust,ignore trait Trait // P0 == Self where WC { @@ -87,10 +87,12 @@ relationships between different kinds of domain goals. The first such rule from the trait header creates the mapping between the `FromEnv` and `Implemented` predicates: - // Rule Implemented-From-Env - forall { - Implemented(Self: Trait) :- FromEnv(Self: Trait) - } +```txt +// Rule Implemented-From-Env +forall { + Implemented(Self: Trait) :- FromEnv(Self: Trait) +} +``` @@ -101,17 +103,19 @@ The next few clauses have to do with implied bounds (see also [RFC 2089]: https://rust-lang.github.io/rfcs/2089-implied-bounds.html - // Rule Implied-Bound-From-Trait - // - // For each where clause WC: - forall { - FromEnv(WC) :- FromEnv(Self: Trait { + FromEnv(WC) :- FromEnv(Self: Trait // P0 == Self where WC { @@ -190,39 +194,43 @@ where WC We will produce a number of program clauses. The first two define the rules by which `ProjectionEq` can succeed; these two clauses are discussed -in detail in the [section on associated types](./traits-associated-types.html),, +in detail in the [section on associated types](./traits-associated-types.html), but reproduced here for reference: - // Rule ProjectionEq-Normalize - // - // ProjectionEq can succeed by normalizing: - forall { - ProjectionEq(>::AssocType = U) :- - Normalize(>::AssocType -> U) - } - - // Rule ProjectionEq-Skolemize - // - // ProjectionEq can succeed by skolemizing, see "associated type" - // chapter for more: - forall { - ProjectionEq( - >::AssocType = - (Trait::AssocType) - ) :- - // But only if the trait is implemented, and the conditions from - // the associated type are met as well: - Implemented(Self: Trait) - && WC1 - } +```txt + // Rule ProjectionEq-Normalize + // + // ProjectionEq can succeed by normalizing: + forall { + ProjectionEq(>::AssocType = U) :- + Normalize(>::AssocType -> U) + } + + // Rule ProjectionEq-Skolemize + // + // ProjectionEq can succeed by skolemizing, see "associated type" + // chapter for more: + forall { + ProjectionEq( + >::AssocType = + (Trait::AssocType) + ) :- + // But only if the trait is implemented, and the conditions from + // the associated type are met as well: + Implemented(Self: Trait) + && WC1 + } +``` The next rule covers implied bounds for the projection. In particular, the `Bounds` declared on the associated type must be proven to hold to show that the impl is well-formed, and hence we can rely on them elsewhere. - // XXX how exactly should we set this up? Have to be careful; - // presumably this has to be a kind of `FromEnv` setup. +```txt +// XXX how exactly should we set this up? Have to be careful; +// presumably this has to be a kind of `FromEnv` setup. +``` ### Lowering function and constant declarations @@ -234,7 +242,7 @@ values below](#constant-vals) for more details. Given an impl of a trait: -```rust +```rust,ignore impl Trait for A0 where WC { @@ -245,10 +253,12 @@ where WC Let `TraitRef` be the trait reference `A0: Trait`. Then we will create the following rules: - // Rule Implemented-From-Impl - forall { - Implemented(TraitRef) :- WC - } +```txt +// Rule Implemented-From-Impl +forall { + Implemented(TraitRef) :- WC +} +``` In addition, we will lower all of the *impl items*. @@ -258,7 +268,7 @@ In addition, we will lower all of the *impl items*. Given an impl that contains: -```rust +```rust,ignore impl Trait for A0 where WC { @@ -268,13 +278,15 @@ where WC We produce the following rule: - // Rule Normalize-From-Impl - forall { - forall { - Normalize(>::AssocType -> T) :- - WC && WC1 - } - } +```txt +// Rule Normalize-From-Impl +forall { + forall { + Normalize(>::AssocType -> T) :- + WC && WC1 + } +} +``` Note that `WC` and `WC1` both encode where-clauses that the impl can rely on. diff --git a/src/traits-lowering-to-logic.md b/src/traits-lowering-to-logic.md index 4e4e7cae9..f36794d2c 100644 --- a/src/traits-lowering-to-logic.md +++ b/src/traits-lowering-to-logic.md @@ -30,7 +30,7 @@ impl Clone for Vec where T: Clone { } We could map these declarations to some Horn clauses, written in a Prolog-like notation, as follows: -``` +```txt Clone(usize). Clone(Vec) :- Clone(?T). @@ -51,18 +51,18 @@ so by applying the rules recursively: - `Clone(Vec>)` is provable if: - `Clone(Vec)` is provable if: - `Clone(usize)` is provable. (Which is is, so we're all good.) - + But now suppose we tried to prove that `Clone(Vec)`. This would fail (after all, I didn't give an impl of `Clone` for `Bar`): - `Clone(Vec)` is provable if: - `Clone(Bar)` is provable. (But it is not, as there are no applicable rules.) - + We can easily extend the example above to cover generic traits with more than one input type. So imagine the `Eq` trait, which declares that `Self` is equatable with a value of type `T`: -```rust +```rust,ignore trait Eq { ... } impl Eq for usize { } impl> Eq> for Vec { } @@ -70,12 +70,12 @@ impl> Eq> for Vec { } That could be mapped as follows: -``` +```txt Eq(usize, usize). Eq(Vec, Vec) :- Eq(?T, ?U). ``` -So far so good. +So far so good. ## Type-checking normal functions @@ -90,7 +90,7 @@ that we need to prove, and those come from type-checking. Consider type-checking the function `foo()` here: -```rust +```rust,ignore fn foo() { bar::() } fn bar>() { } ``` @@ -105,7 +105,7 @@ If we wanted, we could write a Prolog predicate that defines the conditions under which `bar()` can be called. We'll say that those conditions are called being "well-formed": -``` +```txt barWellFormed(?U) :- Eq(?U, ?U). ``` @@ -113,7 +113,7 @@ Then we can say that `foo()` type-checks if the reference to `bar::` (that is, `bar()` applied to the type `usize`) is well-formed: -``` +```txt fooTypeChecks :- barWellFormed(usize). ``` @@ -122,7 +122,7 @@ If we try to prove the goal `fooTypeChecks`, it will succeed: - `fooTypeChecks` is provable if: - `barWellFormed(usize)`, which is provable if: - `Eq(usize, usize)`, which is provable because of an impl. - + Ok, so far so good. Let's move on to type-checking a more complex function. ## Type-checking generic functions: beyond Horn clauses @@ -134,7 +134,7 @@ a generic function, it turns out we need a stronger notion of goal than Prolog can be provide. To see what I'm talking about, let's revamp our previous example to make `foo` generic: -```rust +```rust,ignore fn foo>() { bar::() } fn bar>() { } ``` @@ -144,7 +144,7 @@ To type-check the body of `foo`, we need to be able to hold the type type-safe *for all types `T`*, not just for some specific type. We might express this like so: -``` +```txt fooTypeChecks :- // for all types T... forall { diff --git a/src/ty.md b/src/ty.md index 1fd86a6bd..44017dd5b 100644 --- a/src/ty.md +++ b/src/ty.md @@ -10,7 +10,7 @@ The `tcx` ("typing context") is the central data structure in the compiler. It is the context that you use to perform all manner of queries. The struct `TyCtxt` defines a reference to this shared context: -```rust +```rust,ignore tcx: TyCtxt<'a, 'gcx, 'tcx> // -- ---- ---- // | | | @@ -47,7 +47,7 @@ for the `'gcx` and `'tcx` parameters of `TyCtxt`. Just to be a touch confusing, we tend to use the name `'tcx` in such contexts. Here is an example: -```rust +```rust,ignore fn not_in_inference<'a, 'tcx>(tcx: TyCtxt<'a, 'tcx, 'tcx>, def_id: DefId) { // ---- ---- // Using the same lifetime here asserts @@ -59,7 +59,7 @@ fn not_in_inference<'a, 'tcx>(tcx: TyCtxt<'a, 'tcx, 'tcx>, def_id: DefId) { In contrast, if we want to code that can be usable during type inference, then you need to declare a distinct `'gcx` and `'tcx` lifetime parameter: -```rust +```rust,ignore fn maybe_in_inference<'a, 'gcx, 'tcx>(tcx: TyCtxt<'a, 'gcx, 'tcx>, def_id: DefId) { // ---- ---- // Using different lifetimes here means that @@ -74,7 +74,7 @@ Rust types are represented using the `Ty<'tcx>` defined in the `ty` module (not to be confused with the `Ty` struct from [the HIR]). This is in fact a simple type alias for a reference with `'tcx` lifetime: -```rust +```rust,ignore pub type Ty<'tcx> = &'tcx TyS<'tcx>; ``` @@ -89,7 +89,7 @@ the rustc arenas (never e.g. on the stack). One common operation on types is to **match** and see what kinds of types they are. This is done by doing `match ty.sty`, sort of like this: -```rust +```rust,ignore fn test_type<'tcx>(ty: Ty<'tcx>) { match ty.sty { ty::TyArray(elem_ty, len) => { ... } @@ -111,7 +111,7 @@ To allocate a new type, you can use the various `mk_` methods defined on the `tcx`. These have names that correpond mostly to the various kinds of type variants. For example: -```rust +```rust,ignore let array_ty = tcx.mk_array(elem_ty, len * 2); ``` @@ -158,7 +158,7 @@ module. Here are a few examples: Although there is no hard and fast rule, the `ty` module tends to be used like so: -```rust +```rust,ignore use ty::{self, Ty, TyCtxt}; ``` diff --git a/src/type-checking.md b/src/type-checking.md index 8148ef627..cca032e30 100644 --- a/src/type-checking.md +++ b/src/type-checking.md @@ -17,9 +17,9 @@ similar conversions for where-clauses and other bits of the function signature. To try and get a sense for the difference, consider this function: -```rust +```rust,ignore struct Foo { } -fn foo(x: Foo, y: self::Foo) { .. } +fn foo(x: Foo, y: self::Foo) { ... } // ^^^ ^^^^^^^^^ ``` diff --git a/src/type-inference.md b/src/type-inference.md index 45f9df18a..ad399d47b 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -21,7 +21,7 @@ signature, such as the `'a` in `for<'a> fn(&'a u32)`. A region is You create and "enter" an inference context by doing something like the following: -```rust +```rust,ignore tcx.infer_ctxt().enter(|infcx| { // Use the inference context `infcx` here. }) @@ -88,7 +88,7 @@ The most basic operations you can perform in the type inferencer is recommended way to add an equality constraint is to use the `at` method, roughly like so: -```rust +```rust,ignore infcx.at(...).eq(t, u); ``` @@ -159,7 +159,9 @@ is to first "generalize" `&'a i32` into a type with a region variable: `&'?b i32`, and then unify `?T` with that (`?T = &'?b i32`). We then relate this new variable with the original bound: - &'?b i32 <: &'a i32 +```txt +&'?b i32 <: &'a i32 +``` This will result in a region constraint (see below) of `'?b: 'a`. @@ -176,12 +178,16 @@ eagerly unifying things, we simply collect constraints as we go, but make (almost) no attempt to solve regions. These constraints have the form of an "outlives" constraint: - 'a: 'b +```txt +'a: 'b +``` Actually the code tends to view them as a subregion relation, but it's the same idea: - 'b <= 'a +```txt +'b <= 'a +``` (There are various other kinds of constriants, such as "verifys"; see the `region_constraints` module for details.) @@ -189,7 +195,9 @@ the `region_constraints` module for details.) There is one case where we do some amount of eager unification. If you have an equality constraint between two regions - 'a = 'b +```txt +'a = 'b +``` we will record that fact in a unification table. You can then use `opportunistic_resolve_var` to convert `'b` to `'a` (or vice diff --git a/src/variance.md b/src/variance.md index 16b4a7518..ea575a595 100644 --- a/src/variance.md +++ b/src/variance.md @@ -54,7 +54,7 @@ constraints will be satisfied. As a simple example, consider: -```rust +```rust,ignore enum Option { Some(A), None } enum OptionalFn { Some(|B|), None } enum OptionalMap { Some(|C| -> C), None } @@ -62,19 +62,23 @@ enum OptionalMap { Some(|C| -> C), None } Here, we will generate the constraints: - 1. V(A) <= + - 2. V(B) <= - - 3. V(C) <= + - 4. V(C) <= - +```txt +1. V(A) <= + +2. V(B) <= - +3. V(C) <= + +4. V(C) <= - +``` These indicate that (1) the variance of A must be at most covariant; (2) the variance of B must be at most contravariant; and (3, 4) the variance of C must be at most covariant *and* contravariant. All of these results are based on a variance lattice defined as follows: - * Top (bivariant) - - + - o Bottom (invariant) +```txt + * Top (bivariant) +- + + o Bottom (invariant) +```txt Based on this lattice, the solution `V(A)=+`, `V(B)=-`, `V(C)=o` is the optimal solution. Note that there is always a naive solution which @@ -85,8 +89,10 @@ is that the variance of a use site may itself be a function of the variance of other type parameters. In full generality, our constraints take the form: - V(X) <= Term - Term := + | - | * | o | V(X) | Term x Term +```txt +V(X) <= Term +Term := + | - | * | o | V(X) | Term x Term +``` Here the notation `V(X)` indicates the variance of a type/region parameter `X` with respect to its defining class. `Term x Term` @@ -101,7 +107,7 @@ represents the "variance transform" as defined in the paper: If I have a struct or enum with where clauses: -```rust +```rust,ignore struct Foo { ... } ``` @@ -170,9 +176,11 @@ another. To see what I mean, consider a trait like so: - trait ConvertTo { - fn convertTo(&self) -> A; - } +```rust +trait ConvertTo { + fn convertTo(&self) -> A; +} +``` Intuitively, If we had one object `O=&ConvertTo` and another `S=&ConvertTo`, then `S <: O` because `String <: Object` @@ -200,20 +208,24 @@ But traits aren't only used with objects. They're also used when deciding whether a given impl satisfies a given trait bound. To set the scene here, imagine I had a function: - fn convertAll>(v: &[T]) { - ... - } +```rust,ignore +fn convertAll>(v: &[T]) { ... } +``` Now imagine that I have an implementation of `ConvertTo` for `Object`: - impl ConvertTo for Object { ... } +```rust,ignore +impl ConvertTo for Object { ... } +``` And I want to call `convertAll` on an array of strings. Suppose further that for whatever reason I specifically supply the value of `String` for the type parameter `T`: - let mut vector = vec!["string", ...]; - convertAll::(vector); +```rust,ignore +let mut vector = vec!["string", ...]; +convertAll::(vector); +``` Is this legal? To put another way, can we apply the `impl` for `Object` to the type `String`? The answer is yes, but to see why @@ -222,11 +234,9 @@ we have to expand out what will happen: - `convertAll` will create a pointer to one of the entries in the vector, which will have type `&String` - It will then call the impl of `convertTo()` that is intended - for use with objects. This has the type: - - fn(self: &Object) -> i32 + for use with objects. This has the type `fn(self: &Object) -> i32`. - It is ok to provide a value for `self` of type `&String` because + It is OK to provide a value for `self` of type `&String` because `&String <: &Object`. OK, so intuitively we want this to be legal, so let's bring this back @@ -238,11 +248,15 @@ Maybe it's helpful to think of a dictionary-passing implementation of type classes. In that case, `convertAll()` takes an implicit parameter representing the impl. In short, we *have* an impl of type: - V_O = ConvertTo for Object +```txt +V_O = ConvertTo for Object +``` and the function prototype expects an impl of type: - V_S = ConvertTo for String +```txt +V_S = ConvertTo for String +``` As with any argument, this is legal if the type of the value given (`V_O`) is a subtype of the type expected (`V_S`). So is `V_O <: V_S`? @@ -250,9 +264,11 @@ The answer will depend on the variance of the various parameters. In this case, because the `Self` parameter is contravariant and `A` is covariant, it means that: - V_O <: V_S iff - i32 <: i32 - String <: Object +```txt +V_O <: V_S iff + i32 <: i32 + String <: Object +``` These conditions are satisfied and so we are happy. @@ -263,7 +279,9 @@ expressions -- must be invariant with respect to all of their inputs. To see why this makes sense, consider what subtyping for a trait reference means: - <: +```txt + <: +``` means that if I know that `T as Trait`, I also know that `U as Trait`. Moreover, if you think of it as dictionary passing style, @@ -279,7 +297,7 @@ Another related reason is that if we didn't make traits with associated types invariant, then projection is no longer a function with a single result. Consider: -``` +```rust,ignore trait Identity { type Out; fn foo(&self); } impl Identity for T { type Out = T; ... } ``` @@ -287,9 +305,11 @@ impl Identity for T { type Out = T; ... } Now if I have `<&'static () as Identity>::Out`, this can be validly derived as `&'a ()` for any `'a`: - <&'a () as Identity> <: <&'static () as Identity> - if &'static () < : &'a () -- Identity is contravariant in Self - if 'static : 'a -- Subtyping rules for relations +```txt +<&'a () as Identity> <: <&'static () as Identity> +if &'static () < : &'a () -- Identity is contravariant in Self +if 'static : 'a -- Subtyping rules for relations +``` This change otoh means that `<'static () as Identity>::Out` is always `&'static ()` (which might then be upcast to `'a ()`, From 8889dcd15d7115bfafb708a9ba61af59d4208dbf Mon Sep 17 00:00:00 2001 From: varkor Date: Fri, 6 Apr 2018 19:35:21 +0100 Subject: [PATCH 185/648] Replace txt with text --- src/appendix-stupid-stats.md | 2 +- src/high-level-overview.md | 2 +- src/incrcomp-debugging.md | 10 +++---- src/method-lookup.md | 2 +- src/mir-passes.md | 4 +-- src/mir-regionck.md | 52 ++++++++++++++++----------------- src/mir.md | 4 +-- src/tests/adding.md | 2 +- src/traits-associated-types.md | 8 ++--- src/traits-canonical-queries.md | 20 ++++++------- src/traits-canonicalization.md | 32 ++++++++++---------- src/traits-goals-and-clauses.md | 12 ++++---- src/traits-lowering-module.md | 2 +- src/traits-lowering-rules.md | 14 ++++----- src/traits-lowering-to-logic.md | 10 +++---- src/type-inference.md | 8 ++--- src/variance.md | 18 ++++++------ 17 files changed, 101 insertions(+), 101 deletions(-) diff --git a/src/appendix-stupid-stats.md b/src/appendix-stupid-stats.md index 0c4de5c7a..9b7d9812b 100644 --- a/src/appendix-stupid-stats.md +++ b/src/appendix-stupid-stats.md @@ -177,7 +177,7 @@ foo.rs` (assuming you have a Rust program called `foo.rs`. You can also pass any command line arguments that you would normally pass to rustc). When you run it you'll see output similar to -```txt +```text In crate: foo, Found 12 uses of `println!`; diff --git a/src/high-level-overview.md b/src/high-level-overview.md index e369db28c..9f3b63a54 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -19,7 +19,7 @@ compilation improves, that may change.) The dependency structure of these crates is roughly a diamond: -```txt +```text rustc_driver / | \ / | \ diff --git a/src/incrcomp-debugging.md b/src/incrcomp-debugging.md index 3ad9f3a49..2488aa320 100644 --- a/src/incrcomp-debugging.md +++ b/src/incrcomp-debugging.md @@ -48,7 +48,7 @@ the graph. You can filter in three ways: To filter, use the `RUST_DEP_GRAPH_FILTER` environment variable, which should look like one of the following: -```txt +```text source_filter // nodes originating from source_filter -> target_filter // nodes that can reach target_filter source_filter -> target_filter // nodes in between source_filter and target_filter @@ -58,14 +58,14 @@ source_filter -> target_filter // nodes in between source_filter and target_filt A node is considered to match a filter if all of those strings appear in its label. So, for example: -```txt +```text RUST_DEP_GRAPH_FILTER='-> TypeckTables' ``` would select the predecessors of all `TypeckTables` nodes. Usually though you want the `TypeckTables` node for some particular fn, so you might write: -```txt +```text RUST_DEP_GRAPH_FILTER='-> TypeckTables & bar' ``` @@ -75,7 +75,7 @@ with `bar` in their name. Perhaps you are finding that when you change `foo` you need to re-type-check `bar`, but you don't think you should have to. In that case, you might do: -```txt +```text RUST_DEP_GRAPH_FILTER='Hir & foo -> TypeckTables & bar' ``` @@ -105,7 +105,7 @@ check of `bar` and you don't think there should be. You dump the dep-graph as described in the previous section and open `dep-graph.txt` to see something like: -```txt +```text Hir(foo) -> Collect(bar) Collect(bar) -> TypeckTables(bar) ``` diff --git a/src/method-lookup.md b/src/method-lookup.md index dbbc62fa8..a3d8dca29 100644 --- a/src/method-lookup.md +++ b/src/method-lookup.md @@ -99,7 +99,7 @@ So, let's continue our example. Imagine that we were calling a method that defines it with `&self` for the type `Rc` as well as a method on the type `Box` that defines `Foo` but with `&mut self`. Then we might have two candidates: -```txt +```text &Rc> from the impl of `Foo` for `Rc` where `U=Box &mut Box<[T; 3]>> from the inherent impl on `Box` where `U=[T; 3]` ``` diff --git a/src/mir-passes.md b/src/mir-passes.md index fd0c6cca1..64e72f06e 100644 --- a/src/mir-passes.md +++ b/src/mir-passes.md @@ -52,7 +52,7 @@ fn main() { The files have names like `rustc.main.000-000.CleanEndRegions.after.mir`. These names have a number of parts: -```txt +```text rustc.main.000-000.CleanEndRegions.after.mir ---- --- --- --------------- ----- either before or after | | | name of the pass @@ -159,7 +159,7 @@ ensuring that the reads have already happened (remember that [queries are memoized](./query.html), so executing a query twice simply loads from a cache the second time): -```txt +```text mir_const(D) --read-by--> mir_const_qualif(D) | ^ stolen-by | diff --git a/src/mir-regionck.md b/src/mir-regionck.md index 949b11d90..4158b7d38 100644 --- a/src/mir-regionck.md +++ b/src/mir-regionck.md @@ -122,7 +122,7 @@ stack, for example). But *how* do we reject it and *why*? When we type-check `main`, and in particular the call `bar(foo)`, we are going to wind up with a subtyping relationship like this one: -```txt +```text fn(&'static u32) <: for<'a> fn(&'a u32) ---------------- ------------------- the type of `foo` the type `bar` expects @@ -137,7 +137,7 @@ regions" -- they represent, basically, "some unknown region". Once we've done that replacement, we have the following relation: -```txt +```text fn(&'static u32) <: fn(&'!1 u32) ``` @@ -151,7 +151,7 @@ subtypes, we check if their arguments have the desired relationship (fn arguments are [contravariant](./appendix-background.html#variance), so we swap the left and right here): -```txt +```text &'!1 u32 <: &'static u32 ``` @@ -201,7 +201,7 @@ Here, the name `'b` is not part of the root universe. Instead, when we "enter" into this `for<'b>` (e.g., by skolemizing it), we will create a child universe of the root, let's call it U1: -```txt +```text U0 (root universe) │ └─ U1 (child universe) @@ -224,7 +224,7 @@ When we enter *this* type, we will again create a new universe, which we'll call `U2`. Its parent will be the root universe, and U1 will be its sibling: -```txt +```text U0 (root universe) │ ├─ U1 (child universe) @@ -263,7 +263,7 @@ children, that inference variable X would have to be in U0. And since X is in U0, it cannot name anything from U1 (or U2). This is perhaps easiest to see by using a kind of generic "logic" example: -```txt +```text exists { forall { ... /* Y is in U1 ... */ } forall { ... /* Z is in U2 ... */ } @@ -296,7 +296,7 @@ does not say region elements **will** appear. In the region inference engine, outlives constraints have the form: -```txt +```text V1: V2 @ P ``` @@ -346,7 +346,7 @@ for universal regions from the fn signature.) Put another way, the "universal regions" check can be considered to be checking constraints like: -```txt +```text {skol(1)}: V1 ``` @@ -358,13 +358,13 @@ made to represent the `!1` region. OK, so far so good. Now let's walk through what would happen with our first example: -```txt +```text fn(&'static u32) <: fn(&'!1 u32) @ P // this point P is not imp't here ``` The region inference engine will create a region element domain like this: -```txt +```text { CFG; end('static); skol(1) } --- ------------ ------- from the universe `!1` | 'static is always in scope @@ -375,20 +375,20 @@ It will always create two universal variables, one representing `'static` and one representing `'!1`. Let's call them Vs and V1. They will have initial values like so: -```txt +```text Vs = { CFG; end('static) } // it is in U0, so can't name anything else V1 = { skol(1) } ``` From the subtyping constraint above, we would have an outlives constraint like -```txt +```text '!1: 'static @ P ``` To process this, we would grow the value of V1 to include all of Vs: -```txt +```text Vs = { CFG; end('static) } V1 = { CFG; end('static), skol(1) } ``` @@ -405,7 +405,7 @@ In this case, `V1` *did* grow too large -- it is not known to outlive What about this subtyping relationship? -```txt +```text for<'a> fn(&'a u32, &'a u32) <: for<'b, 'c> fn(&'b u32, &'c u32) @@ -413,7 +413,7 @@ for<'b, 'c> fn(&'b u32, &'c u32) Here we would skolemize the supertype, as before, yielding: -```txt +```text for<'a> fn(&'a u32, &'a u32) <: fn(&'!1 u32, &'!2 u32) @@ -423,7 +423,7 @@ then we instantiate the variable on the left-hand side with an existential in universe U2, yielding the following (`?n` is a notation for an existential variable): -```txt +```text fn(&'?3 u32, &'?3 u32) <: fn(&'!1 u32, &'!2 u32) @@ -431,14 +431,14 @@ fn(&'!1 u32, &'!2 u32) Then we break this down further: -```txt +```text &'!1 u32 <: &'?3 u32 &'!2 u32 <: &'?3 u32 ``` and even further, yield up our region constraints: -```txt +```text '!1: '?3 '!2: '?3 ``` @@ -465,7 +465,7 @@ common lifetime of our arguments. -nmatsakis) Let's look at one last example. We'll extend the previous one to have a return type: -```txt +```text for<'a> fn(&'a u32, &'a u32) -> &'a u32 <: for<'b, 'c> fn(&'b u32, &'c u32) -> &'b u32 @@ -478,7 +478,7 @@ first one. That is unsound. Let's see how it plays out. First, we skolemize the supertype: -```txt +```text for<'a> fn(&'a u32, &'a u32) -> &'a u32 <: fn(&'!1 u32, &'!2 u32) -> &'!1 u32 @@ -486,7 +486,7 @@ fn(&'!1 u32, &'!2 u32) -> &'!1 u32 Then we instantiate the subtype with existentials (in U2): -```txt +```text fn(&'?3 u32, &'?3 u32) -> &'?3 u32 <: fn(&'!1 u32, &'!2 u32) -> &'!1 u32 @@ -494,7 +494,7 @@ fn(&'!1 u32, &'!2 u32) -> &'!1 u32 And now we create the subtyping relationships: -```txt +```text &'!1 u32 <: &'?3 u32 // arg 1 &'!2 u32 <: &'?3 u32 // arg 2 &'?3 u32 <: &'!1 u32 // return type @@ -503,7 +503,7 @@ And now we create the subtyping relationships: And finally the outlives relationships. Here, let V1, V2, and V3 be the variables we assign to `!1`, `!2`, and `?3` respectively: -```txt +```text V1: V3 V2: V3 V3: V1 @@ -511,7 +511,7 @@ V3: V1 Those variables will have these initial values: -```txt +```text V1 in U1 = {skol(1)} V2 in U2 = {skol(2)} V3 in U2 = {} @@ -520,14 +520,14 @@ V3 in U2 = {} Now because of the `V3: V1` constraint, we have to add `skol(1)` into `V3` (and indeed it is visible from `V3`), so we get: -```txt +```text V3 in U2 = {skol(1)} ``` then we have this constraint `V2: V3`, so we wind up having to enlarge `V2` to include `skol(1)` (which it can also see): -```txt +```text V2 in U2 = {skol(1), skol(2)} ``` diff --git a/src/mir.md b/src/mir.md index 8e45a62bc..da468acf4 100644 --- a/src/mir.md +++ b/src/mir.md @@ -159,7 +159,7 @@ _3 = &mut _1; Assignments in general have the form: -```txt +```text = ``` @@ -169,7 +169,7 @@ value: in this case, the rvalue is a mutable borrow expression, which looks like `&mut `. So we can kind of define a grammar for rvalues like so: -```txt +```text = & (mut)? | + | - diff --git a/src/tests/adding.md b/src/tests/adding.md index f97f4a57e..ab5a0adc1 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -298,7 +298,7 @@ default regex flavor provided by `regex` crate. The corresponding reference file will use the normalized output to test both 32-bit and 64-bit platforms: -```txt +```text ... | = note: source type: fn() ($PTR bits) diff --git a/src/traits-associated-types.md b/src/traits-associated-types.md index bcf6b48ea..e2dd94d5a 100644 --- a/src/traits-associated-types.md +++ b/src/traits-associated-types.md @@ -51,7 +51,7 @@ In our logic, normalization is defined by a predicate impls. For example, the `impl` of `IntoIterator` for `Option` that we saw above would be lowered to a program clause like so: -```txt +```text forall { Normalize( as IntoIterator>::Item -> T) } @@ -101,7 +101,7 @@ consider an associated type projection equal to another type?": We now introduce the `ProjectionEq` predicate to bring those two cases together. The `ProjectionEq` predicate looks like so: -```txt +```text ProjectionEq(::Item = U) ``` @@ -109,7 +109,7 @@ and we will see that it can be proven *either* via normalization or skolemization. As part of lowering an associated type declaration from some trait, we create two program clauses for `ProjectionEq`: -```txt +```text forall { ProjectionEq(::Item = U) :- Normalize(::Item -> U) @@ -130,7 +130,7 @@ with unification. As described in the [type inference](./type-inference.html) section, unification is basically a procedure with a signature like this: -```txt +```text Unify(A, B) = Result<(Subgoals, RegionConstraints), NoSolution> ``` diff --git a/src/traits-canonical-queries.md b/src/traits-canonical-queries.md index 99916267d..3c4bb1bfe 100644 --- a/src/traits-canonical-queries.md +++ b/src/traits-canonical-queries.md @@ -19,13 +19,13 @@ In a traditional Prolog system, when you start a query, the solver will run off and start supplying you with every possible answer it can find. So given something like this: -```txt +```text ?- Vec: AsRef ``` The solver might answer: -```txt +```text Vec: AsRef<[i32]> continue? (y/n) ``` @@ -39,7 +39,7 @@ response with our original query -- Rust's solver gives back a substitution instead). If we were to hit `y`, the solver might then give us another possible answer: -```txt +```text Vec: AsRef> continue? (y/n) ``` @@ -48,14 +48,14 @@ This answer derives from the fact that there is a reflexive impl (`impl AsRef for T`) for `AsRef`. If were to hit `y` again, then we might get back a negative response: -```txt +```text no ``` Naturally, in some cases, there may be no possible answers, and hence the solver will just give me back `no` right away: -```txt +```text ?- Box: Copy no ``` @@ -64,7 +64,7 @@ In some cases, there might be an infinite number of responses. So for example if I gave this query, and I kept hitting `y`, then the solver would never stop giving me back answers: -```txt +```text ?- Vec: Clone Vec: Clone continue? (y/n) @@ -82,13 +82,13 @@ layer of `Box` until we ask it to stop, or it runs out of memory. Another interesting thing is that queries might still have variables in them. For example: -```txt +```text ?- Rc: Clone ``` might produce the answer: -```txt +```text Rc: Clone continue? (y/n) ``` @@ -226,13 +226,13 @@ Let's suppose that the type checker decides to revisit the Borrow`. `?U` is no longer an unbound inference variable; it now has a value, `Vec`. So, if we "refresh" the query with that value, we get: -```txt +```text Vec: Borrow> ``` This time, there is only one impl that applies, the reflexive impl: -```txt +```text impl Borrow for T where T: ?Sized ``` diff --git a/src/traits-canonicalization.md b/src/traits-canonicalization.md index 576ca2055..cce91a388 100644 --- a/src/traits-canonicalization.md +++ b/src/traits-canonicalization.md @@ -44,13 +44,13 @@ and treats them equally, so when canonicalizing, we will *also* replace any [free lifetime](./appendix-background.html#free-vs-bound) with a canonical variable. Therefore, we get the following result: -```txt +```text ?0: Foo<'?1, ?2> ``` Sometimes we write this differently, like so: -```txt +```text for { ?0: Foo<'?1, ?2> } ``` @@ -61,7 +61,7 @@ so `?0` and `?2` are types; the `L` indicates a lifetime varibale, so `CanonicalVarValues` array OV with the "original values" for each canonicalized variable: -```txt +```text [?A, 'static, ?B] ``` @@ -76,13 +76,13 @@ we create a substitution S from the canonical form containing a fresh inference variable (of suitable kind) for each canonical variable. So, for our example query: -```txt +```text for { ?0: Foo<'?1, ?2> } ``` the substitution S might be: -```txt +```text S = [?A, '?B, ?C] ``` @@ -90,7 +90,7 @@ We can then replace the bound canonical variables (`?0`, etc) with these inference variables, yielding the following fully instantiated query: -```txt +```text ?A: Foo<'?B, ?C> ``` @@ -135,20 +135,20 @@ result substitution `var_values`, and some region constraints. To create this, we wind up re-using the substitution S that we created when first instantiating our query. To refresh your memory, we had a query -```txt +```text for { ?0: Foo<'?1, ?2> } ``` for which we made a substutition S: -```txt +```text S = [?A, '?B, ?C] ``` We then did some work which unified some of those variables with other things. If we "refresh" S with the latest results, we get: -```txt +```text S = [Vec, '?D, ?E] ``` @@ -157,7 +157,7 @@ our original query. Note though that they include some new variables (like `?E`). We can make those go away by canonicalizing again! We don't just canonicalize S, though, we canonicalize the whole query response QR: -```txt +```text QR = { certainty: Proven, // or whatever var_values: [Vec, '?D, ?E] // this is S @@ -170,7 +170,7 @@ QR = { The result would be as follows: -```txt +```text Canonical(QR) = for { certainty: Proven, var_values: [Vec, '?1, ?2] @@ -194,19 +194,19 @@ In the previous section we produced a canonical query result. We now have to apply that result in our original context. If you recall, way back in the beginning, we were trying to prove this query: -```txt +```text ?A: Foo<'static, ?B> ``` We canonicalized that into this: -```txt +```text for { ?0: Foo<'?1, ?2> } ``` and now we got back a canonical response: -```txt +```text for { certainty: Proven, var_values: [Vec, '?1, ?2] @@ -221,7 +221,7 @@ the result with a fresh inference variable, (b) unify the values in the result with the original values, and then (c) record the region constraints for later. Doing step (a) would yield a result of -```txt +```text { certainty: Proven, var_values: [Vec, '?D, ?C] @@ -233,7 +233,7 @@ constraints for later. Doing step (a) would yield a result of Step (b) would then unify: -```txt +```text ?A with Vec 'static with '?D ?B with ?C diff --git a/src/traits-goals-and-clauses.md b/src/traits-goals-and-clauses.md index 97532195e..5e8ee1469 100644 --- a/src/traits-goals-and-clauses.md +++ b/src/traits-goals-and-clauses.md @@ -12,7 +12,7 @@ a few new superpowers. In Rust's solver, **goals** and **clauses** have the following forms (note that the two definitions reference one another): -```txt +```text Goal = DomainGoal // defined in the section below | Goal && Goal | Goal || Goal @@ -49,7 +49,7 @@ To define the set of *domain goals* in our system, we need to first introduce a few simple formulations. A **trait reference** consists of the name of a trait along with a suitable set of inputs P0..Pn: -```txt +```text TraitRef = P0: TraitName ``` @@ -63,13 +63,13 @@ T>`), that are not part of a trait reference. A **projection** consists of an associated item reference along with its inputs P0..Pm: -```txt +```text Projection = >::AssocItem ``` Given that, we can define a `DomainGoal` as follows: -```txt +```text DomainGoal = Implemented(TraitRef) | ProjectionEq(Projection = Type) | Normalize(Projection -> Type) @@ -112,7 +112,7 @@ DomainGoal = Implemented(TraitRef) Most goals in our system are "inductive". In an inductive goal, circular reasoning is disallowed. Consider this example clause: -```txt +```text Implemented(Foo: Bar) :- Implemented(Foo: Bar). ``` @@ -140,7 +140,7 @@ struct Foo { The default rules for auto traits say that `Foo` is `Send` if the types of its fields are `Send`. Therefore, we would have a rule like -```txt +```text Implemented(Foo: Send) :- Implemented(Option>: Send). ``` diff --git a/src/traits-lowering-module.md b/src/traits-lowering-module.md index fa068f86a..cb3b4a5f7 100644 --- a/src/traits-lowering-module.md +++ b/src/traits-lowering-module.md @@ -49,7 +49,7 @@ standard [ui test] mechanisms to check them. In this case, there is a need only be a prefix of the error), but [the stderr file] contains the full details: -```txt +```text error: Implemented(T: Foo) :- ProjectionEq(::Item == i32), TypeOutlives(T \ : 'static), Implemented(T: std::iter::Iterator), Implemented(T: std::marker::Sized). --> $DIR/lower_impl.rs:15:1 diff --git a/src/traits-lowering-rules.md b/src/traits-lowering-rules.md index 1a9717f9f..f7433221e 100644 --- a/src/traits-lowering-rules.md +++ b/src/traits-lowering-rules.md @@ -87,7 +87,7 @@ relationships between different kinds of domain goals. The first such rule from the trait header creates the mapping between the `FromEnv` and `Implemented` predicates: -```txt +```text // Rule Implemented-From-Env forall { Implemented(Self: Trait) :- FromEnv(Self: Trait) @@ -103,7 +103,7 @@ The next few clauses have to do with implied bounds (see also [RFC 2089]: https://rust-lang.github.io/rfcs/2089-implied-bounds.html -```txt +```text // Rule Implied-Bound-From-Trait // // For each where clause WC: @@ -130,7 +130,7 @@ clauses** but also things that follow from them. The next rule is related; it defines what it means for a trait reference to be **well-formed**: -```txt +```text // Rule WellFormed-TraitRef // // For each where clause WC: @@ -197,7 +197,7 @@ the rules by which `ProjectionEq` can succeed; these two clauses are discussed in detail in the [section on associated types](./traits-associated-types.html), but reproduced here for reference: -```txt +```text // Rule ProjectionEq-Normalize // // ProjectionEq can succeed by normalizing: @@ -227,7 +227,7 @@ the `Bounds` declared on the associated type must be proven to hold to show that the impl is well-formed, and hence we can rely on them elsewhere. -```txt +```text // XXX how exactly should we set this up? Have to be careful; // presumably this has to be a kind of `FromEnv` setup. ``` @@ -253,7 +253,7 @@ where WC Let `TraitRef` be the trait reference `A0: Trait`. Then we will create the following rules: -```txt +```text // Rule Implemented-From-Impl forall { Implemented(TraitRef) :- WC @@ -278,7 +278,7 @@ where WC We produce the following rule: -```txt +```text // Rule Normalize-From-Impl forall { forall { diff --git a/src/traits-lowering-to-logic.md b/src/traits-lowering-to-logic.md index f36794d2c..54b3473d4 100644 --- a/src/traits-lowering-to-logic.md +++ b/src/traits-lowering-to-logic.md @@ -30,7 +30,7 @@ impl Clone for Vec where T: Clone { } We could map these declarations to some Horn clauses, written in a Prolog-like notation, as follows: -```txt +```text Clone(usize). Clone(Vec) :- Clone(?T). @@ -70,7 +70,7 @@ impl> Eq> for Vec { } That could be mapped as follows: -```txt +```text Eq(usize, usize). Eq(Vec, Vec) :- Eq(?T, ?U). ``` @@ -105,7 +105,7 @@ If we wanted, we could write a Prolog predicate that defines the conditions under which `bar()` can be called. We'll say that those conditions are called being "well-formed": -```txt +```text barWellFormed(?U) :- Eq(?U, ?U). ``` @@ -113,7 +113,7 @@ Then we can say that `foo()` type-checks if the reference to `bar::` (that is, `bar()` applied to the type `usize`) is well-formed: -```txt +```text fooTypeChecks :- barWellFormed(usize). ``` @@ -144,7 +144,7 @@ To type-check the body of `foo`, we need to be able to hold the type type-safe *for all types `T`*, not just for some specific type. We might express this like so: -```txt +```text fooTypeChecks :- // for all types T... forall { diff --git a/src/type-inference.md b/src/type-inference.md index ad399d47b..152bbd9da 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -159,7 +159,7 @@ is to first "generalize" `&'a i32` into a type with a region variable: `&'?b i32`, and then unify `?T` with that (`?T = &'?b i32`). We then relate this new variable with the original bound: -```txt +```text &'?b i32 <: &'a i32 ``` @@ -178,14 +178,14 @@ eagerly unifying things, we simply collect constraints as we go, but make (almost) no attempt to solve regions. These constraints have the form of an "outlives" constraint: -```txt +```text 'a: 'b ``` Actually the code tends to view them as a subregion relation, but it's the same idea: -```txt +```text 'b <= 'a ``` @@ -195,7 +195,7 @@ the `region_constraints` module for details.) There is one case where we do some amount of eager unification. If you have an equality constraint between two regions -```txt +```text 'a = 'b ``` diff --git a/src/variance.md b/src/variance.md index ea575a595..5fe84636f 100644 --- a/src/variance.md +++ b/src/variance.md @@ -62,7 +62,7 @@ enum OptionalMap { Some(|C| -> C), None } Here, we will generate the constraints: -```txt +```text 1. V(A) <= + 2. V(B) <= - 3. V(C) <= + @@ -74,11 +74,11 @@ These indicate that (1) the variance of A must be at most covariant; variance of C must be at most covariant *and* contravariant. All of these results are based on a variance lattice defined as follows: -```txt +```text * Top (bivariant) - + o Bottom (invariant) -```txt +```text Based on this lattice, the solution `V(A)=+`, `V(B)=-`, `V(C)=o` is the optimal solution. Note that there is always a naive solution which @@ -89,7 +89,7 @@ is that the variance of a use site may itself be a function of the variance of other type parameters. In full generality, our constraints take the form: -```txt +```text V(X) <= Term Term := + | - | * | o | V(X) | Term x Term ``` @@ -248,13 +248,13 @@ Maybe it's helpful to think of a dictionary-passing implementation of type classes. In that case, `convertAll()` takes an implicit parameter representing the impl. In short, we *have* an impl of type: -```txt +```text V_O = ConvertTo for Object ``` and the function prototype expects an impl of type: -```txt +```text V_S = ConvertTo for String ``` @@ -264,7 +264,7 @@ The answer will depend on the variance of the various parameters. In this case, because the `Self` parameter is contravariant and `A` is covariant, it means that: -```txt +```text V_O <: V_S iff i32 <: i32 String <: Object @@ -279,7 +279,7 @@ expressions -- must be invariant with respect to all of their inputs. To see why this makes sense, consider what subtyping for a trait reference means: -```txt +```text <: ``` @@ -305,7 +305,7 @@ impl Identity for T { type Out = T; ... } Now if I have `<&'static () as Identity>::Out`, this can be validly derived as `&'a ()` for any `'a`: -```txt +```text <&'a () as Identity> <: <&'static () as Identity> if &'static () < : &'a () -- Identity is contravariant in Self if 'static : 'a -- Subtyping rules for relations From 79eaa63c540f8872530ab01dfaf7166fc99703cf Mon Sep 17 00:00:00 2001 From: varkor Date: Fri, 6 Apr 2018 19:36:48 +0100 Subject: [PATCH 186/648] Fix spacing --- src/traits-canonicalization.md | 4 ++-- src/variance.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src/traits-canonicalization.md b/src/traits-canonicalization.md index cce91a388..fa39151d7 100644 --- a/src/traits-canonicalization.md +++ b/src/traits-canonicalization.md @@ -163,8 +163,8 @@ QR = { var_values: [Vec, '?D, ?E] // this is S region_constraints: [?E: '?D], // from the impl value: (), // for our purposes, just (), but - // in some cases this might have - // a type or other info + // in some cases this might have + // a type or other info } ``` diff --git a/src/variance.md b/src/variance.md index 5fe84636f..527c2745c 100644 --- a/src/variance.md +++ b/src/variance.md @@ -75,10 +75,10 @@ variance of C must be at most covariant *and* contravariant. All of these results are based on a variance lattice defined as follows: ```text - * Top (bivariant) + * Top (bivariant) - + - o Bottom (invariant) -```text + o Bottom (invariant) +``` Based on this lattice, the solution `V(A)=+`, `V(B)=-`, `V(C)=o` is the optimal solution. Note that there is always a naive solution which From 6e4286d387509e698cec19a4327b199a32b128cf Mon Sep 17 00:00:00 2001 From: varkor Date: Fri, 6 Apr 2018 19:39:25 +0100 Subject: [PATCH 187/648] Add mdbook test to Travis --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index 45ba4a890..d7f8f8f42 100644 --- a/.travis.yml +++ b/.travis.yml @@ -9,6 +9,7 @@ install: - bash ci/install.sh script: - mdbook build +- mdbook test notifications: email: on_success: never From 46864ca9714707f18e658afba9d2ea62afbe2374 Mon Sep 17 00:00:00 2001 From: varkor Date: Fri, 6 Apr 2018 20:04:59 +0100 Subject: [PATCH 188/648] Fix unstable Rust code block issue --- src/traits-lowering-module.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/traits-lowering-module.md b/src/traits-lowering-module.md index cb3b4a5f7..fbf1d6425 100644 --- a/src/traits-lowering-module.md +++ b/src/traits-lowering-module.md @@ -26,7 +26,7 @@ Unit tests are located in [`src/test/ui/chalkify`][chalkify]. A good example test is [the `lower_impl` test][lower_impl]. At the time of this writing, it looked like this: -```rust +```rust,ignore #![feature(rustc_attrs)] trait Foo { } From ad77d418fabaa7bb6f1bc5bc3bba525304fdbfb9 Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Fri, 6 Apr 2018 20:58:37 -0400 Subject: [PATCH 189/648] update nightly documention for issue#100-b --- src/appendix-stupid-stats.md | 2 +- src/const-eval.md | 2 +- src/method-lookup.md | 4 ++-- src/miri.md | 2 +- src/type-checking.md | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/appendix-stupid-stats.md b/src/appendix-stupid-stats.md index 9b7d9812b..842a2a328 100644 --- a/src/appendix-stupid-stats.md +++ b/src/appendix-stupid-stats.md @@ -113,7 +113,7 @@ between phases. `CompilerCalls` is a trait that you implement in your tool. It contains a fairly ad-hoc set of methods to hook in to the process of processing command line arguments and driving the compiler. For details, see the comments in -[librustc_driver/lib.rs](https://github.com/rust-lang/rust/tree/master/src/librustc_driver/lib.rs). +[librustc_driver/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/index.html). I'll summarise the methods here. `early_callback` and `late_callback` let you call arbitrary code at different diff --git a/src/const-eval.md b/src/const-eval.md index 4a05255c6..70c946f17 100644 --- a/src/const-eval.md +++ b/src/const-eval.md @@ -35,4 +35,4 @@ integer or fat pointer, it will directly yield the value (via `Value::ByVal` or memory allocation (via `Value::ByRef`). This means that the `const_eval` function cannot be used to create miri-pointers to the evaluated constant or static. If you need that, you need to directly work with the functions in -[src/librustc_mir/interpret/const_eval.rs](https://github.com/rust-lang/rust/blob/master/src/librustc_mir/interpret/const_eval.rs). +[src/librustc_mir/interpret/const_eval.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/interpret/const_eval/). diff --git a/src/method-lookup.md b/src/method-lookup.md index a3d8dca29..5aafb6abf 100644 --- a/src/method-lookup.md +++ b/src/method-lookup.md @@ -38,8 +38,8 @@ cacheable across method-call sites. Therefore, it does not include inference variables or other information. [UFCS]: https://github.com/rust-lang/rfcs/blob/master/text/0132-ufcs.md -[probe]: https://github.com/rust-lang/rust/blob/master/src/librustc_typeck/check/method/probe.rs -[confirm]: https://github.com/rust-lang/rust/blob/master/src/librustc_typeck/check/method/confirm.rs +[probe]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_typeck/check/method/probe/ +[confirm]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_typeck/check/method/confirm/ ## The Probe phase diff --git a/src/miri.md b/src/miri.md index fb6675f50..be9587408 100644 --- a/src/miri.md +++ b/src/miri.md @@ -112,7 +112,7 @@ to a pointer to `b`. Although the main entry point to constant evaluation is the `tcx.const_eval` query, there are additional functions in -[librustc_mir/interpret/const_eval.rs](https://github.com/rust-lang/rust/blob/master/src/librustc_mir/interpret/const_eval.rs) +[librustc_mir/interpret/const_eval.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/interpret/const_eval/) that allow accessing the fields of a `Value` (`ByRef` or otherwise). You should never have to access an `Allocation` directly except for translating it to the compilation target (at the moment just LLVM). diff --git a/src/type-checking.md b/src/type-checking.md index cca032e30..cb6d346e4 100644 --- a/src/type-checking.md +++ b/src/type-checking.md @@ -39,6 +39,6 @@ type *checking*). For more details, see the [`collect`][collect] module. [queries]: query.html -[collect]: https://github.com/rust-lang/rust/blob/master/src/librustc_typeck/collect.rs +[collect]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_typeck/collect/ **TODO**: actually talk about type checking... From df6b73fb99b5aaef51130de9d70a18d60f566ca4 Mon Sep 17 00:00:00 2001 From: QuietMisdreavus Date: Tue, 10 Apr 2018 19:45:46 -0500 Subject: [PATCH 190/648] talk about `only-X` in compiletest --- src/tests/adding.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/tests/adding.md b/src/tests/adding.md index ab5a0adc1..1445981eb 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -122,6 +122,8 @@ be compiled or run. * `ignore-X` where `X` is a target detail or stage will ignore the test accordingly (see below) +* `only-X` is like `ignore-X`, but will *only* run the test on that + target or stage * `ignore-pretty` will not compile the pretty-printed test (this is done to test the pretty-printer, but might not always work) * `ignore-test` always ignores the test From 27a32adb7215f9248fef5d56927c02ecac712f6f Mon Sep 17 00:00:00 2001 From: Samuel Wilson Date: Sat, 7 Apr 2018 00:01:51 +1200 Subject: [PATCH 191/648] Add definition of a fat pointer Definition referenced from Programming Rust by Jim Blandy & Jason Orendorff, published by O'Reilly. Page 214: References to Slices and Trait Objects. --- src/appendix-glossary.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/appendix-glossary.md b/src/appendix-glossary.md index 1d82686ba..23de529f2 100644 --- a/src/appendix-glossary.md +++ b/src/appendix-glossary.md @@ -16,6 +16,7 @@ cx | we tend to use "cx" as an abbrevation for context. Se DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](incremental-compilation.html)) data-flow analysis | a static analysis that figures out what properties are true at each point in the control-flow of a program; see [the background chapter for more](./appendix-background.html#dataflow) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. +Fat pointer | a two word value carrying the address of some value, along with some further information necessary to put the value to use. For example: a reference to a slice is a fat pointer, carrying the starting address of the slice and its length. free variable | a "free variable" is one that is not bound within an expression or term; see [the background chapter for more](./appendix-background.html#free-vs-bound) 'gcx | the lifetime of the global arena ([see more](ty.html)) generics | the set of generic type parameters defined on a type or item From a87b323307a6d7ad0ba6c9f9b69280766a8a4637 Mon Sep 17 00:00:00 2001 From: Samuel Wilson Date: Sat, 7 Apr 2018 00:01:51 +1200 Subject: [PATCH 192/648] Add definition of a fat pointer, double pointer, and wide pointer Definition referenced from Programming Rust by Jim Blandy & Jason Orendorff, published by O'Reilly. Page 214: References to Slices and Trait Objects. Double pointer and wide pointer both refer to fat pointer for detail. --- src/appendix-glossary.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/appendix-glossary.md b/src/appendix-glossary.md index 23de529f2..a37f1b22f 100644 --- a/src/appendix-glossary.md +++ b/src/appendix-glossary.md @@ -16,7 +16,8 @@ cx | we tend to use "cx" as an abbrevation for context. Se DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](incremental-compilation.html)) data-flow analysis | a static analysis that figures out what properties are true at each point in the control-flow of a program; see [the background chapter for more](./appendix-background.html#dataflow) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. -Fat pointer | a two word value carrying the address of some value, along with some further information necessary to put the value to use. For example: a reference to a slice is a fat pointer, carrying the starting address of the slice and its length. +Double pointer | a pointer with additional metadata. See "fat pointer" for more. +Fat pointer | a two word value carrying the address of some value, along with some further information necessary to put the value to use. Rust includes two kinds of "fat pointers": references to slices, and trait objects. A reference to a slice carries the starting address of the slice and its length. A trait object carries a value's address and a pointer to the trait's implementation appropriate to that value. "Fat pointers" are also known as "wide pointers", and "double pointers". free variable | a "free variable" is one that is not bound within an expression or term; see [the background chapter for more](./appendix-background.html#free-vs-bound) 'gcx | the lifetime of the global arena ([see more](ty.html)) generics | the set of generic type parameters defined on a type or item @@ -61,6 +62,7 @@ trait reference | a trait and values for its type parameters ([see more ty | the internal representation of a type ([see more](ty.html)). UFCS | Universal Function Call Syntax. An unambiguous syntax for calling a method ([see more](type-checking.html)). variance | variance determines how changes to a generic type/lifetime parameter affect subtyping; for example, if `T` is a subtype of `U`, then `Vec` is a subtype `Vec` because `Vec` is *covariant* in its generic parameter. See [the background chapter](./appendix-background.html#variance) for a more general explanation. See the [variance chapter](./variance.html) for an explanation of how type checking handles variance. +Wide pointer | a pointer with additional metadata. See "fat pointer" for more. [LLVM]: https://llvm.org/ [lto]: https://llvm.org/docs/LinkTimeOptimization.html From 81116ab9772ceca8f6b7a42c31b757193df476a3 Mon Sep 17 00:00:00 2001 From: cg-cnu Date: Mon, 9 Apr 2018 21:57:57 +0530 Subject: [PATCH 193/648] Add: Added the instructions for debugging from rust-forge --- src/SUMMARY.md | 1 + src/debugging.md | 296 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 297 insertions(+) create mode 100644 src/debugging.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 3460d949d..bf770dbb8 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -52,6 +52,7 @@ - [miri const evaluator](./miri.md) - [Parameter Environments](./param_env.md) - [Generating LLVM IR](./trans.md) +- [Debugging](./debugging.md) --- diff --git a/src/debugging.md b/src/debugging.md new file mode 100644 index 000000000..622728d0d --- /dev/null +++ b/src/debugging.md @@ -0,0 +1,296 @@ +--- +layout: default +title: Debugging the Compiler +--- + +# Debugging the compiler +[debugging]: #debugging + +Here are a few tips to debug the compiler: + +## Getting a backtrace +[getting-a-backtrace]: #getting-a-backtrace + +When you have an ICE (panic in the compiler), you can set +`RUST_BACKTRACE=1` to get the stack trace of the `panic!` like in +normal Rust programs. IIRC backtraces **don't work** on Mac and on MinGW, +sorry. If you have trouble or the backtraces are full of `unknown`, +you might want to find some way to use Linux or MSVC on Windows. + +In the default configuration, you don't have line numbers enabled, so the backtrace looks like this: + ``` +stack backtrace: + 0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace + 1: std::sys_common::backtrace::_print + 2: std::panicking::default_hook::{{closure}} + 3: std::panicking::default_hook + 4: std::panicking::rust_panic_with_hook + 5: std::panicking::begin_panic + 6: rustc_typeck::check::cast::>::pointer_kind + (~~~~ LINES REMOVED BY ME FOR BREVITY ~~~~) + 32: rustc_typeck::check_crate + 33: >::with + 34: >::with + 35: rustc::ty::context::TyCtxt::create_and_enter + 36: rustc_driver::driver::compile_input + 37: rustc_driver::run_compiler + ``` + +If you want line numbers for the stack trace, you can enable `debuginfo-lines=true` or `debuginfo=true` in your config.toml and rebuild the compiler. Then the backtrace will look like this: +``` +stack backtrace: + (~~~~ LINES REMOVED BY ME FOR BREVITY ~~~~) + 6: rustc_typeck::check::cast::>::pointer_kind + at /home/user/rust/src/librustc_typeck/check/cast.rs:110 + 7: rustc_typeck::check::cast::CastCheck::check + at /home/user/rust/src/librustc_typeck/check/cast.rs:572 + at /home/user/rust/src/librustc_typeck/check/cast.rs:460 + at /home/user/rust/src/librustc_typeck/check/cast.rs:370 + (~~~~ LINES REMOVED BY ME FOR BREVITY ~~~~) + 33: rustc_driver::driver::compile_input + at /home/user/rust/src/librustc_driver/driver.rs:1010 + at /home/user/rust/src/librustc_driver/driver.rs:212 + 34: rustc_driver::run_compiler + at /home/user/rust/src/librustc_driver/lib.rs:253 +``` + +## Getting a backtrace for errors +[getting-a-backtrace-for-errors]: #getting-a-backtrace-for-errors + +If you want to get a backtrace to the point where the compiler emits +an error message, you can pass the `-Z treat-err-as-bug`, which +will make the compiler panic on the first error it sees. + +This can also help when debugging `delay_span_bug` calls - it will make +the first `delay_span_bug` call panic, which will give you a useful backtrace. + +For example: +``` +$ cat error.rs +fn main() { + 1 + (); +} +$ ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc error.rs +error[E0277]: the trait bound `{integer}: std::ops::Add<()>` is not satisfied + --> error.rs:2:7 + | +2 | 1 + (); + | ^ no implementation for `{integer} + ()` + | + = help: the trait `std::ops::Add<()>` is not implemented for `{integer}` + +error: aborting due to previous error + +$ # Now, where does the error above come from? +$ RUST_BACKTRACE=1 \ + ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc \ + error.rs \ + -Z treat-err-as-bug +error[E0277]: the trait bound `{integer}: std::ops::Add<()>` is not satisfied + --> error.rs:2:7 + | +2 | 1 + (); + | ^ no implementation for `{integer} + ()` + | + = help: the trait `std::ops::Add<()>` is not implemented for `{integer}` + +error: internal compiler error: unexpected panic + +note: the compiler unexpectedly panicked. this is a bug. + +note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports + +note: rustc 1.24.0-dev running on x86_64-unknown-linux-gnu + +note: run with `RUST_BACKTRACE=1` for a backtrace + +thread 'rustc' panicked at 'encountered error with `-Z treat_err_as_bug', /home/user/rust/src/librustc_errors/lib.rs:411:12 +note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. +stack backtrace: + (~~~ IRRELEVANT PART OF BACKTRACE REMOVED BY ME ~~~) + 7: rustc::traits::error_reporting::>::report_selection_error + at /home/user/rust/src/librustc/traits/error_reporting.rs:823 + 8: rustc::traits::error_reporting::>::report_fulfillment_errors + at /home/user/rust/src/librustc/traits/error_reporting.rs:160 + at /home/user/rust/src/librustc/traits/error_reporting.rs:112 + 9: rustc_typeck::check::FnCtxt::select_obligations_where_possible + at /home/user/rust/src/librustc_typeck/check/mod.rs:2192 + (~~~ IRRELEVANT PART OF BACKTRACE REMOVED BY ME ~~~) + 36: rustc_driver::run_compiler + at /home/user/rust/src/librustc_driver/lib.rs:253 +$ # Cool, now I have a backtrace for the error +``` + +## Getting logging output +[getting-logging-output]: #getting-logging-output + +The compiler has a lot of `debug!` calls, which print out logging information +at many points. These are very useful to at least narrow down the location of +a bug if not to find it entirely, or just to orient yourself to why a compiler +is doing a particular thing. + +To see the logs, you need to set the `RUST_LOG` environment variable to +your log filter, e.g. to get the logs for a specific module, you can run the +compiler as `RUST_LOG=module::path rustc my-file.rs`. The Rust logs are +powered by [env-logger], and you can look at the docs linked there to see +the full `RUST_LOG` syntax. All `debug!` output will then appear in +standard error. + +Note that unless you use a very strict filter, the logger will emit a *lot* +of output - so it's typically a good idea to pipe standard error to a file +and look at the log output with a text editor. + +So to put it together. +``` +# This puts the output of all debug calls in `librustc/traits` into +# standard error, which might fill your console backscroll. +$ RUST_LOG=rustc::traits rustc +local my-file.rs + +# This puts the output of all debug calls in `librustc/traits` in +# `traits-log`, so you can then see it with a text editor. +$ RUST_LOG=rustc::traits rustc +local my-file.rs 2>traits-log + +# Not recommended. This will show the output of all `debug!` calls +# in the Rust compiler, and there are a *lot* of them, so it will be +# hard to find anything. +$ RUST_LOG=debug rustc +local my-file.rs 2>all-log + +# This will show the output of all `info!` calls in `rustc_trans`. +# +# There's an `info!` statement in `trans_instance` that outputs +# every function that is translated. This is useful to find out +# which function triggers an LLVM assertion, and this is an `info!` +# log rather than a `debug!` log so it will work on the official +# compilers. +$ RUST_LOG=rustc_trans=info rustc +local my-file.rs +``` + +While calls to `info!` are included in every build of the compiler, +calls to `debug!` are only included in the program if the +`debug-assertions=yes` is turned on in config.toml (it is +turned off by default), so if you don't see `DEBUG` logs, especially +if you run the compiler with `RUST_LOG=rustc rustc some.rs` and only see +`INFO` logs, make sure that `debug-assertions=yes` is turned on in your +config.toml. + +I also think that in some cases just setting it will not trigger a rebuild, +so if you changed it and you already have a compiler built, you might +want to call `x.py clean` to force one. + +### Logging etiquette + +Because calls to `debug!` are removed by default, in most cases, don't worry +about adding "unnecessary" calls to `debug!` and leaving them in in code +you commit - they won't slow +down the performance of what we ship, and if they helped you pinning down +a bug, they will probably help someone else with a different one. + +However, there are still a few concerns that you might care about: + +### Expensive operations in logs + +A note of caution: the expressions *within* the `debug!` call are run +whenever RUST_LOG is set, even if the filter would exclude the log. This means that if in the module `rustc::foo` you have a statement + +```Rust +debug!("{:?}", random_operation(tcx)); +``` + +Then if someone runs a debug `rustc` with `RUST_LOG=rustc::bar`, then `random_operation()` will still run - even while it's output will never be needed! + +This means that you should not put anything too expensive or likely +to crash there - that would annoy anyone who wants to use logging for their own module. Note that if `RUST_LOG` is unset (the default), then the code will not run - this means that if your logging code panics, then no-one will know it until someone tries to use logging to find *another* bug. + +If you *need* to do an expensive operation in a log, be aware that while log expressions are *evaluated* even if logging is not enabled in your module, they are not *formatted* unless it *is*. This means you can put your expensive/crashy operations inside an `fmt::Debug` impl, and they will not be run unless your log is enabled: + +```Rust +use std::fmt; + +struct ExpensiveOperationContainer<'a, 'gcx, 'tcx> + where 'tcx: 'gcx, 'a: 'tcx +{ + tcx: TyCtxt<'a, 'gcx, 'tcx> +} + +impl<'a, 'gcx, 'tcx> fmt::Debug for ExpensiveOperationContainer<'a, 'gcx, 'tcx> { + fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result { + let value = random_operation(tcx); + fmt::Debug::fmt(&value, fmt) + } +} + +debug!("{:?}", ExpensiveOperationContainer { tcx }); +``` + +## Formatting Graphviz output (.dot files) +[formatting-graphviz-output]: #formatting-graphviz-output + +Some compiler options for debugging specific features yield graphviz graphs - e.g. +the `#[rustc_mir(borrowck_graphviz_postflow="suffix.dot")]` attribute +dumps various borrow-checker dataflow graphs. + +These all produce `.dot` files. To view these files, install graphviz (e.g. +`apt-get install graphviz`) and then run the following commands: + +``` +$ dot -T pdf maybe_init_suffix.dot > maybe_init_suffix.pdf +$ firefox maybe_init_suffix.pdf # Or your favorite pdf viewer +``` + +## Debugging LLVM +[debugging-llvm]: #debugging-llvm + +LLVM is a big project on its own that probably needs to have its own debugging +document (not that I could find one). But here are some tips that are important +in a rustc context: + +The official compilers (including nightlies) have LLVM assertions disabled, +which means that LLVM assertion failures can show up as compiler crashes (not +ICEs but "real" crashes) and other sorts of weird behavior. If you are +encountering these, it is a good idea to try using a compiler with LLVM +assertions enabled - either an "alt" nightly or a compiler you build yourself +by setting `[llvm] assertions=true` in your config.toml - and +see whether anything turns up. + +The rustc build process builds the LLVM tools into `./build//llvm/bin`. They can be called directly. + +The default rustc compilation pipeline has multiple codegen units, which is hard +to replicate manually and means that LLVM is called multiple times in parallel. +If you can get away with it (i.e. if it doesn't make your bug disappear), +passing `-C codegen-units=1` to rustc will make debugging easier. + +If you want to play with the optimization pipeline, you can use the `opt` from +there on the IR rustc emits with `--emit=llvm-ir`. Note +that rustc emits different IR depending on whether `-O` is enabled, even without +LLVM's optimizations, so if you want to play with the IR rustc emits, you should: +``` +$ rustc +local my-file.rs --emit=llvm-ir -O -C no-prepopulate-passes \ + -C codegen-units=1 +$ OPT=./build/$TRIPLE/llvm/bin/opt +$ $OPT -S -O2 < my-file.ll > my +``` + +If you just want to get the LLVM IR during the LLVM pipeline, to e.g. see which +IR causes an optimization-time assertion to fail, or to see when +LLVM performs a particular optimization, you can pass the rustc flag +`-C llvm-args=-print-after-all`, and possibly add +`-C llvm-args='-filter-print-funcs=EXACT_FUNCTION_NAME` (e.g. +`-C llvm-args='-filter-print-funcs=_ZN11collections3str21_$LT$impl$u20$str$GT$7replace17hbe10ea2e7c809b0bE'`). + +That produces a lot of output into standard error, so you'll want to pipe +that to some file. Also, if you are using neither `-filter-print-funcs` nor +`-C codegen-units=1`, then, because the multiple codegen units run in parallel, +the printouts will mix together and you won't be able to read anything. + +If you want just the IR for a specific function (say, you want to see +why it causes an assertion or doesn't optimize correctly), you can use +`llvm-extract`, e.g. +``` +$ ./build/$TRIPLE/llvm/bin/llvm-extract \ + -func='_ZN11collections3str21_$LT$impl$u20$str$GT$7replace17hbe10ea2e7c809b0bE' \ + -S \ + < unextracted.ll \ + > extracted.ll +``` + +[env-logger]: https://docs.rs/env_logger/0.4.3/env_logger/ From 5653c13077706d8c88658bb261ff10056348ebf6 Mon Sep 17 00:00:00 2001 From: cg-cnu Date: Mon, 9 Apr 2018 22:04:53 +0530 Subject: [PATCH 194/648] refactor: Change the name from debugging to compiler-debuggin.md --- src/SUMMARY.md | 2 +- src/{debugging.md => compiler-debugging.md} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename src/{debugging.md => compiler-debugging.md} (100%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index bf770dbb8..1b9670261 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -52,7 +52,7 @@ - [miri const evaluator](./miri.md) - [Parameter Environments](./param_env.md) - [Generating LLVM IR](./trans.md) -- [Debugging](./debugging.md) +- [Debugging the Compiler](./compiler-debugging.md) --- diff --git a/src/debugging.md b/src/compiler-debugging.md similarity index 100% rename from src/debugging.md rename to src/compiler-debugging.md From 8e02b3734980185e159ebd2983f5fa6e4964ec31 Mon Sep 17 00:00:00 2001 From: cg-cnu Date: Mon, 9 Apr 2018 22:27:43 +0530 Subject: [PATCH 195/648] refactor: Fixed all the lines exceeding more than 80 characters --- src/compiler-debugging.md | 58 +++++++++++++++++++++++++++------------ 1 file changed, 41 insertions(+), 17 deletions(-) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index 622728d0d..41b50b04a 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -17,7 +17,9 @@ normal Rust programs. IIRC backtraces **don't work** on Mac and on MinGW, sorry. If you have trouble or the backtraces are full of `unknown`, you might want to find some way to use Linux or MSVC on Windows. -In the default configuration, you don't have line numbers enabled, so the backtrace looks like this: +In the default configuration, you don't have line numbers enabled, so the +backtrace looks like this: + ``` stack backtrace: 0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace @@ -26,7 +28,8 @@ stack backtrace: 3: std::panicking::default_hook 4: std::panicking::rust_panic_with_hook 5: std::panicking::begin_panic - 6: rustc_typeck::check::cast::>::pointer_kind + 6: rustc_typeck::check::cast::>::pointer_kind (~~~~ LINES REMOVED BY ME FOR BREVITY ~~~~) 32: rustc_typeck::check_crate 33: >::with @@ -36,11 +39,15 @@ stack backtrace: 37: rustc_driver::run_compiler ``` -If you want line numbers for the stack trace, you can enable `debuginfo-lines=true` or `debuginfo=true` in your config.toml and rebuild the compiler. Then the backtrace will look like this: +If you want line numbers for the stack trace, you can enable +`debuginfo-lines=true` or `debuginfo=true` in your config.toml and rebuild the +compiler. Then the backtrace will look like this: + ``` stack backtrace: (~~~~ LINES REMOVED BY ME FOR BREVITY ~~~~) - 6: rustc_typeck::check::cast::>::pointer_kind + 6: rustc_typeck::check::cast::>::pointer_kind at /home/user/rust/src/librustc_typeck/check/cast.rs:110 7: rustc_typeck::check::cast::CastCheck::check at /home/user/rust/src/librustc_typeck/check/cast.rs:572 @@ -104,13 +111,17 @@ note: rustc 1.24.0-dev running on x86_64-unknown-linux-gnu note: run with `RUST_BACKTRACE=1` for a backtrace -thread 'rustc' panicked at 'encountered error with `-Z treat_err_as_bug', /home/user/rust/src/librustc_errors/lib.rs:411:12 -note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. +thread 'rustc' panicked at 'encountered error with `-Z treat_err_as_bug', +/home/user/rust/src/librustc_errors/lib.rs:411:12 +note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose +backtrace. stack backtrace: (~~~ IRRELEVANT PART OF BACKTRACE REMOVED BY ME ~~~) - 7: rustc::traits::error_reporting::>::report_selection_error + 7: rustc::traits::error_reporting::>::report_selection_error at /home/user/rust/src/librustc/traits/error_reporting.rs:823 - 8: rustc::traits::error_reporting::>::report_fulfillment_errors + 8: rustc::traits::error_reporting::>::report_fulfillment_errors at /home/user/rust/src/librustc/traits/error_reporting.rs:160 at /home/user/rust/src/librustc/traits/error_reporting.rs:112 9: rustc_typeck::check::FnCtxt::select_obligations_where_possible @@ -190,18 +201,28 @@ However, there are still a few concerns that you might care about: ### Expensive operations in logs A note of caution: the expressions *within* the `debug!` call are run -whenever RUST_LOG is set, even if the filter would exclude the log. This means that if in the module `rustc::foo` you have a statement +whenever RUST_LOG is set, even if the filter would exclude the log. This means +that if in the module `rustc::foo` you have a statement ```Rust debug!("{:?}", random_operation(tcx)); ``` -Then if someone runs a debug `rustc` with `RUST_LOG=rustc::bar`, then `random_operation()` will still run - even while it's output will never be needed! +Then if someone runs a debug `rustc` with `RUST_LOG=rustc::bar`, then +`random_operation()` will still run - even while it's output will never be +needed! This means that you should not put anything too expensive or likely -to crash there - that would annoy anyone who wants to use logging for their own module. Note that if `RUST_LOG` is unset (the default), then the code will not run - this means that if your logging code panics, then no-one will know it until someone tries to use logging to find *another* bug. +to crash there - that would annoy anyone who wants to use logging for their own +module. Note that if `RUST_LOG` is unset (the default), then the code will not +run - this means that if your logging code panics, then no-one will know it +until someone tries to use logging to find *another* bug. -If you *need* to do an expensive operation in a log, be aware that while log expressions are *evaluated* even if logging is not enabled in your module, they are not *formatted* unless it *is*. This means you can put your expensive/crashy operations inside an `fmt::Debug` impl, and they will not be run unless your log is enabled: +If you *need* to do an expensive operation in a log, be aware that while log +expressions are *evaluated* even if logging is not enabled in your module, +they are not *formatted* unless it *is*. This means you can put your +expensive/crashy operations inside an `fmt::Debug` impl, and they will not be +run unless your log is enabled: ```Rust use std::fmt; @@ -225,8 +246,8 @@ debug!("{:?}", ExpensiveOperationContainer { tcx }); ## Formatting Graphviz output (.dot files) [formatting-graphviz-output]: #formatting-graphviz-output -Some compiler options for debugging specific features yield graphviz graphs - e.g. -the `#[rustc_mir(borrowck_graphviz_postflow="suffix.dot")]` attribute +Some compiler options for debugging specific features yield graphviz graphs - +e.g. the `#[rustc_mir(borrowck_graphviz_postflow="suffix.dot")]` attribute dumps various borrow-checker dataflow graphs. These all produce `.dot` files. To view these files, install graphviz (e.g. @@ -252,7 +273,8 @@ assertions enabled - either an "alt" nightly or a compiler you build yourself by setting `[llvm] assertions=true` in your config.toml - and see whether anything turns up. -The rustc build process builds the LLVM tools into `./build//llvm/bin`. They can be called directly. +The rustc build process builds the LLVM tools into +`./build//llvm/bin`. They can be called directly. The default rustc compilation pipeline has multiple codegen units, which is hard to replicate manually and means that LLVM is called multiple times in parallel. @@ -262,7 +284,8 @@ passing `-C codegen-units=1` to rustc will make debugging easier. If you want to play with the optimization pipeline, you can use the `opt` from there on the IR rustc emits with `--emit=llvm-ir`. Note that rustc emits different IR depending on whether `-O` is enabled, even without -LLVM's optimizations, so if you want to play with the IR rustc emits, you should: +LLVM's optimizations, so if you want to play with the IR rustc emits, +you should: ``` $ rustc +local my-file.rs --emit=llvm-ir -O -C no-prepopulate-passes \ -C codegen-units=1 @@ -275,7 +298,8 @@ IR causes an optimization-time assertion to fail, or to see when LLVM performs a particular optimization, you can pass the rustc flag `-C llvm-args=-print-after-all`, and possibly add `-C llvm-args='-filter-print-funcs=EXACT_FUNCTION_NAME` (e.g. -`-C llvm-args='-filter-print-funcs=_ZN11collections3str21_$LT$impl$u20$str$GT$7replace17hbe10ea2e7c809b0bE'`). +`-C llvm-args='-filter-print-funcs=_ZN11collections3str21_$LT$impl$u20$str$GT$\ + 7replace17hbe10ea2e7c809b0bE'`). That produces a lot of output into standard error, so you'll want to pipe that to some file. Also, if you are using neither `-filter-print-funcs` nor From 29572cd2e4918ad40f0a2028a21ee4e26ed2969e Mon Sep 17 00:00:00 2001 From: cg-cnu Date: Tue, 10 Apr 2018 10:40:06 +0530 Subject: [PATCH 196/648] refactor: fixed typos, text formatting suggested in the review --- src/SUMMARY.md | 2 +- src/compiler-debugging.md | 47 +++++++++++++++++++++------------------ 2 files changed, 26 insertions(+), 23 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 1b9670261..1839f59cb 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -8,6 +8,7 @@ - [Adding new tests](./tests/adding.md) - [Using `compiletest` + commands to control test execution](./compiletest.md) +- [Debugging the Compiler](./compiler-debugging.md) - [Walkthrough: a typical contribution](./walkthrough.md) - [High-level overview of the compiler source](./high-level-overview.md) - [The Rustc Driver](./rustc-driver.md) @@ -52,7 +53,6 @@ - [miri const evaluator](./miri.md) - [Parameter Environments](./param_env.md) - [Generating LLVM IR](./trans.md) -- [Debugging the Compiler](./compiler-debugging.md) --- diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index 41b50b04a..fc74caa8f 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -20,7 +20,7 @@ you might want to find some way to use Linux or MSVC on Windows. In the default configuration, you don't have line numbers enabled, so the backtrace looks like this: - ``` +```text stack backtrace: 0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace 1: std::sys_common::backtrace::_print @@ -28,8 +28,6 @@ stack backtrace: 3: std::panicking::default_hook 4: std::panicking::rust_panic_with_hook 5: std::panicking::begin_panic - 6: rustc_typeck::check::cast::>::pointer_kind (~~~~ LINES REMOVED BY ME FOR BREVITY ~~~~) 32: rustc_typeck::check_crate 33: >::with @@ -37,17 +35,15 @@ stack backtrace: 35: rustc::ty::context::TyCtxt::create_and_enter 36: rustc_driver::driver::compile_input 37: rustc_driver::run_compiler - ``` +``` If you want line numbers for the stack trace, you can enable `debuginfo-lines=true` or `debuginfo=true` in your config.toml and rebuild the compiler. Then the backtrace will look like this: -``` +```text stack backtrace: (~~~~ LINES REMOVED BY ME FOR BREVITY ~~~~) - 6: rustc_typeck::check::cast::>::pointer_kind at /home/user/rust/src/librustc_typeck/check/cast.rs:110 7: rustc_typeck::check::cast::CastCheck::check at /home/user/rust/src/librustc_typeck/check/cast.rs:572 @@ -72,11 +68,15 @@ This can also help when debugging `delay_span_bug` calls - it will make the first `delay_span_bug` call panic, which will give you a useful backtrace. For example: -``` + +```rust $ cat error.rs fn main() { 1 + (); } +``` + +```bash $ ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc error.rs error[E0277]: the trait bound `{integer}: std::ops::Add<()>` is not satisfied --> error.rs:2:7 @@ -137,8 +137,8 @@ $ # Cool, now I have a backtrace for the error The compiler has a lot of `debug!` calls, which print out logging information at many points. These are very useful to at least narrow down the location of -a bug if not to find it entirely, or just to orient yourself to why a compiler -is doing a particular thing. +a bug if not to find it entirely, or just to orient yourself as to why the +compiler is doing a particular thing. To see the logs, you need to set the `RUST_LOG` environment variable to your log filter, e.g. to get the logs for a specific module, you can run the @@ -152,7 +152,8 @@ of output - so it's typically a good idea to pipe standard error to a file and look at the log output with a text editor. So to put it together. -``` + +```bash # This puts the output of all debug calls in `librustc/traits` into # standard error, which might fill your console backscroll. $ RUST_LOG=rustc::traits rustc +local my-file.rs @@ -191,10 +192,10 @@ want to call `x.py clean` to force one. ### Logging etiquette Because calls to `debug!` are removed by default, in most cases, don't worry -about adding "unnecessary" calls to `debug!` and leaving them in in code -you commit - they won't slow -down the performance of what we ship, and if they helped you pinning down -a bug, they will probably help someone else with a different one. +about adding "unnecessary" calls to `debug!` and leaving them in code you +commit - they won't slow down the performance of what we ship, and if they +helped you pinning down a bug, they will probably help someone else with a +different one. However, there are still a few concerns that you might care about: @@ -253,7 +254,7 @@ dumps various borrow-checker dataflow graphs. These all produce `.dot` files. To view these files, install graphviz (e.g. `apt-get install graphviz`) and then run the following commands: -``` +```bash $ dot -T pdf maybe_init_suffix.dot > maybe_init_suffix.pdf $ firefox maybe_init_suffix.pdf # Or your favorite pdf viewer ``` @@ -281,12 +282,13 @@ to replicate manually and means that LLVM is called multiple times in parallel. If you can get away with it (i.e. if it doesn't make your bug disappear), passing `-C codegen-units=1` to rustc will make debugging easier. -If you want to play with the optimization pipeline, you can use the `opt` from -there on the IR rustc emits with `--emit=llvm-ir`. Note -that rustc emits different IR depending on whether `-O` is enabled, even without -LLVM's optimizations, so if you want to play with the IR rustc emits, +If you want to play with the optimization pipeline, you can use the opt tool +from `./build//llvm/bin/` with the the LLVM IR emitted by rustc. +Note that rustc emits different IR depending on whether `-O` is enabled, even +without LLVM's optimizations, so if you want to play with the IR rustc emits, you should: -``` + +```bash $ rustc +local my-file.rs --emit=llvm-ir -O -C no-prepopulate-passes \ -C codegen-units=1 $ OPT=./build/$TRIPLE/llvm/bin/opt @@ -309,7 +311,8 @@ the printouts will mix together and you won't be able to read anything. If you want just the IR for a specific function (say, you want to see why it causes an assertion or doesn't optimize correctly), you can use `llvm-extract`, e.g. -``` + +```bash $ ./build/$TRIPLE/llvm/bin/llvm-extract \ -func='_ZN11collections3str21_$LT$impl$u20$str$GT$7replace17hbe10ea2e7c809b0bE' \ -S \ From b4228a73ce217866eb1714e50c6f8485a6754b04 Mon Sep 17 00:00:00 2001 From: cg-cnu Date: Tue, 10 Apr 2018 10:43:09 +0530 Subject: [PATCH 197/648] fix: changed rust code to bash formatting --- src/compiler-debugging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index fc74caa8f..80cbf6735 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -69,7 +69,7 @@ the first `delay_span_bug` call panic, which will give you a useful backtrace. For example: -```rust +```bash $ cat error.rs fn main() { 1 + (); From eb396e7883e10fb4f338c881609a4f70b67bfacc Mon Sep 17 00:00:00 2001 From: cg-cnu Date: Sun, 15 Apr 2018 11:49:15 +0530 Subject: [PATCH 198/648] add: note about copy from rust-forge --- src/compiler-debugging.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index 80cbf6735..4b445c924 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -3,6 +3,10 @@ layout: default title: Debugging the Compiler --- +**Note: This is copied from the +[rust-forge](https://github.com/rust-lang-nursery/rust-forge). If anything needs + updating, please open an issue or make a PR on the github repo.** + # Debugging the compiler [debugging]: #debugging From 30d20184b3d27f7ec044c6cfe210842a4ea10738 Mon Sep 17 00:00:00 2001 From: "Ehsan M.Kermani" Date: Sun, 22 Apr 2018 16:34:32 -0700 Subject: [PATCH 199/648] expand compilation stages --- src/how-to-build-and-run.md | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 535823dfd..dfaf8f79b 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -2,7 +2,7 @@ The compiler is built using a tool called `x.py`. You will need to have Python installed to run it. But before we get to that, if you're going to -be hacking on rustc, you'll want to tweak the configuration of the compiler. +be hacking on `rustc`, you'll want to tweak the configuration of the compiler. The default configuration is oriented towards running the compiler as a user, not a developer. @@ -55,13 +55,20 @@ compiler to compile the newer version. In particular, the newer version of the compiler, `libstd`, and other tooling may use some unstable features internally. The result is the compiling `rustc` is done in stages. -- **Stage 0:** the stage0 compiler is the current _beta_ compiler; we - download this binary from the internet. -- **Stage 1:** the code in your clone is then compiled with the stage - 0 compiler to produce the stage 1 compiler. -- **Stage 2:** the code in your clone is then compiled with the stage - 1 compiler *again* to produce the stage 2 compiler (i.e. it builds - itself). +- **Stage 0:** the stage0 compiler can be your existing (perhaps older version of) + Rust compiler, the current _beta_ compiler or you may download the binary + from the internet +- **Stage 1:** the code in your clone (for new version) + is then compiled with the stage0 + compiler to produce the stage1 compiler. + However, it was built with an older compiler (stage0), + so to optimize the stage1 compiler we go to next stage +- **Stage 2:** we rebuild our stage1 compiler with itself + to produce the stage2 compiler (i.e. it builds + itself) to have all the _latest optimizations_ +- _(Optional)_ **Stage 3**: to sanity check of our new compiler, we can build it again + with stage2 compiler which must be identical to itself, + unless something has broken For hacking, often building the stage 1 compiler is enough, but for final testing and release, the stage 2 compiler is used. From 9140b49321440ca4f275d4a7601768b7e1928c81 Mon Sep 17 00:00:00 2001 From: "Ehsan M.Kermani" Date: Sun, 22 Apr 2018 16:46:12 -0700 Subject: [PATCH 200/648] fix max line len --- src/how-to-build-and-run.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index dfaf8f79b..2d979c0d1 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -28,8 +28,8 @@ settings (and possibly others, such as `llvm.ccache`): assertions = true [rust] -# This enables some assertions, but more importantly it enables the `debug!` logging -# macros that are essential for debugging rustc. +# This enables some assertions, but more importantly it enables the `debug!` +# logging macros that are essential for debugging rustc. debug-assertions = true # This will make your build more parallel; it costs a bit of runtime @@ -55,7 +55,8 @@ compiler to compile the newer version. In particular, the newer version of the compiler, `libstd`, and other tooling may use some unstable features internally. The result is the compiling `rustc` is done in stages. -- **Stage 0:** the stage0 compiler can be your existing (perhaps older version of) +- **Stage 0:** the stage0 compiler can be your existing + (perhaps older version of) Rust compiler, the current _beta_ compiler or you may download the binary from the internet - **Stage 1:** the code in your clone (for new version) @@ -66,7 +67,8 @@ internally. The result is the compiling `rustc` is done in stages. - **Stage 2:** we rebuild our stage1 compiler with itself to produce the stage2 compiler (i.e. it builds itself) to have all the _latest optimizations_ -- _(Optional)_ **Stage 3**: to sanity check of our new compiler, we can build it again +- _(Optional)_ **Stage 3**: to sanity check of our new compiler, + we can build it again with stage2 compiler which must be identical to itself, unless something has broken From 7d4aee2431e6362d24ce1f530ae214ff72165100 Mon Sep 17 00:00:00 2001 From: "Ehsan M.Kermani" Date: Sat, 28 Apr 2018 10:40:06 -0700 Subject: [PATCH 201/648] lint add dots --- src/how-to-build-and-run.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 2d979c0d1..493f642a5 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -58,19 +58,19 @@ internally. The result is the compiling `rustc` is done in stages. - **Stage 0:** the stage0 compiler can be your existing (perhaps older version of) Rust compiler, the current _beta_ compiler or you may download the binary - from the internet + from the internet. - **Stage 1:** the code in your clone (for new version) is then compiled with the stage0 compiler to produce the stage1 compiler. However, it was built with an older compiler (stage0), - so to optimize the stage1 compiler we go to next stage + so to optimize the stage1 compiler we go to next stage. - **Stage 2:** we rebuild our stage1 compiler with itself to produce the stage2 compiler (i.e. it builds - itself) to have all the _latest optimizations_ + itself) to have all the _latest optimizations_. - _(Optional)_ **Stage 3**: to sanity check of our new compiler, we can build it again with stage2 compiler which must be identical to itself, - unless something has broken + unless something has broken. For hacking, often building the stage 1 compiler is enough, but for final testing and release, the stage 2 compiler is used. From 55ddb4df95b202eeb2490b13f0469fc391110328 Mon Sep 17 00:00:00 2001 From: Nick Fitzgerald Date: Wed, 25 Apr 2018 15:26:40 -0700 Subject: [PATCH 202/648] appendix-background: Link to *Static Program Analysis* A+++++ resource IMHO! --- src/appendix-background.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/appendix-background.md b/src/appendix-background.md index 285d74477..a0e298b05 100644 --- a/src/appendix-background.md +++ b/src/appendix-background.md @@ -76,6 +76,9 @@ cycle. ## What is a dataflow analysis? +[*Static Program Analysis*](https://cs.au.dk/~amoeller/spa/) by Anders Møller +and Michael I. Schwartzbach is an incredible resource! + *to be written* From e39ea751a08f135009aa7aaa07f68b4eabd05e1a Mon Sep 17 00:00:00 2001 From: gnzlbg Date: Tue, 8 May 2018 09:40:43 +0200 Subject: [PATCH 203/648] Add how to run tidy script in isolation to text This is already mentioned elsewhere, but important things are repeated twice. Closes #124 --- src/conventions.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/conventions.md b/src/conventions.md index 89a986789..7c2a958c0 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -11,7 +11,8 @@ rustc is slowly moving towards the [Rust standard coding style][fmt]; at the moment, however, it follows a rather more *chaotic* style. We do have some mandatory formatting conventions, which are automatically enforced by a script we affectionately call the "tidy" script. The -tidy script runs automatically when you do `./x.py test`. +tidy script runs automatically when you do `./x.py test` and can be run +in isolation with `./x.py test src/tools/tidy`. [fmt]: https://github.com/rust-lang-nursery/fmt-rfcs From e5bed1889527ee86ad6823d5bb61e266c7d8c306 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 8 May 2018 12:44:02 -0500 Subject: [PATCH 204/648] update to mdbook 0.1.7 --- ci/install.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install.sh b/ci/install.sh index ea90e853d..69b96cc73 100755 --- a/ci/install.sh +++ b/ci/install.sh @@ -20,5 +20,5 @@ function cargo_install() { fi } -cargo_install mdbook 0.1.5 +cargo_install mdbook 0.1.7 cargo_install mdbook-linkcheck 0.1.2 From 1cea2615e16d6cf1b8e9a41bc26a1b403994a238 Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Thu, 10 May 2018 08:09:56 -0400 Subject: [PATCH 205/648] Issue84 Add note about ./x.py build --- src/how-to-build-and-run.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 493f642a5..66c736548 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -75,6 +75,11 @@ internally. The result is the compiling `rustc` is done in stages. For hacking, often building the stage 1 compiler is enough, but for final testing and release, the stage 2 compiler is used. +`./x.py check` is really fast to build the rust compiler. +It is, in particular, very useful when you're doing some kind of +"type-based refactoring", like renaming a method, or changing the +signature of some function. + Once you've created a config.toml, you are now ready to run `x.py`. There are a lot of options here, but let's start with what is probably the best "go to" command for building a local rust: From 45cb1d368b060382b316a3befcc9019871d4fdd9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tom=C3=A1=C5=A1=20Gaven=C4=8Diak?= Date: Thu, 3 May 2018 00:25:31 +0200 Subject: [PATCH 206/648] Update adding.md Rename `must-compile-successfully` to `compile-pass` in ui test doc as per https://github.com/rust-lang/rust/issues/49568. --- src/tests/adding.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/tests/adding.md b/src/tests/adding.md index 1445981eb..5d4c383f2 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -150,7 +150,7 @@ source. * `min-{gdb,lldb}-version` * `min-llvm-version` -* `must-compile-successfully` for UI tests, indicates that the test is +* `compile-pass` for UI tests, indicates that the test is supposed to compile, as opposed to the default where the test is supposed to error out. * `compile-flags` passes extra command-line args to the compiler, @@ -258,7 +258,7 @@ can also make UI tests where compilation is expected to succeed, and you can even run the resulting program. Just add one of the following [header commands](#header_commands): -- `// must-compile-successfully` -- compilation should succeed but do +- `// compile-pass` -- compilation should succeed but do not run the resulting binary - `// run-pass` -- compilation should succeed and we should run the resulting binary From eeddfdaf703cb17c3824ca85dc1542fc2ee93329 Mon Sep 17 00:00:00 2001 From: Sunjay Varma Date: Wed, 9 May 2018 17:43:43 -0700 Subject: [PATCH 207/648] Deleted tag which was messing up the associated types page --- src/traits-associated-types.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/traits-associated-types.md b/src/traits-associated-types.md index e2dd94d5a..eaf497875 100644 --- a/src/traits-associated-types.md +++ b/src/traits-associated-types.md @@ -22,8 +22,6 @@ though that is something we may want to change in the future.) [intoiter-item]: https://doc.rust-lang.org/nightly/core/iter/trait.IntoIterator.html#associatedtype.Item - - In some cases, associated type projections can be **normalized** -- that is, simplified -- based on the types given in an impl. So, to continue with our example, the impl of `IntoIterator` for `Option` @@ -152,4 +150,3 @@ to process that constraint. variable X and asks us to prove that `ProjectionEq(P1 = X)` and `ProjectionEq(P2 = X)`. That used to be needed in an older system to prevent cycles; I rather doubt it still is. -nmatsakis) - From 0dc0dddb7904516feaf1c9bd261596bf8ba526fe Mon Sep 17 00:00:00 2001 From: Sunjay Varma Date: Thu, 10 May 2018 11:51:34 -0700 Subject: [PATCH 208/648] Closing all tags --- src/appendix-background.md | 10 +++++----- src/conventions.md | 10 +++++----- src/incremental-compilation.md | 2 +- src/mir-regionck.md | 4 ++-- src/mir.md | 2 +- src/tests/adding.md | 8 ++++---- src/traits-associated-types.md | 2 ++ src/traits-bibliography.md | 4 ++-- src/traits-canonical-queries.md | 2 +- src/traits-goals-and-clauses.md | 8 ++++---- src/traits-lowering-module.md | 2 +- src/traits-lowering-rules.md | 6 +++--- src/type-inference.md | 2 +- src/variance.md | 2 +- 14 files changed, 33 insertions(+), 31 deletions(-) diff --git a/src/appendix-background.md b/src/appendix-background.md index a0e298b05..069131947 100644 --- a/src/appendix-background.md +++ b/src/appendix-background.md @@ -4,7 +4,7 @@ This section covers a numbers of common compiler terms that arise in this guide. We try to give the general definition while providing some Rust-specific context. - + ## What is a control-flow graph? @@ -72,7 +72,7 @@ When using a control-flow graph, a loop simply appears as a cycle in the graph, and the `break` keyword translates into a path out of that cycle. - + ## What is a dataflow analysis? @@ -81,13 +81,13 @@ and Michael I. Schwartzbach is an incredible resource! *to be written* - + ## What is "universally quantified"? What about "existentially quantified"? *to be written* - + ## What is co- and contra-variance? @@ -97,7 +97,7 @@ Check out the subtyping chapter from the See the [variance](./variance.html) chapter of this guide for more info on how the type checker handles variance. - + ## What is a "free region" or a "free variable"? What about "bound region"? diff --git a/src/conventions.md b/src/conventions.md index 7c2a958c0..5b2b7b7f6 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -3,7 +3,7 @@ chapter covers [formatting](#formatting), [coding for correctness](#cc), [using crates from crates.io](#cio), and some tips on [structuring your PR for easy review](#er). - + # Formatting and the tidy script @@ -16,7 +16,7 @@ in isolation with `./x.py test src/tools/tidy`. [fmt]: https://github.com/rust-lang-nursery/fmt-rfcs - + ### Copyright notice @@ -57,7 +57,7 @@ the copyright notice) like so: Prefer 4-space indent. - + # Coding for correctness @@ -99,7 +99,7 @@ if foo { } ``` - + # Using crates from crates.io @@ -108,7 +108,7 @@ dependencies should not be added gratuitously. All such crates must have a suitably permissive license. There is an automatic check which inspects the Cargo metadata to ensure this. - + # How to structure your PR diff --git a/src/incremental-compilation.md b/src/incremental-compilation.md index a209207fc..0a25e1664 100644 --- a/src/incremental-compilation.md +++ b/src/incremental-compilation.md @@ -76,7 +76,7 @@ Try-mark-green works as follows: - Otherwise, **all** of the nodes in `reads(Q)` must be **green**. In that case, we can color Q as **green** and return. - + ### The query DAG diff --git a/src/mir-regionck.md b/src/mir-regionck.md index 4158b7d38..e67e91138 100644 --- a/src/mir-regionck.md +++ b/src/mir-regionck.md @@ -51,7 +51,7 @@ the role of `liveness_constraints` vs other `constraints`, plus *to be written* - + ## The MIR type-check @@ -88,7 +88,7 @@ The kinds of region elements are as follows: *to be written* -- describe how we can extend the values of a variable with causal tracking etc - + ## Skolemization and universes diff --git a/src/mir.md b/src/mir.md index da468acf4..81bbf6bd6 100644 --- a/src/mir.md +++ b/src/mir.md @@ -234,7 +234,7 @@ but [you can read about those below](#promoted)). *to be written* - + ### Promoted constants diff --git a/src/tests/adding.md b/src/tests/adding.md index 5d4c383f2..96fc0cf84 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -68,7 +68,7 @@ then it might make sense to put the tests in directories like: In other cases, there may already be a suitable directory. (The proper directory structure to use is actually an area of active debate.) - + ## Comment explaining what the test is about @@ -90,7 +90,7 @@ test must be rewritten because it no longer tests what is was meant to test, and then it's useful to know what it *was* meant to test exactly). - + ## Header commands: configuring rustc @@ -167,7 +167,7 @@ source. [`header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs - + ## Error annotations @@ -229,7 +229,7 @@ currently only apply to the test as a whole, not to particular revisions. The only headers that are intended to really work when customized to a revision are error patterns and compiler flags. - + ## Guide to the UI tests diff --git a/src/traits-associated-types.md b/src/traits-associated-types.md index eaf497875..51972c41e 100644 --- a/src/traits-associated-types.md +++ b/src/traits-associated-types.md @@ -22,6 +22,8 @@ though that is something we may want to change in the future.) [intoiter-item]: https://doc.rust-lang.org/nightly/core/iter/trait.IntoIterator.html#associatedtype.Item + + In some cases, associated type projections can be **normalized** -- that is, simplified -- based on the types given in an impl. So, to continue with our example, the impl of `IntoIterator` for `Option` diff --git a/src/traits-bibliography.md b/src/traits-bibliography.md index 02607b525..b8f345d9d 100644 --- a/src/traits-bibliography.md +++ b/src/traits-bibliography.md @@ -10,7 +10,7 @@ new every time you open it. [phl]: https://www.amazon.com/Programming-Higher-Order-Logic-Dale-Miller/dp/052187940X - + ["A proof procedure for the logic of Hereditary Harrop formulas"][pphhf], by Gopalan Nadathur. This paper covers the basics of universes, @@ -18,7 +18,7 @@ environments, and Lambda Prolog-style proof search. Quite readable. [pphhf]: https://dl.acm.org/citation.cfm?id=868380 - + ["A new formulation of tabled resolution with delay"][nftrd], by [Theresa Swift]. This paper gives a kind of abstract treatment of the diff --git a/src/traits-canonical-queries.md b/src/traits-canonical-queries.md index 3c4bb1bfe..2737737bb 100644 --- a/src/traits-canonical-queries.md +++ b/src/traits-canonical-queries.md @@ -95,7 +95,7 @@ Rc: Clone After all, `Rc` is true **no matter what type `?T` is**. - + ## A trait query in rustc diff --git a/src/traits-goals-and-clauses.md b/src/traits-goals-and-clauses.md index 5e8ee1469..0cbdb7077 100644 --- a/src/traits-goals-and-clauses.md +++ b/src/traits-goals-and-clauses.md @@ -39,11 +39,11 @@ gives the details. [pphhf]: ./traits-bibliography.html#pphhf - + ## Domain goals - + To define the set of *domain goals* in our system, we need to first introduce a few simple formulations. A **trait reference** consists of @@ -58,7 +58,7 @@ IntoIterator`. Note that Rust surface syntax also permits some extra things, like associated type bindings (`Vec: IntoIterator`), that are not part of a trait reference. - + A **projection** consists of an associated item reference along with its inputs P0..Pm: @@ -105,7 +105,7 @@ DomainGoal = Implemented(TraitRef) [n]: ./traits-associated-types.html#normalize - + ## Coinductive goals diff --git a/src/traits-lowering-module.md b/src/traits-lowering-module.md index fbf1d6425..08e0b9523 100644 --- a/src/traits-lowering-module.md +++ b/src/traits-lowering-module.md @@ -18,7 +18,7 @@ returns a vector of program clauses. [query]: ./query.html - + ## Unit tests diff --git a/src/traits-lowering-rules.md b/src/traits-lowering-rules.md index f7433221e..73660c42c 100644 --- a/src/traits-lowering-rules.md +++ b/src/traits-lowering-rules.md @@ -94,7 +94,7 @@ forall { } ``` - + #### Implied bounds @@ -176,7 +176,7 @@ we must show that `WellFormed(TraitRef)`. This in turn justifies the implied bounds rules that allow us to extend the set of `FromEnv` items. - + ## Lowering trait items @@ -291,7 +291,7 @@ forall { Note that `WC` and `WC1` both encode where-clauses that the impl can rely on. - + ### Function and constant values diff --git a/src/type-inference.md b/src/type-inference.md index 152bbd9da..e480b2b56 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -43,7 +43,7 @@ The `tcx.infer_ctxt` method actually returns a builder, which means there are some kinds of configuration you can do before the `infcx` is created. See `InferCtxtBuilder` for more information. - + ## Inference variables diff --git a/src/variance.md b/src/variance.md index 527c2745c..08399b5b3 100644 --- a/src/variance.md +++ b/src/variance.md @@ -141,7 +141,7 @@ will wind up being considered green after it is re-evaluated. [rga]: ./incremental-compilation.html - + ## Addendum: Variance on traits From 8cfd5c5aede5fe17cc286c230ccd2eba4cc966bd Mon Sep 17 00:00:00 2001 From: Sunjay Varma Date: Thu, 10 May 2018 15:22:12 -0700 Subject: [PATCH 209/648] Added an overview of chalk --- src/SUMMARY.md | 1 + src/chalk-overview.md | 140 ++++++++++++++++++++++++++++++++++++++++++ src/traits-slg.md | 2 + 3 files changed, 143 insertions(+) create mode 100644 src/chalk-overview.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 1839f59cb..4923d4972 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -38,6 +38,7 @@ - [The lowering module in rustc](./traits-lowering-module.md) - [Well-formedness checking](./traits-wf.md) - [The SLG solver](./traits-slg.md) + - [An Overview of Chalk](./chalk-overview.md) - [Bibliography](./traits-bibliography.md) - [Type checking](./type-checking.md) - [Method Lookup](./method-lookup.md) diff --git a/src/chalk-overview.md b/src/chalk-overview.md new file mode 100644 index 000000000..63757f7ba --- /dev/null +++ b/src/chalk-overview.md @@ -0,0 +1,140 @@ +# An Overview of Chalk + +> Chalk is under heavy development, so if any of these links are broken or if +> any of the information is inconsistent with the code or outdated, please open +> an issue so we can fix it. If you are able to fix the issue yourself, we would +> love your contribution! + +[Chalk][chalk] recasts Rust's trait system explicitly in terms of logic +programming by "lowering" Rust code into a kind of logic program we can then +execute queries against. Its goal is to be an executable, highly readable +specification of the Rust trait system.[^negativechalk] + +There are many expected benefits from this work. It will consolidate our +existing, somewhat ad-hoc implementation into something far more principled and +expressive, which should behave better in corner cases, and be much easier to +extend.[^negativechalk] + +## Resources + +* [Chalk Source Code](https://github.com/rust-lang-nursery/chalk) +* [Chalk Glossary](https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md) +* The traits section of the rustc guide (you are here) + +### Blog Posts + +* [Lowering Rust traits to logic](http://smallcultfollowing.com/babysteps/blog/2017/01/26/lowering-rust-traits-to-logic/) +* [Unification in Chalk, part 1](http://smallcultfollowing.com/babysteps/blog/2017/03/25/unification-in-chalk-part-1/) +* [Unification in Chalk, part 2](http://smallcultfollowing.com/babysteps/blog/2017/04/23/unification-in-chalk-part-2/) +* [Negative reasoning in Chalk](http://aturon.github.io/blog/2017/04/24/negative-chalk/) +* [Query structure in chalk](http://smallcultfollowing.com/babysteps/blog/2017/05/25/query-structure-in-chalk/) +* [Cyclic queries in chalk](http://smallcultfollowing.com/babysteps/blog/2017/09/12/tabling-handling-cyclic-queries-in-chalk/) +* [An on-demand SLG solver for chalk](http://smallcultfollowing.com/babysteps/blog/2018/01/31/an-on-demand-slg-solver-for-chalk/) + +## Parsing + +Chalk is designed to be incorporated with the Rust compiler, so the syntax and +concepts it deals with heavily borrow from Rust. It is convenient for the sake +of testing to be able to run chalk on its own, so chalk includes a parser for a +Rust-like syntax. + +The parser takes that syntax and produces an [Abstract Syntax Tree (AST)][ast]. +You can find the [complete definition of the AST][chalk-ast] in the source code. + +The syntax contains things from Rust that we know and love for example traits, +impls, and struct definitions. Parsing is often the first "phase" of +transformation that a program goes through in order to become a format that +chalk can understand. + +## Lowering + +After parsing, there is a "lowering" phase. This aims to convert traits/impls +into "program clauses". A [`ProgramClause` (source code)][programclause] is +essentially one of the following: + +* A [clause] of the form `consequence :- conditions` where `:-` is read as + "if" and `conditions = cond1 && cond2 && ...` +* A universally quantified clause of the form `forall { consequence :- conditions }` + * `forall { ... }` is used to represent [universal quantification]. See the + section on [Lowering to logic][lowering-forall] for more information. + * A key thing to note about `forall` is that we don't allow you to "quantify" + over traits, only types and regions (lifetimes). That is, you can't make a + rule like `forall { u32: Trait }` which would say "`u32` implements + all traits". You can however say `forall { T: Trait }` meaning "`Trait` + is implemented by all types". + * `forall { ... }` is represented in the code using the [`Binders` + struct][binders-struct]. + +This is the phase where we encode the rules of the trait system into logic. For +example, if we have: + +```rust +impl Clone for Vec {} +``` + +We generate: + +```rust +forall { (Vec: Clone) :- (T: Clone) } +``` + +This rule dictates that `Vec: Clone` is only satisfied if `T: Clone` is also +satisfied (i.e. "provable"). + +### Well-formedness checks + +As part of lowering from the AST to the internal IR, we also do some "well +formedness" checks. See the [source code][well-formedness-checks] for where +those are done. The call to `record_specialization_priorities` checks +"coherence" which means that it ensures that two impls of the same trait for the +same type cannot exist. + +## Intermediate Representation (IR) + +The second intermediate representation in chalk is called, well, the "ir". :) +The [IR source code][ir-code] contains the complete definition. The +`ir::Program` struct contains some "rust things" but indexed and accessible in +a different way. This is sort of analogous to the [HIR] in Rust. + +For example, if you have a type like `Foo`, we would represent `Foo` as a +string in the AST but in `ir::Program`, we use numeric indices (`ItemId`). + +In addition to `ir::Program` which has "rust-like things", there is also +`ir::ProgramEnvironment` which is "pure logic". The main field in that is +`program_clauses` which contains the `ProgramClause`s that we generated +previously. + +## Rules + +The `rules` module works by iterating over every trait, impl, etc. and emitting +the rules that come from each one. The traits section of the rustc-guide (that +you are currently reading) contains the most up-to-date reference on that. + +The `ir::ProgramEnvironment` is created [in this module][rules-environment]. + +## Testing + +TODO: Basically, [there is a macro](https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/solve/test/mod.rs#L112-L148) +that will take syntax and run it through the full pipeline described above. +[This](https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/solve/test/mod.rs#L83-L110) +is the function that is ultimately called. + +## Solver + +See [The SLG Solver][slg]. + +[^negativechalk]: [*Negative reasoning in Chalk* by Aaron Turon](http://aturon.github.io/blog/2017/04/24/negative-chalk/) + +[chalk]: https://github.com/rust-lang-nursery/chalk +[ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree +[chalk-ast]: https://github.com/rust-lang-nursery/chalk/blob/master/chalk-parse/src/ast.rs +[universal quantification]: https://en.wikipedia.org/wiki/Universal_quantification +[lowering-forall]: traits-lowering-to-logic.html#type-checking-generic-functions-beyond-horn-clauses +[programclause]: https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/ir/mod.rs#L721 +[clause]: https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md#clause +[well-formedness-checks]: https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/ir/lowering/mod.rs#L230-L232 +[ir-code]: https://github.com/rust-lang-nursery/chalk/blob/master/src/ir/mod.rs +[HIR]: hir.html +[binders-struct]: https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/ir/mod.rs#L661 +[rules-environment]: https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/rules/mod.rs#L9 +[slg]: traits-slg.html diff --git a/src/traits-slg.md b/src/traits-slg.md index cdd52ece9..1dc56e14c 100644 --- a/src/traits-slg.md +++ b/src/traits-slg.md @@ -1 +1,3 @@ # The SLG solver + +TODO: From 58ef5a78deb90d0e0306b85bb5ee70fd76eecf6a Mon Sep 17 00:00:00 2001 From: Sunjay Varma Date: Thu, 10 May 2018 15:43:18 -0700 Subject: [PATCH 210/648] Lines must be <= 80 characters UNLESS there is a link --- src/chalk-overview.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/chalk-overview.md b/src/chalk-overview.md index 63757f7ba..8c5bc1594 100644 --- a/src/chalk-overview.md +++ b/src/chalk-overview.md @@ -54,7 +54,8 @@ essentially one of the following: * A [clause] of the form `consequence :- conditions` where `:-` is read as "if" and `conditions = cond1 && cond2 && ...` -* A universally quantified clause of the form `forall { consequence :- conditions }` +* A universally quantified clause of the form + `forall { consequence :- conditions }` * `forall { ... }` is used to represent [universal quantification]. See the section on [Lowering to logic][lowering-forall] for more information. * A key thing to note about `forall` is that we don't allow you to "quantify" @@ -129,7 +130,7 @@ See [The SLG Solver][slg]. [ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree [chalk-ast]: https://github.com/rust-lang-nursery/chalk/blob/master/chalk-parse/src/ast.rs [universal quantification]: https://en.wikipedia.org/wiki/Universal_quantification -[lowering-forall]: traits-lowering-to-logic.html#type-checking-generic-functions-beyond-horn-clauses +[lowering-forall]: https://rust-lang-nursery.github.io/rustc-guide/traits-lowering-to-logic.html#type-checking-generic-functions-beyond-horn-clauses [programclause]: https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/ir/mod.rs#L721 [clause]: https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md#clause [well-formedness-checks]: https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/ir/lowering/mod.rs#L230-L232 From b9eb2b6c1369fbaf4a5a845ab3a7dde050c71e23 Mon Sep 17 00:00:00 2001 From: Sunjay Varma Date: Thu, 10 May 2018 15:48:24 -0700 Subject: [PATCH 211/648] Ignoring code examples that aren't actually compile-able --- src/chalk-overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/chalk-overview.md b/src/chalk-overview.md index 8c5bc1594..49f685149 100644 --- a/src/chalk-overview.md +++ b/src/chalk-overview.md @@ -69,13 +69,13 @@ essentially one of the following: This is the phase where we encode the rules of the trait system into logic. For example, if we have: -```rust +```rust,ignore impl Clone for Vec {} ``` We generate: -```rust +```rust,ignore forall { (Vec: Clone) :- (T: Clone) } ``` From c11f57a262f69c0dfce3310e0a70d3da0a3040a6 Mon Sep 17 00:00:00 2001 From: C Jones Date: Fri, 11 May 2018 18:16:15 -0400 Subject: [PATCH 212/648] Make chapter links in compiletest not be code mdBook doesn't display links inside code, so these links look very confusing. Additionally, these types of links don't seem to be code on other pages of the book. --- src/compiletest.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/compiletest.md b/src/compiletest.md index 363c12d3b..cdcede33a 100644 --- a/src/compiletest.md +++ b/src/compiletest.md @@ -13,7 +13,7 @@ cases, even failure to compile. Tests are typically organized as a Rust source file with annotations in comments before and/or within the test code, which serve to direct `compiletest` on if or how to run the test, what behavior to expect, and more. If you are unfamiliar with the compiler testing framework, -see [`this chapter`](./tests/intro.html) for additional background. +see [this chapter](./tests/intro.html) for additional background. The tests themselves are typically (but not always) organized into "suites"--for example, `run-pass`, a folder representing tests that should @@ -22,8 +22,8 @@ but return a failure (non-zero status), `compile-fail`, a folder holding tests that should fail to compile, and many more. The various suites are defined in [src/tools/compiletest/src/common.rs][common] in the `pub struct Config` declaration. And a very good introduction to the different suites of compiler -tests along with details about them can be found in [`Adding new -tests`](./tests/adding.html). +tests along with details about them can be found in [Adding new +tests](./tests/adding.html). ## Adding a new test file From f601a3aacaa32b77325e051b9f644d85d0e650b0 Mon Sep 17 00:00:00 2001 From: Sunjay Varma Date: Fri, 11 May 2018 17:14:26 -0700 Subject: [PATCH 213/648] Updating links to match latest code --- src/chalk-overview.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/chalk-overview.md b/src/chalk-overview.md index 49f685149..7cea6acef 100644 --- a/src/chalk-overview.md +++ b/src/chalk-overview.md @@ -115,9 +115,9 @@ The `ir::ProgramEnvironment` is created [in this module][rules-environment]. ## Testing -TODO: Basically, [there is a macro](https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/solve/test/mod.rs#L112-L148) +TODO: Basically, [there is a macro](https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/solve/test.rs#L112-L148) that will take syntax and run it through the full pipeline described above. -[This](https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/solve/test/mod.rs#L83-L110) +[This](https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/solve/test.rs#L83-L110) is the function that is ultimately called. ## Solver @@ -131,11 +131,11 @@ See [The SLG Solver][slg]. [chalk-ast]: https://github.com/rust-lang-nursery/chalk/blob/master/chalk-parse/src/ast.rs [universal quantification]: https://en.wikipedia.org/wiki/Universal_quantification [lowering-forall]: https://rust-lang-nursery.github.io/rustc-guide/traits-lowering-to-logic.html#type-checking-generic-functions-beyond-horn-clauses -[programclause]: https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/ir/mod.rs#L721 +[programclause]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L721 [clause]: https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md#clause -[well-formedness-checks]: https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/ir/lowering/mod.rs#L230-L232 -[ir-code]: https://github.com/rust-lang-nursery/chalk/blob/master/src/ir/mod.rs +[well-formedness-checks]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir/lowering.rs#L230-L232 +[ir-code]: https://github.com/rust-lang-nursery/chalk/blob/master/src/ir.rs [HIR]: hir.html -[binders-struct]: https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/ir/mod.rs#L661 -[rules-environment]: https://github.com/rust-lang-nursery/chalk/blob/17abbabe53c2f78b04af04a9bc9e8a0e3fc676e3/src/rules/mod.rs#L9 +[binders-struct]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L661 +[rules-environment]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/rules.rs#L9 [slg]: traits-slg.html From 359e814c8c46cb917e318b753b2ecfc0cb98f6cb Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 8 May 2018 12:40:43 -0500 Subject: [PATCH 214/648] reword HIR intro. Fix #116 --- src/hir.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/src/hir.md b/src/hir.md index 3d2fbede3..51296e55e 100644 --- a/src/hir.md +++ b/src/hir.md @@ -1,11 +1,12 @@ # The HIR -The HIR – "High-level IR" – is the primary IR used in most of rustc. -It is a desugared version of the "abstract syntax tree" (AST) that is generated -after parsing, macro expansion, and name resolution have completed. Many parts -of HIR resemble Rust surface syntax quite closely, with the exception that some -of Rust's expression forms have been desugared away (as an example, `for` loops -are converted into a `loop` and do not appear in the HIR). +The HIR – "High-level IR" – is the primary IR used in most of rustc. It is a +compiler-friendly representation of the abstract syntax tree (AST) that is +generated after parsing, macro expansion, and name resolution have completed. +Many parts of HIR resemble Rust surface syntax quite closely, with the +exception that some of Rust's expression forms have been desugared away. For +example, `for` loops are converted into a `loop` and do not appear in the HIR. +This makes HIR more amenable to analysis than a normal AST. This chapter covers the main concepts of the HIR. From b6f63df3eb956af20b00c520697301a75046285e Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sat, 12 May 2018 15:01:58 -0500 Subject: [PATCH 215/648] remove have completed --- src/hir.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/hir.md b/src/hir.md index 51296e55e..06ee5e105 100644 --- a/src/hir.md +++ b/src/hir.md @@ -2,7 +2,7 @@ The HIR – "High-level IR" – is the primary IR used in most of rustc. It is a compiler-friendly representation of the abstract syntax tree (AST) that is -generated after parsing, macro expansion, and name resolution have completed. +generated after parsing, macro expansion, and name resolution. Many parts of HIR resemble Rust surface syntax quite closely, with the exception that some of Rust's expression forms have been desugared away. For example, `for` loops are converted into a `loop` and do not appear in the HIR. From 5f58b0dc98c78f31f8b692543f0c798995034b51 Mon Sep 17 00:00:00 2001 From: Sunjay Varma Date: Mon, 14 May 2018 10:52:21 -0700 Subject: [PATCH 216/648] Changes from review --- src/chalk-overview.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/src/chalk-overview.md b/src/chalk-overview.md index 7cea6acef..cd2c98b88 100644 --- a/src/chalk-overview.md +++ b/src/chalk-overview.md @@ -1,19 +1,20 @@ # An Overview of Chalk > Chalk is under heavy development, so if any of these links are broken or if -> any of the information is inconsistent with the code or outdated, please open -> an issue so we can fix it. If you are able to fix the issue yourself, we would +> any of the information is inconsistent with the code or outdated, please +> [open an issue][rustc-issues] so we can fix it. If you are able to fix the issue yourself, we would > love your contribution! [Chalk][chalk] recasts Rust's trait system explicitly in terms of logic programming by "lowering" Rust code into a kind of logic program we can then -execute queries against. Its goal is to be an executable, highly readable -specification of the Rust trait system.[^negativechalk] +execute queries against. (See [*Lowering to Logic*][lowering-to-logic] and +[*Lowering Rules*][lowering-rules]) Its goal is to be an executable, highly +readable specification of the Rust trait system. There are many expected benefits from this work. It will consolidate our existing, somewhat ad-hoc implementation into something far more principled and expressive, which should behave better in corner cases, and be much easier to -extend.[^negativechalk] +extend. ## Resources @@ -124,9 +125,10 @@ is the function that is ultimately called. See [The SLG Solver][slg]. -[^negativechalk]: [*Negative reasoning in Chalk* by Aaron Turon](http://aturon.github.io/blog/2017/04/24/negative-chalk/) - +[rustc-issues]: https://github.com/rust-lang-nursery/rustc-guide/issues [chalk]: https://github.com/rust-lang-nursery/chalk +[lowering-to-logic]: traits-lowering-to-logic.html +[lowering-rules]: traits-lowering-rules.html [ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree [chalk-ast]: https://github.com/rust-lang-nursery/chalk/blob/master/chalk-parse/src/ast.rs [universal quantification]: https://en.wikipedia.org/wiki/Universal_quantification From edddf9fa3b606526cf059e52e64c64f2ce23c590 Mon Sep 17 00:00:00 2001 From: Sunjay Varma Date: Mon, 14 May 2018 11:18:14 -0700 Subject: [PATCH 217/648] More review changes --- src/chalk-overview.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/src/chalk-overview.md b/src/chalk-overview.md index cd2c98b88..926c7f5ac 100644 --- a/src/chalk-overview.md +++ b/src/chalk-overview.md @@ -2,8 +2,8 @@ > Chalk is under heavy development, so if any of these links are broken or if > any of the information is inconsistent with the code or outdated, please -> [open an issue][rustc-issues] so we can fix it. If you are able to fix the issue yourself, we would -> love your contribution! +> [open an issue][rustc-issues] so we can fix it. If you are able to fix the +> issue yourself, we would love your contribution! [Chalk][chalk] recasts Rust's trait system explicitly in terms of logic programming by "lowering" Rust code into a kind of logic program we can then @@ -42,7 +42,7 @@ Rust-like syntax. The parser takes that syntax and produces an [Abstract Syntax Tree (AST)][ast]. You can find the [complete definition of the AST][chalk-ast] in the source code. -The syntax contains things from Rust that we know and love for example traits, +The syntax contains things from Rust that we know and love, for example: traits, impls, and struct definitions. Parsing is often the first "phase" of transformation that a program goes through in order to become a format that chalk can understand. @@ -67,14 +67,14 @@ essentially one of the following: * `forall { ... }` is represented in the code using the [`Binders` struct][binders-struct]. -This is the phase where we encode the rules of the trait system into logic. For -example, if we have: +Lowering is the phase where we encode the rules of the trait system into logic. +For example, if we have the following Rust: ```rust,ignore impl Clone for Vec {} ``` -We generate: +We generate the following program clause: ```rust,ignore forall { (Vec: Clone) :- (T: Clone) } @@ -102,22 +102,23 @@ For example, if you have a type like `Foo`, we would represent `Foo` as a string in the AST but in `ir::Program`, we use numeric indices (`ItemId`). In addition to `ir::Program` which has "rust-like things", there is also -`ir::ProgramEnvironment` which is "pure logic". The main field in that is +`ir::ProgramEnvironment` which is "pure logic". The main field in that struct is `program_clauses` which contains the `ProgramClause`s that we generated previously. ## Rules The `rules` module works by iterating over every trait, impl, etc. and emitting -the rules that come from each one. The traits section of the rustc-guide (that -you are currently reading) contains the most up-to-date reference on that. +the rules that come from each one. See [Lowering Rules][lowering-rules] for the +most up-to-date reference on that. The `ir::ProgramEnvironment` is created [in this module][rules-environment]. ## Testing TODO: Basically, [there is a macro](https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/solve/test.rs#L112-L148) -that will take syntax and run it through the full pipeline described above. +that will take chalk's Rust-like syntax and run it through the full pipeline +described above. [This](https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/solve/test.rs#L83-L110) is the function that is ultimately called. From 7f83d68118ee2b48768691fcb2bd0d3388b0c41a Mon Sep 17 00:00:00 2001 From: Sunjay Varma Date: Mon, 14 May 2018 11:21:20 -0700 Subject: [PATCH 218/648] Even more review changes --- src/chalk-overview.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/chalk-overview.md b/src/chalk-overview.md index 926c7f5ac..02e094293 100644 --- a/src/chalk-overview.md +++ b/src/chalk-overview.md @@ -37,7 +37,8 @@ extend. Chalk is designed to be incorporated with the Rust compiler, so the syntax and concepts it deals with heavily borrow from Rust. It is convenient for the sake of testing to be able to run chalk on its own, so chalk includes a parser for a -Rust-like syntax. +Rust-like syntax. This syntax is orthogonal to the Rust AST and grammar. It is +not intended to look exactly like it or support the exact same syntax. The parser takes that syntax and produces an [Abstract Syntax Tree (AST)][ast]. You can find the [complete definition of the AST][chalk-ast] in the source code. From 389bd079a8c2c6032914b60790fbde220c93ebb4 Mon Sep 17 00:00:00 2001 From: Sunjay Varma Date: Mon, 14 May 2018 11:26:15 -0700 Subject: [PATCH 219/648] Missed a few things because of GitHub's UI --- src/chalk-overview.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/chalk-overview.md b/src/chalk-overview.md index 02e094293..76c119a5e 100644 --- a/src/chalk-overview.md +++ b/src/chalk-overview.md @@ -68,6 +68,8 @@ essentially one of the following: * `forall { ... }` is represented in the code using the [`Binders` struct][binders-struct]. +*See also: [Goals and Clauses][goals-and-clauses]* + Lowering is the phase where we encode the rules of the trait system into logic. For example, if we have the following Rust: @@ -137,6 +139,7 @@ See [The SLG Solver][slg]. [lowering-forall]: https://rust-lang-nursery.github.io/rustc-guide/traits-lowering-to-logic.html#type-checking-generic-functions-beyond-horn-clauses [programclause]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L721 [clause]: https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md#clause +[goals-and-clauses]: traits-goals-and-clauses.html [well-formedness-checks]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir/lowering.rs#L230-L232 [ir-code]: https://github.com/rust-lang-nursery/chalk/blob/master/src/ir.rs [HIR]: hir.html From 2dc19d097a9f5a8ab0a72ba79b0440cf6baa706f Mon Sep 17 00:00:00 2001 From: Takanori Ishibashi Date: Tue, 15 May 2018 21:59:30 +0900 Subject: [PATCH 220/648] invokations -> invocations --- src/macro-expansion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/macro-expansion.md b/src/macro-expansion.md index ba807faf2..a90ad517b 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -2,7 +2,7 @@ Macro expansion happens during parsing. `rustc` has two parsers, in fact: the normal Rust parser, and the macro parser. During the parsing phase, the normal -Rust parser will set aside the contents of macros and their invokations. Later, +Rust parser will set aside the contents of macros and their invocations. Later, before name resolution, macros are expanded using these portions of the code. The macro parser, in turn, may call the normal Rust parser when it needs to bind a metavariable (e.g. `$my_expr`) while parsing the contents of a macro From 2bd5354411251de6512b025f8f24b56111b56719 Mon Sep 17 00:00:00 2001 From: Alex Kitchens Date: Thu, 17 May 2018 12:53:11 -0500 Subject: [PATCH 221/648] Define HIR more specifically IR is a foreign acronym to me, so having it fully expressed in the beginning as Intermediate Representation helps me comprehend the subject. --- src/high-level-overview.md | 5 +++-- src/hir.md | 10 +++++----- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/src/high-level-overview.md b/src/high-level-overview.md index 9f3b63a54..be396054b 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -104,8 +104,8 @@ take: nodes, and hence may strip things out of the AST as well. 3. **Lowering to HIR** - Once name resolution completes, we convert the AST into the HIR, - or "high-level IR". The HIR is defined in `src/librustc/hir/`; - that module also includes the lowering code. + or "[high-level intermediate representation]". The HIR is defined in + `src/librustc/hir/`; that module also includes the lowering code. - The HIR is a lightly desugared variant of the AST. It is more processed than the AST and more suitable for the analyses that follow. It is **not** required to match the syntax of the Rust language. @@ -138,3 +138,4 @@ take: [query model]: query.html +[high-level intermediate representation]: hir.html diff --git a/src/hir.md b/src/hir.md index 06ee5e105..44c4968b1 100644 --- a/src/hir.md +++ b/src/hir.md @@ -1,10 +1,10 @@ # The HIR -The HIR – "High-level IR" – is the primary IR used in most of rustc. It is a -compiler-friendly representation of the abstract syntax tree (AST) that is -generated after parsing, macro expansion, and name resolution. -Many parts of HIR resemble Rust surface syntax quite closely, with the -exception that some of Rust's expression forms have been desugared away. For +The HIR – "High-Level Intermediate Representation" – is the primary IR used in +most of rustc. It is a compiler-friendly representation of the abstract syntax +tree (AST) that is generated after parsing, macro expansion, and name +resolution. Many parts of HIR resemble Rust surface syntax quite closely, with +the exception that some of Rust's expression forms have been desugared away. For example, `for` loops are converted into a `loop` and do not appear in the HIR. This makes HIR more amenable to analysis than a normal AST. From f91de90ea2bf253765de82ebc233d048c592013a Mon Sep 17 00:00:00 2001 From: Philipp Hansch Date: Mon, 21 May 2018 13:16:16 +0200 Subject: [PATCH 222/648] Add type inference example This should make the chapter a bit more approachable, as it doesn't start with a reference to the HM type inference algorithm. --- src/type-inference.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/src/type-inference.md b/src/type-inference.md index e480b2b56..4795e23c1 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -1,5 +1,21 @@ # Type inference +Type inference is the process of automatic detection of the type of an +expression. + +It is what allows Rust to work with fewer or no type annotations, +making things easier for users: + +```rust,ignore +fn main() { + let mut things = vec![]; + things.push("thing") +} +``` + +Here, `things` is *inferenced* to be `&str` because that's the value we push +into `things`. + The type inference is based on the standard Hindley-Milner (HM) type inference algorithm, but extended in various way to accommodate subtyping, region inference, and higher-ranked types. @@ -173,7 +189,7 @@ mechanism. You'll have to try again when more details about `?T` or ## Region constraints -Regions are inferred somewhat differently from types. Rather than +Regions are inferenced somewhat differently from types. Rather than eagerly unifying things, we simply collect constraints as we go, but make (almost) no attempt to solve regions. These constraints have the form of an "outlives" constraint: @@ -189,7 +205,7 @@ idea: 'b <= 'a ``` -(There are various other kinds of constriants, such as "verifys"; see +(There are various other kinds of constraints, such as "verifys"; see the `region_constraints` module for details.) There is one case where we do some amount of eager unification. If you have an From 6315267608a7a611228acb79a15dee214908de59 Mon Sep 17 00:00:00 2001 From: Philipp Hansch Date: Mon, 21 May 2018 19:44:07 +0200 Subject: [PATCH 223/648] The type is inferenced, not things itself. --- src/type-inference.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/type-inference.md b/src/type-inference.md index 4795e23c1..9ae88decd 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -13,8 +13,8 @@ fn main() { } ``` -Here, `things` is *inferenced* to be `&str` because that's the value we push -into `things`. +Here, the type of `things` is *inferenced* to be `&str` because that's the value +we push into `things`. The type inference is based on the standard Hindley-Milner (HM) type inference algorithm, but extended in various way to accommodate subtyping, region From 3b0d3f27a5548c4405d189fd4e54cd78bde5e0bc Mon Sep 17 00:00:00 2001 From: Alex Kitchens Date: Tue, 22 May 2018 16:30:05 -0500 Subject: [PATCH 224/648] Define a Cycle I wasn't clear on what a Cycle was when reading through the document. Defining it will be helpful for other readers not familiar with it as well. --- src/query.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/query.md b/src/query.md index 2c518ee55..d48b6e7e3 100644 --- a/src/query.md +++ b/src/query.md @@ -47,6 +47,9 @@ let ty = tcx.type_of(some_def_id); ### Cycles between queries +A cycle is when a query becomes stuck in a loop e.g. query A generates query B +which generates query A again. + Currently, cycles during query execution should always result in a compilation error. Typically, they arise because of illegal programs that contain cyclic references they shouldn't (though sometimes they From 3d48815cf4b899a48a70365de977721dbc30a51f Mon Sep 17 00:00:00 2001 From: Oliver Schneider Date: Wed, 23 May 2018 11:55:16 +0200 Subject: [PATCH 225/648] Fixes #141 --- src/tests/adding.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/src/tests/adding.md b/src/tests/adding.md index 96fc0cf84..16e4982b0 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -266,13 +266,16 @@ you can even run the resulting program. Just add one of the following ### Editing and updating the reference files If you have changed the compiler's output intentionally, or you are -making a new test, you can use the script `ui/update-references.sh` to -update the references. When you run the test framework, it will report -various errors: in those errors is a command you can use to run the -`ui/update-references.sh` script, which will then copy over the files -from the build directory and use them as the new reference. You can -also just run `ui/update-all-references.sh`. In both cases, you can run -the script with `--help` to get a help message. +making a new test, you can pass `--bless` to the test subcommand. E.g. +if some tests in `src/test/ui` are failing, you can run + +``` +./x.py test --stage 1 src/test/ui --bless +``` + +to automatically adjust the `.stderr`, `.stdout` or `.fixed` files of +all tests. Of course you can also target just specific tests with the +`--test-args your_test_name` flag, just like when running the tests. ### Normalization From 50f44216f4082cb66c76cc636b400faa2dc212cf Mon Sep 17 00:00:00 2001 From: Takanori Ishibashi Date: Sat, 26 May 2018 16:36:27 +0900 Subject: [PATCH 226/648] Fix typo --- src/macro-expansion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/macro-expansion.md b/src/macro-expansion.md index a90ad517b..df0dc7b1d 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -89,7 +89,7 @@ against the pattern `ms`. Using our examples, `tts` could be the stream of tokens containing the inside of the example invocation `print foo`, while `ms` might be the sequence of token (trees) `print $mvar:ident`. -The output of the parser is a `NamedParserResult`, which indicates which of +The output of the parser is a `NamedParseResult`, which indicates which of three cases has occured: - Success: `tts` matches the given matcher `ms`, and we have produced a binding From 2debb43b5a8e8e1a3feec20d05b88b2de0210284 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Fri, 11 May 2018 21:06:21 -0500 Subject: [PATCH 227/648] Clarify language in Trait Resolution --- src/trait-resolution.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/trait-resolution.md b/src/trait-resolution.md index 5bf8f8716..b9fa81bd6 100644 --- a/src/trait-resolution.md +++ b/src/trait-resolution.md @@ -202,8 +202,8 @@ impl Get for Box { What happens when we invoke `get_it(&Box::new(1_u16))`, for example? In this case, the `Self` type is `Box` – that unifies with both impls, -because the first applies to all types, and the second to all -boxes. In order for this to be unambiguous, the compiler does a *winnowing* +because the first applies to all types `T`, and the second to all +`Box`. In order for this to be unambiguous, the compiler does a *winnowing* pass that considers `where` clauses and attempts to remove candidates. In this case, the first impl only applies if `Box : Copy`, which doesn't hold. After winnowing, @@ -242,7 +242,7 @@ fn foo(x: X) { In the body of `foo`, clearly we can use methods of `A1`, `A2`, or `B` on variable `x`. The line marked `(*)` will incur an obligation `X: A1`, -which the line marked `(#)` will incur an obligation `X: B`. Meanwhile, +while the line marked `(#)` will incur an obligation `X: B`. Meanwhile, the parameter environment will contain two where-clauses: `X : A2` and `X : B`. For each obligation, then, we search this list of where-clauses. The obligation `X: B` trivially matches against the where-clause `X: B`. From bb867865fe48a8ff2bcce9c1d85683f2836a71eb Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Fri, 25 May 2018 23:03:09 -0400 Subject: [PATCH 228/648] Trait logic: Explain what each domain goal means --- src/traits-goals-and-clauses.md | 95 +++++++++++++++++++++++++-------- src/traits-lowering-rules.md | 2 +- 2 files changed, 73 insertions(+), 24 deletions(-) diff --git a/src/traits-goals-and-clauses.md b/src/traits-goals-and-clauses.md index 0cbdb7077..882f86b8e 100644 --- a/src/traits-goals-and-clauses.md +++ b/src/traits-goals-and-clauses.md @@ -67,7 +67,7 @@ its inputs P0..Pm: Projection = >::AssocItem ``` -Given that, we can define a `DomainGoal` as follows: +Given these, we can define a `DomainGoal` as follows: ```text DomainGoal = Implemented(TraitRef) @@ -78,33 +78,82 @@ DomainGoal = Implemented(TraitRef) | WellFormed(Type) | WellFormed(TraitRef) | WellFormed(Projection = Type) - | Outlives(Type, Region) - | Outlives(Region, Region) + | Outlives(Type: Region) + | Outlives(Region: Region) ``` -- `Implemented(TraitRef)` -- true if the given trait is - implemented for the given input types and lifetimes -- `FromEnv(TraitEnv)` -- true if the given trait is *assumed* to be implemented; - that is, if it can be derived from the in-scope where clauses - - as we'll see in the section on lowering, `FromEnv(X)` implies - `Implemented(X)` but not vice versa. This distinction is crucial - to [implied bounds]. -- `ProjectionEq(Projection = Type)` -- the given associated type `Projection` - is equal to `Type`; see [the section on associated - types](./traits-associated-types.html) - - in general, proving `ProjectionEq(TraitRef::Item = Type)` also - requires proving `Implemented(TraitRef)` -- `Normalize(Projection -> Type)` -- the given associated type `Projection` can - be [normalized][n] to `Type` - - as discussed in [the section on associated - types](./traits-associated-types.html), - `Normalize` implies `ProjectionEq` but not vice versa -- `WellFormed(..)` -- these goals imply that the given item is - *well-formed* - - well-formedness is important to [implied bounds]. +Let's break down each one of these, one-by-one. + +#### Implemented(TraitRef) +e.g. `Implemented(i32: Copy)` + +True if the given trait is implemented for the given input types and lifetimes. + +#### ProjectionEq(Projection = Type) +e.g. `ProjectionEq::Item = u8` + +The given associated type `Projection` is equal to `Type`; this can be proved +with either normalization or using skolemized types. See [the section +on associated types](./traits-associated-types.html). + +#### Normalize(Projection -> Type) +e.g. `ProjectionEq::Item -> u8` + +The given associated type `Projection` can be [normalized][n] to `Type`. + +As discussed in [the section on associated +types](./traits-associated-types.html), `Normalize` implies `ProjectionEq`, +but not vice versa. In general, proving `Normalize(::Item -> U)` +also requires proving `Implemented(T: Trait)`. [n]: ./traits-associated-types.html#normalize +#### FromEnv(TraitRef), FromEnv(Projection = Type) +e.g. `FromEnv(Self: Add)` + +e.g. `FromEnv(::Item<'a> = &'a [u8])` + +True if the inner `TraitRef` or projection equality is *assumed* to be true; +that is, if it can be derived from the in-scope where clauses. + +For example, given the following function: + +```rust +fn loud_clone(stuff: &T) -> T { + println!("cloning!"); + stuff.clone() +} +``` + +Inside the body of our function, we would have `FromEnv(T: Clone)`. In-scope +where clauses nest, so a function body inside an impl body inherits the +impl body's where clauses, too. + +This and the next rule are used to implement [implied bounds]. As we'll see +in the section on lowering, `FromEnv(X)` implies `Implemented(X)`, but not +vice versa. This distinction is crucial to implied bounds. + +#### WellFormed(Item) +These goals imply that the given item is *well-formed*. + +We can talk about different types of items being well-formed: + +**Types**, like `WellFormed(Vec)`, which is true in Rust, or + `WellFormed(Vec)`, which is not (because `str` is not `Sized`.) + +**TraitRefs**, like `WellFormed(Vec: Clone)`. + +**Projections**, like `WellFormed(T: Iterator)`. + +Well-formedness is important to [implied bounds]. In particular, the reason +it is okay to assume `FromEnv(T: Clone)` in the example above is that we +_also_ verify `WellFormed(T: Clone)` for each call site of `loud_clone`. + +#### Outlives(Type: Region), Outlives(Region: Region) +e.g. `Outlives(&'a str: 'b)`, `Outlives('a: 'static)` + +True if the given type or region on the left outlives the right-hand region. + ## Coinductive goals diff --git a/src/traits-lowering-rules.md b/src/traits-lowering-rules.md index 73660c42c..80884cbdd 100644 --- a/src/traits-lowering-rules.md +++ b/src/traits-lowering-rules.md @@ -113,7 +113,7 @@ forall { ``` This clause says that if we are assuming that the trait holds, then we can also -assume that it's where-clauses hold. It's perhaps useful to see an example: +assume that its where-clauses hold. It's perhaps useful to see an example: ```rust,ignore trait Eq: PartialEq { ... } From 5106793010757c9fe7eae9a3b8ea8788bdaeb5f6 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 22 May 2018 13:40:37 -0500 Subject: [PATCH 229/648] Add info about emitting lints and errors --- src/SUMMARY.md | 1 + src/appendix-code-index.md | 3 + src/diag.md | 176 +++++++++++++++++++++++++++++++++++++ 3 files changed, 180 insertions(+) create mode 100644 src/diag.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 4923d4972..2db3e62c3 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -54,6 +54,7 @@ - [miri const evaluator](./miri.md) - [Parameter Environments](./param_env.md) - [Generating LLVM IR](./trans.md) +- [Emitting Diagnostics](./diag.md) --- diff --git a/src/appendix-code-index.md b/src/appendix-code-index.md index 62edd0f5b..915d00828 100644 --- a/src/appendix-code-index.md +++ b/src/appendix-code-index.md @@ -8,11 +8,13 @@ Item | Kind | Short description | Chapter | ----------------|----------|-----------------------------|--------------------|------------------- `CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser] | [src/libsyntax/codemap.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html) `CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver] | [src/librustc_driver/driver.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/struct.CompileState.html) +`DiagnosticBuilder` | struct | A struct for building up compiler diagnostics, such as errors or lints | [Emitting Diagnostics] | [src/librustc_errors/diagnostic_builder.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.DiagnosticBuilder.html) `DocContext` | struct | A state container used by rustdoc when crawling through a crate to gather its documentation | [Rustdoc] | [src/librustdoc/core.rs](https://github.com/rust-lang/rust/blob/master/src/librustdoc/core.rs) `ast::Crate` | struct | Syntax-level representation of a parsed crate | [The parser] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Crate.html) `hir::Crate` | struct | More abstract, compiler-friendly form of a crate's AST | [The Hir] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Crate.html) `ParseSess` | struct | This struct contains information about a parsing session | [the Parser] | [src/libsyntax/parse/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/struct.ParseSess.html) `Session` | struct | The data associated with a compilation session | [the Parser], [The Rustc Driver] | [src/librustc/session/mod.html](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html) +`Span` | struct | A location in the user's source code, used for error reporting primarily | [Emitting Diagnostics] | [src/libsyntax_pos/span_encoding.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax_pos/struct.Span.html) `StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser] | [src/libsyntax/parse/lexer/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/lexer/struct.StringReader.html) `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules] | [src/librustc/ty/trait_def.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/trait_def/struct.TraitDef.html) `Ty<'tcx>` | struct | This is the internal representation of a type used for type checking | [Type checking] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/type.Ty.html) @@ -24,3 +26,4 @@ Item | Kind | Short description | Chapter | [Type checking]: type-checking.html [The `ty` modules]: ty.html [Rustdoc]: rustdoc.html +[Emitting Diagnostics]: diag.html diff --git a/src/diag.md b/src/diag.md new file mode 100644 index 000000000..25f45e002 --- /dev/null +++ b/src/diag.md @@ -0,0 +1,176 @@ +# Emitting Diagnostics + +A lot of effort has been put into making `rustc` have great error messages. +This chapter is about how to emit compile errors and lints from the compiler. + +## `Span` + +`Span` is the primary data structure in `rustc` used to represent a location in +the code being compiled. `Span`s are attached to most constructs in HIR and MIR, +allowing for easier error reporting whenever an error comes up. + +A `Span` can be looked up in a `CodeMap` to get a "snippet" useful for +displaying errors with [`span_to_snippet` and other similar methods][sptosnip] +on the `CodeMap`. + +[sptosnip]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html#method.span_to_snippet + +## Error messages + +The [`rustc_errors`][errors] crate defines most of the utilities used for +reporting errors. + +[errors]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html + +Most "session"-like types in the compiler (e.g. [`Session`][session]) have +methods (or fields with methods) that allow reporting errors. These methods +usually have names like `span_err` or `struct_span_err` or `span_warn`, etc... +There are lots of them; they emit different types of "errors", such as +warnings, errors, fatal errors, suggestions, etc. + +[session]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html + +In general, there are two class of such methods: ones that emit an error +directly and ones that allow finer control over what to emit. For example, +[`span_err`][spanerr] emits the given error message at the given `Span`, but +[`struct_span_err`][strspanerr] instead returns a [`DiagnosticBuilder`][diagbuild]. + +`DiagnosticBuilder` allows you to add related notes and suggestions to an error +before emitting it by calling the [`emit`][emit] method. See the +[docs][diagbuild] for more info on what you can do. + +[spanerr]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html#method.span_err +[strspanerr]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html#method.struct_span_err +[diagbuild]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/diagnostic_builder/struct.DiagnosticBuilder.html +[emit]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/diagnostic_builder/struct.DiagnosticBuilder.html#method.emit + +For example, to add a help message to an error, one might do: + +```rust,ignore +let snip = sess.codemap().span_to_snippet(sp); + +sess.struct_span_err(sp, "oh no! this is an error!") + .span_suggestion(other_sp, "try using a qux here", format!("qux {}", snip)) + .emit(); +``` + +This might emit an error like + +```console +$ rustc mycode.rs +error[E0999]: oh no! this is an error! + --> mycode.rs:3:5 + | +3 | sad() + | ^ help: try using a qux here: `qux sad()` + +error: aborting due to previous error + +For more information about this error, try `rustc --explain E0999`. +``` + +## Lints + +The compiler linting infrastructure is defined in the [`rustc::lint`][rlint] +module. + +[rlint]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/lint/index.html + +### Declaring a lint + +The built-in compiler lints are defined in the [`rustc_lint`][builtin] +crate. + +[builtin]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lint/index.html + +Each lint is defined as a `struct` that implements the `LintPass` `trait`. The +trait implementation allows you to check certain syntactic constructs the +linter walks the source code. You can then choose to emit lints in a very +similar way to compile errors. Finally, you register the lint to actually get +it to be run by the compiler by using the `declare_lint!` macro. + +For example, the following lint checks for uses +of `while true { ... }` and suggests using `loop { ... }` instead. + +```rust,ignore +// Declare a lint called `WHILE_TRUE` +declare_lint! { + WHILE_TRUE, + + // warn-by-default + Warn, + + // This string is the lint description + "suggest using `loop { }` instead of `while true { }`" +} + +// Define a struct and `impl LintPass` for it. +#[derive(Copy, Clone)] +pub struct WhileTrue; + +impl LintPass for WhileTrue { + fn get_lints(&self) -> LintArray { + lint_array!(WHILE_TRUE) + } +} + +// LateLintPass has lots of methods. We only override the definition of +// `check_expr` for this lint because that's all we need, but you could +// override other methods for your own lint. See the rustc docs for a full +// list of methods. +impl<'a, 'tcx> LateLintPass<'a, 'tcx> for WhileTrue { + fn check_expr(&mut self, cx: &LateContext, e: &hir::Expr) { + if let hir::ExprWhile(ref cond, ..) = e.node { + if let hir::ExprLit(ref lit) = cond.node { + if let ast::LitKind::Bool(true) = lit.node { + if lit.span.ctxt() == SyntaxContext::empty() { + let msg = "denote infinite loops with `loop { ... }`"; + let condition_span = cx.tcx.sess.codemap().def_span(e.span); + let mut err = cx.struct_span_lint(WHILE_TRUE, condition_span, msg); + err.span_suggestion_short(condition_span, "use `loop`", "loop".to_owned()); + err.emit(); + } + } + } + } + } +} +``` + +### Edition Lints + +Sometimes we want to change the behavior of a lint in a new edition. To do this, +we just add the transition to our invocation of `declare_lint!`: + +```rust,ignore +declare_lint! { + pub ANONYMOUS_PARAMETERS, + Allow, + "detects anonymous parameters", + Edition::Edition2018 => Warn, +} +``` + +This makes the `ANONYMOUS_PARAMETERS` lint allow-by-default in the 2015 edition +but warn-by-default in the 2018 edition. + +### Lint Groups + +Lints can be turned on in groups. These groups are declared in the +[`register_builtins`][rbuiltins] function in [`rustc_lint::lib`][builtin]. The +`add_lint_group!` macro is used to declare a new group. + +[rbuiltins]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lint/fn.register_builtins.html + +For example, + +```rust,ignore + add_lint_group!(sess, + "nonstandard_style", + NON_CAMEL_CASE_TYPES, + NON_SNAKE_CASE, + NON_UPPER_CASE_GLOBALS); +``` + +This defines the `nonstandard_style` group which turns on the listed lints. A user +can turn on these lints by using `!#[warn(nonstandard_style)]`. From c6ecc1f9b359623b18a57c8f6fe9f324295ad886 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 22 May 2018 13:46:33 -0500 Subject: [PATCH 230/648] 80 chars --- src/diag.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/diag.md b/src/diag.md index 25f45e002..6679e4249 100644 --- a/src/diag.md +++ b/src/diag.md @@ -33,7 +33,8 @@ warnings, errors, fatal errors, suggestions, etc. In general, there are two class of such methods: ones that emit an error directly and ones that allow finer control over what to emit. For example, [`span_err`][spanerr] emits the given error message at the given `Span`, but -[`struct_span_err`][strspanerr] instead returns a [`DiagnosticBuilder`][diagbuild]. +[`struct_span_err`][strspanerr] instead returns a +[`DiagnosticBuilder`][diagbuild]. `DiagnosticBuilder` allows you to add related notes and suggestions to an error before emitting it by calling the [`emit`][emit] method. See the @@ -172,5 +173,5 @@ For example, NON_UPPER_CASE_GLOBALS); ``` -This defines the `nonstandard_style` group which turns on the listed lints. A user -can turn on these lints by using `!#[warn(nonstandard_style)]`. +This defines the `nonstandard_style` group which turns on the listed lints. A +user can turn on these lints by using `!#[warn(nonstandard_style)]`. From 8a79f8b881287c923341c4a57f5c8ff99cc36217 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 22 May 2018 14:55:55 -0500 Subject: [PATCH 231/648] Address reviewers' comments --- src/diag.md | 86 +++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 77 insertions(+), 9 deletions(-) diff --git a/src/diag.md b/src/diag.md index 6679e4249..a2ec3bb7e 100644 --- a/src/diag.md +++ b/src/diag.md @@ -22,12 +22,13 @@ reporting errors. [errors]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html -Most "session"-like types in the compiler (e.g. [`Session`][session]) have +[`Session`][session] and [`ParseSess`][parsesses] have methods (or fields with methods) that allow reporting errors. These methods usually have names like `span_err` or `struct_span_err` or `span_warn`, etc... There are lots of them; they emit different types of "errors", such as warnings, errors, fatal errors, suggestions, etc. +[parsesses]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/struct.ParseSess.html [session]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html In general, there are two class of such methods: ones that emit an error @@ -45,20 +46,64 @@ before emitting it by calling the [`emit`][emit] method. See the [diagbuild]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/diagnostic_builder/struct.DiagnosticBuilder.html [emit]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/diagnostic_builder/struct.DiagnosticBuilder.html#method.emit -For example, to add a help message to an error, one might do: +```rust,ignore +// Get a DiagnosticBuilder. This does _not_ emit an error yet. +let mut err = sess.struct_span_err(sp, "oh no! this is an error!"); + +// In some cases, you might need to check if `sp` is generated by a macro to +// avoid printing weird errors about macro-generated code. + +if let Some(snippet) = sess.codemap().span_to_snippet(sp) { + // Use the snippet to generate a suggested fix + err.span_suggestion(suggestion_sp, "try using a qux here", format!("qux {}", snip)); +} else { + // If we weren't able to generate a snippet, then emit a "help" message + // instead of a concrete "suggestion". In practice this is unlikely to be + // reached. + err.span_help(suggestion_sp, "you could use a qux here instead"); +} + +// emit the error +err.emit(); +``` + +## Suggestions + +We would like to make edition transitions as smooth as possible. To that end, +`rustfix` can use compiler suggestions to automatically fix code. For example, +we could use `rustfix` to mechanically apply the `qux` suggestion from the +previous example. However, not all suggestions are mechanically applicable. We +use the [`span_suggestion_with_applicability`][sswa] method of +`DiagnosticBuilder` to inform the emitter of whether a suggestion is +mechanically applicable or not. This information, in turn, is outputed by +rustc when the error format is `json`, which is used by `rustfix`. + +[sswa]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.DiagnosticBuilder.html#method.span_suggestion_with_applicability + +For example, to make our `qux` suggestion machine-applicable, we would do: ```rust,ignore -let snip = sess.codemap().span_to_snippet(sp); +let mut err = sess.struct_span_err(sp, "oh no! this is an error!"); + +if let Some(snippet) = sess.codemap().span_to_snippet(sp) { + // Add applicability info! + err.span_suggestion_with_applicability( + suggestion_sp, + "try using a qux here", + format!("qux {}", snip), + Applicability::MachineApplicable, + ); +} else { + err.span_help(suggestion_sp, "you could use a qux here instead"); +} -sess.struct_span_err(sp, "oh no! this is an error!") - .span_suggestion(other_sp, "try using a qux here", format!("qux {}", snip)) - .emit(); +err.emit(); ``` This might emit an error like ```console -$ rustc mycode.rs +$ rustc mycode.rs error[E0999]: oh no! this is an error! --> mycode.rs:3:5 | @@ -70,10 +115,29 @@ error: aborting due to previous error For more information about this error, try `rustc --explain E0999`. ``` +In some cases, like when the suggestion spans multiple lines or when there are +multiple suggestions, the suggestions are displayed on their own: + +```console +error[E0999]: oh no! this is an error! + --> mycode.rs:3:5 + | +3 | sad() + | ^ +help: try using a qux here: + | +3 | qux sad() + | ^^^ + +error: aborting due to previous error + +For more information about this error, try `rustc --explain E0999`. +``` + ## Lints The compiler linting infrastructure is defined in the [`rustc::lint`][rlint] -module. +module. [rlint]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/lint/index.html @@ -138,7 +202,7 @@ impl<'a, 'tcx> LateLintPass<'a, 'tcx> for WhileTrue { } ``` -### Edition Lints +### Edition-gated Lints Sometimes we want to change the behavior of a lint in a new edition. To do this, we just add the transition to our invocation of `declare_lint!`: @@ -155,6 +219,10 @@ declare_lint! { This makes the `ANONYMOUS_PARAMETERS` lint allow-by-default in the 2015 edition but warn-by-default in the 2018 edition. +Lints that represent an incompatibility (i.e. error) in the upcoming edition should +also be registered as `FutureIncompatibilityLint`s in +[`register_builtins`][rbuiltins] function in [`rustc_lint::lib`][builtin]. + ### Lint Groups Lints can be turned on in groups. These groups are declared in the From 1624ba9923ef4ea161c8d1f8c51f1486040b83fe Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 22 May 2018 15:00:27 -0500 Subject: [PATCH 232/648] add more on applicabilities --- src/diag.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/diag.md b/src/diag.md index a2ec3bb7e..8b067fde7 100644 --- a/src/diag.md +++ b/src/diag.md @@ -134,6 +134,18 @@ error: aborting due to previous error For more information about this error, try `rustc --explain E0999`. ``` +There are a few other [`Applicability`][appl] possibilities: + +- `MachineApplicable`: Can be applied mechanically. +- `HasPlaceholders`: Cannot be applied mechanically and has placeholder text in + the suggestions. For example, "Try adding a type: \`let x: \`". +- `MaybeIncorrect`: Cannot be applied mechanically because the suggestion may + or may not be a good one. +- `Unspecified`: Cannot be applied mechanically because we don't know which + of the above cases it falls into. + +[appl]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/enum.Applicability.html + ## Lints The compiler linting infrastructure is defined in the [`rustc::lint`][rlint] From 3ea3ff8c70d5329043f42387baed707de06b4bf8 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 22 May 2018 15:01:04 -0500 Subject: [PATCH 233/648] line length --- src/diag.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/diag.md b/src/diag.md index 8b067fde7..af619ed83 100644 --- a/src/diag.md +++ b/src/diag.md @@ -231,8 +231,8 @@ declare_lint! { This makes the `ANONYMOUS_PARAMETERS` lint allow-by-default in the 2015 edition but warn-by-default in the 2018 edition. -Lints that represent an incompatibility (i.e. error) in the upcoming edition should -also be registered as `FutureIncompatibilityLint`s in +Lints that represent an incompatibility (i.e. error) in the upcoming edition +should also be registered as `FutureIncompatibilityLint`s in [`register_builtins`][rbuiltins] function in [`rustc_lint::lib`][builtin]. ### Lint Groups From dbac00dedacc2988bcb8c19a88bbaece4097c3bb Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 22 May 2018 15:05:57 -0500 Subject: [PATCH 234/648] fix typo --- src/diag.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/diag.md b/src/diag.md index af619ed83..6043a79b7 100644 --- a/src/diag.md +++ b/src/diag.md @@ -138,7 +138,7 @@ There are a few other [`Applicability`][appl] possibilities: - `MachineApplicable`: Can be applied mechanically. - `HasPlaceholders`: Cannot be applied mechanically and has placeholder text in - the suggestions. For example, "Try adding a type: \`let x: \`". + the suggestions. For example, "Try adding a type: \`let x: \\`". - `MaybeIncorrect`: Cannot be applied mechanically because the suggestion may or may not be a good one. - `Unspecified`: Cannot be applied mechanically because we don't know which From 1a25691cb42005ce25c2b247fcbfe8d3e992c164 Mon Sep 17 00:00:00 2001 From: Oliver Schneider Date: Mon, 28 May 2018 13:08:15 +0200 Subject: [PATCH 235/648] Mention "run-rustfix" --- src/tests/adding.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/tests/adding.md b/src/tests/adding.md index 16e4982b0..e8075e539 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -148,6 +148,9 @@ exhaustive. Header commands can generally be found by browsing the `TestProps` structure found in [`header.rs`] from the compiletest source. +* `run-rustfix` for UI tests, indicates that the test produces + structured suggestions, which are then applied and the final + source is compiled again. * `min-{gdb,lldb}-version` * `min-llvm-version` * `compile-pass` for UI tests, indicates that the test is From 79ea4eeb149f549e5afbedbd4b30e95b8362f1a4 Mon Sep 17 00:00:00 2001 From: Oliver Schneider Date: Tue, 29 May 2018 15:37:24 +0200 Subject: [PATCH 236/648] Explain .fixed files --- src/tests/adding.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/tests/adding.md b/src/tests/adding.md index e8075e539..daaaace7e 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -150,7 +150,8 @@ source. * `run-rustfix` for UI tests, indicates that the test produces structured suggestions, which are then applied and the final - source is compiled again. + source is stored in a `.fixed` file and compiled again. The final + compilation is required to succeed. * `min-{gdb,lldb}-version` * `min-llvm-version` * `compile-pass` for UI tests, indicates that the test is From 2d387350e2bb2d22c7051965f466c915962e77d5 Mon Sep 17 00:00:00 2001 From: Oliver Schneider Date: Tue, 29 May 2018 16:31:32 +0200 Subject: [PATCH 237/648] Satisfy travis --- src/tests/adding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/tests/adding.md b/src/tests/adding.md index daaaace7e..3788e7c41 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -273,7 +273,7 @@ If you have changed the compiler's output intentionally, or you are making a new test, you can pass `--bless` to the test subcommand. E.g. if some tests in `src/test/ui` are failing, you can run -``` +```text ./x.py test --stage 1 src/test/ui --bless ``` From 8a98e397505dcfe57590a6018085cc9905d86973 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Wed, 30 May 2018 22:57:34 -0500 Subject: [PATCH 238/648] Update lowering rules for GATs --- src/traits-associated-types.md | 7 ++- src/traits-lowering-rules.md | 106 +++++++++++++++++++++------------ 2 files changed, 74 insertions(+), 39 deletions(-) diff --git a/src/traits-associated-types.md b/src/traits-associated-types.md index 51972c41e..ec20985ff 100644 --- a/src/traits-associated-types.md +++ b/src/traits-associated-types.md @@ -53,12 +53,15 @@ we saw above would be lowered to a program clause like so: ```text forall { - Normalize( as IntoIterator>::Item -> T) + Normalize( as IntoIterator>::Item -> T) :- + Implemented(Option: IntoIterator) } ``` +where in this case, the one `Implemented` condition is always true. + (An aside: since we do not permit quantification over traits, this is -really more like a family of predicates, one for each associated +really more like a family of program clauses, one for each associated type.) We could apply that rule to normalize either of the examples that diff --git a/src/traits-lowering-rules.md b/src/traits-lowering-rules.md index 80884cbdd..06dc67401 100644 --- a/src/traits-lowering-rules.md +++ b/src/traits-lowering-rules.md @@ -132,8 +132,6 @@ to be **well-formed**: ```text // Rule WellFormed-TraitRef -// -// For each where clause WC: forall { WellFormed(Self: Trait) :- Implemented(Self: Trait) && WellFormed(WC) } @@ -198,38 +196,72 @@ in detail in the [section on associated types](./traits-associated-types.html), but reproduced here for reference: ```text - // Rule ProjectionEq-Normalize - // - // ProjectionEq can succeed by normalizing: - forall { - ProjectionEq(>::AssocType = U) :- - Normalize(>::AssocType -> U) - } +// Rule ProjectionEq-Normalize +// +// ProjectionEq can succeed by normalizing: +forall { + ProjectionEq(>::AssocType = U) :- + Normalize(>::AssocType -> U) +} +``` - // Rule ProjectionEq-Skolemize - // - // ProjectionEq can succeed by skolemizing, see "associated type" - // chapter for more: - forall { - ProjectionEq( - >::AssocType = - (Trait::AssocType) - ) :- - // But only if the trait is implemented, and the conditions from - // the associated type are met as well: - Implemented(Self: Trait) - && WC1 - } +```text +// Rule ProjectionEq-Skolemize +// +// ProjectionEq can succeed by skolemizing, see "associated type" +// chapter for more: +forall { + ProjectionEq( + >::AssocType = + (Trait::AssocType) + ) +} ``` The next rule covers implied bounds for the projection. In particular, -the `Bounds` declared on the associated type must be proven to hold to -show that the impl is well-formed, and hence we can rely on them +the `Bounds` declared on the associated type must have been proven to hold +to show that the impl is well-formed, and hence we can rely on them elsewhere. ```text -// XXX how exactly should we set this up? Have to be careful; -// presumably this has to be a kind of `FromEnv` setup. +// Rule Implied-Bound-From-AssocTy +// +// For each `Bound` in `Bounds`: +forall { + FromEnv(>::AssocType>: Bound) :- + FromEnv(Self: Trait) +} +``` + +Next, we define the requirements for an instantiation of our associated +type to be well-formed... + +```text +// Rule WellFormed-AssocTy +forall { + WellFormed((Trait::AssocType)) :- + WC1, Implemented(Self: Trait) +} +``` + +...along with the reverse implications, when we can assume that it is +well-formed. + +```text +// Rule Implied-WC-From-AssocTy +// +// For each where clause WC1: +forall { + FromEnv(WC1) :- FromEnv((Trait::AssocType)) +} +``` + +```text +// Rule Implied-Trait-From-AssocTy +forall { + FromEnv(Self: Trait) :- + FromEnv((Trait::AssocType)) +} ``` ### Lowering function and constant declarations @@ -269,27 +301,29 @@ In addition, we will lower all of the *impl items*. Given an impl that contains: ```rust,ignore -impl Trait for A0 -where WC +impl Trait for P0 +where WC_impl { - type AssocType where WC1 = T; + type AssocType = T; } ``` -We produce the following rule: +and our where clause `WC1` on the trait associated type from above, we +produce the following rule: ```text // Rule Normalize-From-Impl forall { forall { - Normalize(>::AssocType -> T) :- - WC && WC1 + Normalize(>::AssocType -> T) :- + Implemented(P0 as Trait) && WC1 } } ``` -Note that `WC` and `WC1` both encode where-clauses that the impl can -rely on. +Note that `WC_impl` and `WC1` both encode where-clauses that the impl can +rely on. (`WC_impl` is not used here, because it is implied by +`Implemented(P0 as Trait)`.) @@ -300,5 +334,3 @@ like to treat them exactly like normalization. This presumably involves adding a new kind of parameter (constant), and then having a `NormalizeValue` domain goal. This is *to be written* because the details are a bit up in the air. - - From dab4531fdc8707e563aa03bb2e0ceb70bf40d380 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Sat, 2 Jun 2018 16:06:00 -0500 Subject: [PATCH 239/648] clarify run-rustfix compiletest header --- src/tests/adding.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/tests/adding.md b/src/tests/adding.md index 3788e7c41..28f4c1140 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -149,9 +149,12 @@ exhaustive. Header commands can generally be found by browsing the source. * `run-rustfix` for UI tests, indicates that the test produces - structured suggestions, which are then applied and the final - source is stored in a `.fixed` file and compiled again. The final - compilation is required to succeed. + structured suggestions. The test writer should create a `.fixed` + file, which contains the source with the suggestions applied. + When the test is run, compiletest first checks that the correct + lint/warning is generated. Then, it applies the suggestion and + compares against `.fixed` (they must match). Finally, the fixed + source is compiled, and this compilation is required to succeed. * `min-{gdb,lldb}-version` * `min-llvm-version` * `compile-pass` for UI tests, indicates that the test is From 1af6fcfc67aa51a952f0dda23211396bb63859e4 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 23 May 2018 11:46:47 -0500 Subject: [PATCH 240/648] move appendix --- src/{appendix-background.md => appendix/background.md} | 0 src/{appendix-code-index.md => appendix/code-index.md} | 0 src/{appendix-glossary.md => appendix/glossary.md} | 0 src/{appendix-stupid-stats.md => appendix/stupid-stats.md} | 0 4 files changed, 0 insertions(+), 0 deletions(-) rename src/{appendix-background.md => appendix/background.md} (100%) rename src/{appendix-code-index.md => appendix/code-index.md} (100%) rename src/{appendix-glossary.md => appendix/glossary.md} (100%) rename src/{appendix-stupid-stats.md => appendix/stupid-stats.md} (100%) diff --git a/src/appendix-background.md b/src/appendix/background.md similarity index 100% rename from src/appendix-background.md rename to src/appendix/background.md diff --git a/src/appendix-code-index.md b/src/appendix/code-index.md similarity index 100% rename from src/appendix-code-index.md rename to src/appendix/code-index.md diff --git a/src/appendix-glossary.md b/src/appendix/glossary.md similarity index 100% rename from src/appendix-glossary.md rename to src/appendix/glossary.md diff --git a/src/appendix-stupid-stats.md b/src/appendix/stupid-stats.md similarity index 100% rename from src/appendix-stupid-stats.md rename to src/appendix/stupid-stats.md From 1b0fe8995aae7c17da01eebf27869b6f632df0ed Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 23 May 2018 11:49:24 -0500 Subject: [PATCH 241/648] move mir --- src/{mir-borrowck.md => mir/borrowck.md} | 0 src/{mir-construction.md => mir/construction.md} | 0 src/{mir.md => mir/index.md} | 0 src/{mir-optimizations.md => mir/optimizations.md} | 0 src/{mir-passes.md => mir/passes.md} | 0 src/{mir-regionck.md => mir/regionck.md} | 0 src/{mir-visitor.md => mir/visitor.md} | 0 7 files changed, 0 insertions(+), 0 deletions(-) rename src/{mir-borrowck.md => mir/borrowck.md} (100%) rename src/{mir-construction.md => mir/construction.md} (100%) rename src/{mir.md => mir/index.md} (100%) rename src/{mir-optimizations.md => mir/optimizations.md} (100%) rename src/{mir-passes.md => mir/passes.md} (100%) rename src/{mir-regionck.md => mir/regionck.md} (100%) rename src/{mir-visitor.md => mir/visitor.md} (100%) diff --git a/src/mir-borrowck.md b/src/mir/borrowck.md similarity index 100% rename from src/mir-borrowck.md rename to src/mir/borrowck.md diff --git a/src/mir-construction.md b/src/mir/construction.md similarity index 100% rename from src/mir-construction.md rename to src/mir/construction.md diff --git a/src/mir.md b/src/mir/index.md similarity index 100% rename from src/mir.md rename to src/mir/index.md diff --git a/src/mir-optimizations.md b/src/mir/optimizations.md similarity index 100% rename from src/mir-optimizations.md rename to src/mir/optimizations.md diff --git a/src/mir-passes.md b/src/mir/passes.md similarity index 100% rename from src/mir-passes.md rename to src/mir/passes.md diff --git a/src/mir-regionck.md b/src/mir/regionck.md similarity index 100% rename from src/mir-regionck.md rename to src/mir/regionck.md diff --git a/src/mir-visitor.md b/src/mir/visitor.md similarity index 100% rename from src/mir-visitor.md rename to src/mir/visitor.md From 13f38bebad77c79d1fcd03bed3e9802b821c7578 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 23 May 2018 11:54:23 -0500 Subject: [PATCH 242/648] mv traits --- src/{traits-associated-types.md => traits/associated-types.md} | 0 src/{traits-bibliography.md => traits/bibliography.md} | 0 src/{trait-caching.md => traits/caching.md} | 0 src/{traits-canonical-queries.md => traits/canonical-queries.md} | 0 src/{traits-canonicalization.md => traits/canonicalization.md} | 0 src/{traits-goals-and-clauses.md => traits/goals-and-clauses.md} | 0 src/{trait-hrtb.md => traits/hrtb.md} | 0 src/{traits-implied-bounds.md => traits/implied-bounds.md} | 0 src/{traits.md => traits/index.md} | 0 src/{traits-lowering-module.md => traits/lowering-module.md} | 0 src/{traits-lowering-rules.md => traits/lowering-rules.md} | 0 src/{traits-lowering-to-logic.md => traits/lowering-to-logic.md} | 0 src/{traits-regions.md => traits/regions.md} | 0 src/{trait-resolution.md => traits/resolution.md} | 0 src/{traits-slg.md => traits/slg.md} | 0 src/{traits-wf.md => traits/wf.md} | 0 16 files changed, 0 insertions(+), 0 deletions(-) rename src/{traits-associated-types.md => traits/associated-types.md} (100%) rename src/{traits-bibliography.md => traits/bibliography.md} (100%) rename src/{trait-caching.md => traits/caching.md} (100%) rename src/{traits-canonical-queries.md => traits/canonical-queries.md} (100%) rename src/{traits-canonicalization.md => traits/canonicalization.md} (100%) rename src/{traits-goals-and-clauses.md => traits/goals-and-clauses.md} (100%) rename src/{trait-hrtb.md => traits/hrtb.md} (100%) rename src/{traits-implied-bounds.md => traits/implied-bounds.md} (100%) rename src/{traits.md => traits/index.md} (100%) rename src/{traits-lowering-module.md => traits/lowering-module.md} (100%) rename src/{traits-lowering-rules.md => traits/lowering-rules.md} (100%) rename src/{traits-lowering-to-logic.md => traits/lowering-to-logic.md} (100%) rename src/{traits-regions.md => traits/regions.md} (100%) rename src/{trait-resolution.md => traits/resolution.md} (100%) rename src/{traits-slg.md => traits/slg.md} (100%) rename src/{traits-wf.md => traits/wf.md} (100%) diff --git a/src/traits-associated-types.md b/src/traits/associated-types.md similarity index 100% rename from src/traits-associated-types.md rename to src/traits/associated-types.md diff --git a/src/traits-bibliography.md b/src/traits/bibliography.md similarity index 100% rename from src/traits-bibliography.md rename to src/traits/bibliography.md diff --git a/src/trait-caching.md b/src/traits/caching.md similarity index 100% rename from src/trait-caching.md rename to src/traits/caching.md diff --git a/src/traits-canonical-queries.md b/src/traits/canonical-queries.md similarity index 100% rename from src/traits-canonical-queries.md rename to src/traits/canonical-queries.md diff --git a/src/traits-canonicalization.md b/src/traits/canonicalization.md similarity index 100% rename from src/traits-canonicalization.md rename to src/traits/canonicalization.md diff --git a/src/traits-goals-and-clauses.md b/src/traits/goals-and-clauses.md similarity index 100% rename from src/traits-goals-and-clauses.md rename to src/traits/goals-and-clauses.md diff --git a/src/trait-hrtb.md b/src/traits/hrtb.md similarity index 100% rename from src/trait-hrtb.md rename to src/traits/hrtb.md diff --git a/src/traits-implied-bounds.md b/src/traits/implied-bounds.md similarity index 100% rename from src/traits-implied-bounds.md rename to src/traits/implied-bounds.md diff --git a/src/traits.md b/src/traits/index.md similarity index 100% rename from src/traits.md rename to src/traits/index.md diff --git a/src/traits-lowering-module.md b/src/traits/lowering-module.md similarity index 100% rename from src/traits-lowering-module.md rename to src/traits/lowering-module.md diff --git a/src/traits-lowering-rules.md b/src/traits/lowering-rules.md similarity index 100% rename from src/traits-lowering-rules.md rename to src/traits/lowering-rules.md diff --git a/src/traits-lowering-to-logic.md b/src/traits/lowering-to-logic.md similarity index 100% rename from src/traits-lowering-to-logic.md rename to src/traits/lowering-to-logic.md diff --git a/src/traits-regions.md b/src/traits/regions.md similarity index 100% rename from src/traits-regions.md rename to src/traits/regions.md diff --git a/src/trait-resolution.md b/src/traits/resolution.md similarity index 100% rename from src/trait-resolution.md rename to src/traits/resolution.md diff --git a/src/traits-slg.md b/src/traits/slg.md similarity index 100% rename from src/traits-slg.md rename to src/traits/slg.md diff --git a/src/traits-wf.md b/src/traits/wf.md similarity index 100% rename from src/traits-wf.md rename to src/traits/wf.md From 55883c4a7e96a0161aaba98991de2866367389b5 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 23 May 2018 12:13:46 -0500 Subject: [PATCH 243/648] first round of link fixes --- src/SUMMARY.md | 52 ++++++++++++++++----------------- src/rustc-driver.md | 22 +++++++------- src/traits/caching.md | 4 +-- src/traits/canonical-queries.md | 6 ++-- src/traits/canonicalization.md | 6 ++-- src/traits/goals-and-clauses.md | 13 +++++---- src/traits/index.md | 12 ++++---- src/traits/lowering-module.md | 2 +- src/traits/lowering-rules.md | 10 +++---- src/traits/lowering-to-logic.md | 4 +-- src/traits/resolution.md | 2 +- src/type-inference.md | 2 +- 12 files changed, 68 insertions(+), 67 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 2db3e62c3..a2b7d3e13 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -22,34 +22,34 @@ - [The HIR (High-level IR)](./hir.md) - [The `ty` module: representing types](./ty.md) - [Type inference](./type-inference.md) -- [Trait solving (old-style)](./trait-resolution.md) - - [Higher-ranked trait bounds](./trait-hrtb.md) - - [Caching subtleties](./trait-caching.md) - - [Specialization](./trait-specialization.md) +- [Trait solving (old-style)](./traits/resolution.md) + - [Higher-ranked trait bounds](./traits/hrtb.md) + - [Caching subtleties](./traits/caching.md) + - [Specialization](./traits/specialization.md) - [Trait solving (new-style)](./traits.md) - - [Lowering to logic](./traits-lowering-to-logic.md) - - [Goals and clauses](./traits-goals-and-clauses.md) - - [Equality and associated types](./traits-associated-types.md) - - [Implied bounds](./traits-implied-bounds.md) - - [Region constraints](./traits-regions.md) - - [Canonical queries](./traits-canonical-queries.md) - - [Canonicalization](./traits-canonicalization.md) - - [Lowering rules](./traits-lowering-rules.md) - - [The lowering module in rustc](./traits-lowering-module.md) - - [Well-formedness checking](./traits-wf.md) - - [The SLG solver](./traits-slg.md) + - [Lowering to logic](./traits/lowering-to-logic.md) + - [Goals and clauses](./traits/goals-and-clauses.md) + - [Equality and associated types](./traits/associated-types.md) + - [Implied bounds](./traits/implied-bounds.md) + - [Region constraints](./traits/regions.md) + - [Canonical queries](./traits/canonical-queries.md) + - [Canonicalization](./traits/canonicalization.md) + - [Lowering rules](./traits/lowering-rules.md) + - [The lowering module in rustc](./traits/lowering-module.md) + - [Well-formedness checking](./traits/wf.md) + - [The SLG solver](./traits/slg.md) - [An Overview of Chalk](./chalk-overview.md) - - [Bibliography](./traits-bibliography.md) + - [Bibliography](./traits/bibliography.md) - [Type checking](./type-checking.md) - [Method Lookup](./method-lookup.md) - [Variance](./variance.md) - [The MIR (Mid-level IR)](./mir.md) - - [MIR construction](./mir-construction.md) - - [MIR visitor and traversal](./mir-visitor.md) - - [MIR passes: getting the MIR for a function](./mir-passes.md) - - [MIR borrowck](./mir-borrowck.md) - - [MIR-based region checking (NLL)](./mir-regionck.md) - - [MIR optimizations](./mir-optimizations.md) + - [MIR construction](./mir/construction.md) + - [MIR visitor and traversal](./mir/visitor.md) + - [MIR passes: getting the MIR for a function](./mir/passes.md) + - [MIR borrowck](./mir/borrowck.md) + - [MIR-based region checking (NLL)](./mir/regionck.md) + - [MIR optimizations](./mir/optimizations.md) - [Constant evaluation](./const-eval.md) - [miri const evaluator](./miri.md) - [Parameter Environments](./param_env.md) @@ -58,7 +58,7 @@ --- -- [Appendix A: Stupid Stats](./appendix-stupid-stats.md) -- [Appendix B: Background material](./appendix-background.md) -- [Appendix C: Glossary](./appendix-glossary.md) -- [Appendix D: Code Index](./appendix-code-index.md) +- [Appendix A: Stupid Stats](./appendix/stupid-stats.md) +- [Appendix B: Background material](./appendix/background.md) +- [Appendix C: Glossary](./appendix/glossary.md) +- [Appendix D: Code Index](./appendix/code-index.md) diff --git a/src/rustc-driver.md b/src/rustc-driver.md index af3c3c099..1550b14e9 100644 --- a/src/rustc-driver.md +++ b/src/rustc-driver.md @@ -20,14 +20,14 @@ of each phase. From `rustc_driver`'s perspective, the main phases of the compiler are: 1. *Parse Input:* Initial crate parsing -2. *Configure and Expand:* Resolve `#[cfg]` attributes, name resolution, and +2. *Configure and Expand:* Resolve `#[cfg]` attributes, name resolution, and expand macros 3. *Run Analysis Passes:* Run trait resolution, typechecking, region checking and other miscellaneous analysis passes on the crate -4. *Translate to LLVM:* Translate to the in-memory form of LLVM IR and turn it +4. *Translate to LLVM:* Translate to the in-memory form of LLVM IR and turn it into an executable/object files -The `CompileController` then gives users the ability to inspect the ongoing +The `CompileController` then gives users the ability to inspect the ongoing compilation process - after parsing @@ -39,7 +39,7 @@ compilation process The `CompileState`'s various `state_after_*()` constructors can be inspected to determine what bits of information are available to which callback. -For a more detailed explanation on using `rustc_driver`, check out the +For a more detailed explanation on using `rustc_driver`, check out the [stupid-stats] guide by `@nrc` (attached as [Appendix A]). > **Warning:** By its very nature, the internal compiler APIs are always going @@ -47,23 +47,23 @@ For a more detailed explanation on using `rustc_driver`, check out the ## A Note On Lifetimes -The Rust compiler is a fairly large program containing lots of big data +The Rust compiler is a fairly large program containing lots of big data structures (e.g. the AST, HIR, and the type system) and as such, arenas and -references are heavily relied upon to minimize unnecessary memory use. This +references are heavily relied upon to minimize unnecessary memory use. This manifests itself in the way people can plug into the compiler, preferring a "push"-style API (callbacks) instead of the more Rust-ic "pull" style (think the `Iterator` trait). -For example the [`CompileState`], the state passed to callbacks after each +For example the [`CompileState`], the state passed to callbacks after each phase, is essentially just a box of optional references to pieces inside the compiler. The lifetime bound on the `CompilerCalls` trait then helps to ensure -compiler internals don't "escape" the compiler (e.g. if you tried to keep a +compiler internals don't "escape" the compiler (e.g. if you tried to keep a reference to the AST after the compiler is finished), while still letting users record *some* state for use after the `run_compiler()` function finishes. Thread-local storage and interning are used a lot through the compiler to reduce -duplication while also preventing a lot of the ergonomic issues due to many -pervasive lifetimes. The `rustc::ty::tls` module is used to access these +duplication while also preventing a lot of the ergonomic issues due to many +pervasive lifetimes. The `rustc::ty::tls` module is used to access these thread-locals, although you should rarely need to touch it. @@ -73,4 +73,4 @@ thread-locals, although you should rarely need to touch it. [`TyCtxt`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html [`CodeMap`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html [stupid-stats]: https://github.com/nrc/stupid-stats -[Appendix A]: appendix-stupid-stats.html \ No newline at end of file +[Appendix A]: appendix/stupid-stats.html diff --git a/src/traits/caching.md b/src/traits/caching.md index 4b7d7e096..228ff0917 100644 --- a/src/traits/caching.md +++ b/src/traits/caching.md @@ -24,7 +24,7 @@ On the other hand, if there is no hit, we need to go through the [selection process] from scratch. Suppose, we come to the conclusion that the only possible impl is this one, with def-id 22: -[selection process]: ./trait-resolution.html#selection +[selection process]: ./traits/resolution.html#selection ```rust,ignore impl Foo for usize { ... } // Impl #22 @@ -34,7 +34,7 @@ We would then record in the cache `usize : Foo<$0> => ImplCandidate(22)`. Next we would [confirm] `ImplCandidate(22)`, which would (as a side-effect) unify `$t` with `isize`. -[confirm]: ./trait-resolution.html#confirmation +[confirm]: ./traits/resolution.html#confirmation Now, at some later time, we might come along and see a `usize : Foo<$u>`. When skolemized, this would yield `usize : Foo<$0>`, just as diff --git a/src/traits/canonical-queries.md b/src/traits/canonical-queries.md index 2737737bb..56637dace 100644 --- a/src/traits/canonical-queries.md +++ b/src/traits/canonical-queries.md @@ -7,7 +7,7 @@ would like to know the answer to -- and in the checker or other parts of the system, may in the course of doing their thing want to know whether some trait is implemented for some type (e.g., is `u32: Debug` true?). Or they may want to -[normalize some associated type](./traits-associated-types.html). +[normalize some associated type](./traits/associated-types.html). This section covers queries at a fairly high level of abstraction. The subsections look a bit more closely at how these ideas are implemented @@ -106,7 +106,7 @@ value for a type variable, that means that this is the **only possible instantiation** that you could use, given the current set of impls and where-clauses, that would be provable. (Internally within the solver, though, they can potentially enumerate all possible answers. See -[the description of the SLG solver](./traits-slg.html) for details.) +[the description of the SLG solver](./traits/slg.html) for details.) The response to a trait query in rustc is typically a `Result, NoSolution>` (where the `T` will vary a bit @@ -132,7 +132,7 @@ we did find. It consists of four parts: - **Region constraints:** these are relations that must hold between the lifetimes that you supplied as inputs. We'll ignore these here, but see the - [section on handling regions in traits](./traits-regions.html) for + [section on handling regions in traits](./traits/regions.html) for more details. - **Value:** The query result also comes with a value of type `T`. For some specialized queries -- like normalizing associated types -- diff --git a/src/traits/canonicalization.md b/src/traits/canonicalization.md index fa39151d7..37a59026b 100644 --- a/src/traits/canonicalization.md +++ b/src/traits/canonicalization.md @@ -16,7 +16,7 @@ starting from zero and numbered in a fixed order (left to right, for the most part, but really it doesn't matter as long as it is consistent). -[cq]: ./traits-canonical-queries.html +[cq]: ./traits/canonical-queries.html So, for example, if we have the type `X = (?T, ?U)`, where `?T` and `?U` are distinct, unbound inference variables, then the canonical @@ -98,12 +98,12 @@ Remember that substitution S though! We're going to need it later. OK, now that we have a fresh inference context and an instantiated query, we can go ahead and try to solve it. The trait solver itself is -explained in more detail in [another section](./traits-slg.html), but +explained in more detail in [another section](./traits/slg.html), but suffice to say that it will compute a [certainty value][cqqr] (`Proven` or `Ambiguous`) and have side-effects on the inference variables we've created. For example, if there were only one impl of `Foo`, like so: -[cqqr]: ./traits-canonical-queries.html#query-response +[cqqr]: ./traits/canonical-queries.html#query-response ```rust,ignore impl<'a, X> Foo<'a, X> for Vec diff --git a/src/traits/goals-and-clauses.md b/src/traits/goals-and-clauses.md index 882f86b8e..5844e8d45 100644 --- a/src/traits/goals-and-clauses.md +++ b/src/traits/goals-and-clauses.md @@ -2,7 +2,7 @@ In logic programming terms, a **goal** is something that you must prove and a **clause** is something that you know is true. As -described in the [lowering to logic](./traits-lowering-to-logic.html) +described in the [lowering to logic](./traits/lowering-to-logic.html) chapter, Rust's trait solver is based on an extension of hereditary harrop (HH) clauses, which extend traditional Prolog Horn clauses with a few new superpowers. @@ -37,7 +37,7 @@ paper ["A Proof Procedure for the Logic of Hereditary Harrop Formulas"][pphhf] gives the details. -[pphhf]: ./traits-bibliography.html#pphhf +[pphhf]: ./traits/bibliography.html#pphhf @@ -94,7 +94,7 @@ e.g. `ProjectionEq::Item = u8` The given associated type `Projection` is equal to `Type`; this can be proved with either normalization or using skolemized types. See [the section -on associated types](./traits-associated-types.html). +on associated types](./traits/associated-types.html). #### Normalize(Projection -> Type) e.g. `ProjectionEq::Item -> u8` @@ -102,11 +102,12 @@ e.g. `ProjectionEq::Item -> u8` The given associated type `Projection` can be [normalized][n] to `Type`. As discussed in [the section on associated -types](./traits-associated-types.html), `Normalize` implies `ProjectionEq`, +types](./traits/associated-types.html), `Normalize` implies `ProjectionEq`, but not vice versa. In general, proving `Normalize(::Item -> U)` also requires proving `Implemented(T: Trait)`. -[n]: ./traits-associated-types.html#normalize +[n]: ./traits/associated-types.html#normalize +[at]: ./traits/associated-types.html #### FromEnv(TraitRef), FromEnv(Projection = Type) e.g. `FromEnv(Self: Add)` @@ -211,7 +212,7 @@ In addition to auto traits, `WellFormed` predicates are co-inductive. These are used to achieve a similar "enumerate all the cases" pattern, as described in the section on [implied bounds]. -[implied bounds]: ./traits-lowering-rules.html#implied-bounds +[implied bounds]: ./traits/lowering-rules.html#implied-bounds ## Incomplete chapter diff --git a/src/traits/index.md b/src/traits/index.md index 175e6418b..6aad2474d 100644 --- a/src/traits/index.md +++ b/src/traits/index.md @@ -13,20 +13,20 @@ instructions for getting involved in the Trait solving is based around a few key ideas: -- [Lowering to logic](./traits-lowering-to-logic.html), which expresses +- [Lowering to logic](./traits/lowering-to-logic.html), which expresses Rust traits in terms of standard logical terms. - - The [goals and clauses](./traits-goals-and-clauses.html) chapter + - The [goals and clauses](./traits/goals-and-clauses.html) chapter describes the precise form of rules we use, and - [lowering rules](./traits-lowering-rules.html) gives the complete set of + [lowering rules](./traits/lowering-rules.html) gives the complete set of lowering rules in a more reference-like form. -- [Canonical queries](./traits-canonical-queries.html), which allow us +- [Canonical queries](./traits/canonical-queries.html), which allow us to solve trait problems (like "is `Foo` implemented for the type `Bar`?") once, and then apply that same result independently in many different inference contexts. -- [Lazy normalization](./traits-associated-types.html), which is the +- [Lazy normalization](./traits/associated-types.html), which is the technique we use to accommodate associated types when figuring out whether types are equal. -- [Region constraints](./traits-regions.html), which are accumulated +- [Region constraints](./traits/regions.html), which are accumulated during trait solving but mostly ignored. This means that trait solving effectively ignores the precise regions involved, always -- but we still remember the constraints on them so that those diff --git a/src/traits/lowering-module.md b/src/traits/lowering-module.md index 08e0b9523..205e9a171 100644 --- a/src/traits/lowering-module.md +++ b/src/traits/lowering-module.md @@ -1,7 +1,7 @@ # The lowering module in rustc The program clauses described in the -[lowering rules](./traits-lowering-rules.html) section are actually +[lowering rules](./traits/lowering-rules.html) section are actually created in the [`rustc_traits::lowering`][lowering] module. [lowering]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_traits/lowering/ diff --git a/src/traits/lowering-rules.md b/src/traits/lowering-rules.md index 06dc67401..f635ee857 100644 --- a/src/traits/lowering-rules.md +++ b/src/traits/lowering-rules.md @@ -4,8 +4,8 @@ This section gives the complete lowering rules for Rust traits into [program clauses][pc]. It is a kind of reference. These rules reference the [domain goals][dg] defined in an earlier section. -[pc]: ./traits-goals-and-clauses.html -[dg]: ./traits-goals-and-clauses.html#domain-goals +[pc]: ./traits/goals-and-clauses.html +[dg]: ./traits/goals-and-clauses.html#domain-goals ## Notation @@ -16,7 +16,7 @@ The nonterminal `Ai` is used to mean some generic *argument*, which might be a lifetime like `'a` or a type like `Vec`. When defining the lowering rules, we will give goals and clauses in -the [notation given in this section](./traits-goals-and-clauses.html). +the [notation given in this section](./traits/goals-and-clauses.html). We sometimes insert "macros" like `LowerWhereClause!` into these definitions; these macros reference other sections within this chapter. @@ -141,7 +141,7 @@ This `WellFormed` rule states that `T: Trait` is well-formed if (a) `T: Trait` is implemented and (b) all the where-clauses declared on `Trait` are well-formed (and hence they are implemented). Remember that the `WellFormed` predicate is -[coinductive](./traits-goals-and-clauses.html#coinductive); in this +[coinductive](./traits/goals-and-clauses.html#coinductive); in this case, it is serving as a kind of "carrier" that allows us to enumerate all the where clauses that are transitively implied by `T: Trait`. @@ -192,7 +192,7 @@ where WC We will produce a number of program clauses. The first two define the rules by which `ProjectionEq` can succeed; these two clauses are discussed -in detail in the [section on associated types](./traits-associated-types.html), +in detail in the [section on associated types](./traits/associated-types.html), but reproduced here for reference: ```text diff --git a/src/traits/lowering-to-logic.md b/src/traits/lowering-to-logic.md index 54b3473d4..2de28ce4e 100644 --- a/src/traits/lowering-to-logic.md +++ b/src/traits/lowering-to-logic.md @@ -170,8 +170,8 @@ example Gopalan Nadathur's excellent ["A Proof Procedure for the Logic of Hereditary Harrop Formulas"][pphhf] in [the bibliography]. -[the bibliography]: ./traits-bibliography.html -[pphhf]: ./traits-bibliography.html#pphhf +[the bibliography]: ./traits/bibliography.html +[pphhf]: ./traits/bibliography.html#pphhf It turns out that supporting FOHH is not really all that hard. And once we are able to do that, we can easily describe the type-checking diff --git a/src/traits/resolution.md b/src/traits/resolution.md index b9fa81bd6..e33e0c8ec 100644 --- a/src/traits/resolution.md +++ b/src/traits/resolution.md @@ -6,7 +6,7 @@ some non-obvious things. **Note:** This chapter (and its subchapters) describe how the trait solver **currently** works. However, we are in the process of designing a new trait solver. If you'd prefer to read about *that*, -see [*this* traits chapter](./traits.html). +see [*this* traits chapter](./traits/index.html). ## Major concepts diff --git a/src/type-inference.md b/src/type-inference.md index 9ae88decd..a3d012b5d 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -125,7 +125,7 @@ actual return type is not `()`, but rather `InferOk<()>`. The to ensure that these are fulfilled (typically by enrolling them in a fulfillment context). See the [trait chapter] for more background on that. -[trait chapter]: trait-resolution.html +[trait chapter]: traits/resolution.html You can similarly enforce subtyping through `infcx.at(..).sub(..)`. The same basic concepts as above apply. From c0851bc66e0771869e76b17d301b6ec1d22c238c Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 23 May 2018 12:30:10 -0500 Subject: [PATCH 244/648] fix remaining links --- src/SUMMARY.md | 6 ++-- src/appendix/glossary.md | 30 +++++++++---------- src/mir/borrowck.md | 12 ++++---- src/mir/index.md | 4 +-- src/mir/passes.md | 2 +- src/mir/regionck.md | 8 ++--- src/traits/canonicalization.md | 2 +- src/{ => traits}/chalk-overview.md | 10 +++---- src/traits/index.md | 2 +- .../specialization.md} | 0 src/type-checking.md | 2 +- src/variance.md | 2 +- 12 files changed, 40 insertions(+), 40 deletions(-) rename src/{ => traits}/chalk-overview.md (95%) rename src/{trait-specialization.md => traits/specialization.md} (100%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index a2b7d3e13..8efe304b7 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -26,7 +26,7 @@ - [Higher-ranked trait bounds](./traits/hrtb.md) - [Caching subtleties](./traits/caching.md) - [Specialization](./traits/specialization.md) -- [Trait solving (new-style)](./traits.md) +- [Trait solving (new-style)](./traits/index.md) - [Lowering to logic](./traits/lowering-to-logic.md) - [Goals and clauses](./traits/goals-and-clauses.md) - [Equality and associated types](./traits/associated-types.md) @@ -38,12 +38,12 @@ - [The lowering module in rustc](./traits/lowering-module.md) - [Well-formedness checking](./traits/wf.md) - [The SLG solver](./traits/slg.md) - - [An Overview of Chalk](./chalk-overview.md) + - [An Overview of Chalk](./traits/chalk-overview.md) - [Bibliography](./traits/bibliography.md) - [Type checking](./type-checking.md) - [Method Lookup](./method-lookup.md) - [Variance](./variance.md) -- [The MIR (Mid-level IR)](./mir.md) +- [The MIR (Mid-level IR)](./mir/index.md) - [MIR construction](./mir/construction.md) - [MIR visitor and traversal](./mir/visitor.md) - [MIR passes: getting the MIR for a function](./mir/passes.md) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index a37f1b22f..ad2604e63 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -7,18 +7,18 @@ them better. Term | Meaning ------------------------|-------- AST | the abstract syntax tree produced by the syntax crate; reflects user syntax very closely. -binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and \|`a`\|` ...` is a binder for the parameter `a`. See [the background chapter for more](./appendix-background.html#free-vs-bound) -bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession \|`a`\|` a * 2`. See [the background chapter for more](./appendix-background.html#free-vs-bound) +binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and \|`a`\|` ...` is a binder for the parameter `a`. See [the background chapter for more](./appendix/background.html#free-vs-bound) +bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession \|`a`\|` a * 2`. See [the background chapter for more](./appendix/background.html#free-vs-bound) codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. completeness | completeness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). -control-flow graph | a representation of the control-flow of a program; see [the background chapter for more](./appendix-background.html#cfg) +control-flow graph | a representation of the control-flow of a program; see [the background chapter for more](./appendix/background.html#cfg) cx | we tend to use "cx" as an abbrevation for context. See also `tcx`, `infcx`, etc. DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](incremental-compilation.html)) -data-flow analysis | a static analysis that figures out what properties are true at each point in the control-flow of a program; see [the background chapter for more](./appendix-background.html#dataflow) +data-flow analysis | a static analysis that figures out what properties are true at each point in the control-flow of a program; see [the background chapter for more](./appendix/background.html#dataflow) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. Double pointer | a pointer with additional metadata. See "fat pointer" for more. Fat pointer | a two word value carrying the address of some value, along with some further information necessary to put the value to use. Rust includes two kinds of "fat pointers": references to slices, and trait objects. A reference to a slice carries the starting address of the slice and its length. A trait object carries a value's address and a pointer to the trait's implementation appropriate to that value. "Fat pointers" are also known as "wide pointers", and "double pointers". -free variable | a "free variable" is one that is not bound within an expression or term; see [the background chapter for more](./appendix-background.html#free-vs-bound) +free variable | a "free variable" is one that is not bound within an expression or term; see [the background chapter for more](./appendix/background.html#free-vs-bound) 'gcx | the lifetime of the global arena ([see more](ty.html)) generics | the set of generic type parameters defined on a type or item HIR | the High-level IR, created by lowering and desugaring the AST ([see more](hir.html)) @@ -32,36 +32,36 @@ IR | Intermediate Representation. A general term in compil local crate | the crate currently being compiled. LTO | Link-Time Optimizations. A set of optimizations offered by LLVM that occur just before the final binary is linked. These include optmizations like removing functions that are never used in the final program, for example. _ThinLTO_ is a variant of LTO that aims to be a bit more scalable and efficient, but possibly sacrifices some optimizations. You may also read issues in the Rust repo about "FatLTO", which is the loving nickname given to non-Thin LTO. LLVM documentation: [here][lto] and [here][thinlto] [LLVM] | (actually not an acronym :P) an open-source compiler backend. It accepts LLVM IR and outputs native binaries. Various languages (e.g. Rust) can then implement a compiler front-end that output LLVM IR and use LLVM to compile to all the platforms LLVM supports. -MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir.html)) +MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir/index.html)) miri | an interpreter for MIR used for constant evaluation ([see more](./miri.html)) -normalize | a general term for converting to a more canonical form, but in the case of rustc typically refers to [associated type normalization](./traits-associated-types.html#normalize) +normalize | a general term for converting to a more canonical form, but in the case of rustc typically refers to [associated type normalization](./traits/associated-types.html#normalize) newtype | a "newtype" is a wrapper around some other type (e.g., `struct Foo(T)` is a "newtype" for `T`). This is commonly used in Rust to give a stronger type for indices. -NLL | [non-lexical lifetimes](./mir-regionck.html), an extension to Rust's borrowing system to make it be based on the control-flow graph. +NLL | [non-lexical lifetimes](./mir/regionck.html), an extension to Rust's borrowing system to make it be based on the control-flow graph. node-id or NodeId | an index identifying a particular node in the AST or HIR; gradually being phased out and replaced with `HirId`. -obligation | something that must be proven by the trait system ([see more](trait-resolution.html)) -projection | a general term for a "relative path", e.g. `x.f` is a "field projection", and `T::Item` is an ["associated type projection"](./traits-goals-and-clauses.html#trait-ref) -promoted constants | constants extracted from a function and lifted to static scope; see [this section](./mir.html#promoted) for more details. +obligation | something that must be proven by the trait system ([see more](traits/resolution.html)) +projection | a general term for a "relative path", e.g. `x.f` is a "field projection", and `T::Item` is an ["associated type projection"](./traits/goals-and-clauses.html#trait-ref) +promoted constants | constants extracted from a function and lifted to static scope; see [this section](./mir/index.html#promoted) for more details. provider | the function that executes a query ([see more](query.html)) -quantified | in math or logic, existential and universal quantification are used to ask questions like "is there any type T for which is true?" or "is this true for all types T?"; see [the background chapter for more](./appendix-background.html#quantified) +quantified | in math or logic, existential and universal quantification are used to ask questions like "is there any type T for which is true?" or "is this true for all types T?"; see [the background chapter for more](./appendix/background.html#quantified) query | perhaps some sub-computation during compilation ([see more](query.html)) region | another term for "lifetime" often used in the literature and in the borrow checker. sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. sigil | like a keyword but composed entirely of non-alphanumeric tokens. For example, `&` is a sigil for references. -skolemization | a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)`) as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on skolemization and universes](./mir-regionck.html#skol) for more details. +skolemization | a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)`) as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on skolemization and universes](./mir/regionck.html#skol) for more details. soundness | soundness is a technical term in type theory. Roughly, if a type system is sound, then if a program type-checks, it is type-safe; i.e. I can never (in safe rust) force a value into a variable of the wrong type. (see "completeness"). span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. substs | the substitutions for a given generic type or item (e.g. the `i32`, `u32` in `HashMap`) tcx | the "typing context", main data structure of the compiler ([see more](ty.html)) 'tcx | the lifetime of the currently active inference context ([see more](ty.html)) -trait reference | the name of a trait along with a suitable set of input type/lifetimes ([see more](./traits-goals-and-clauses.html#trait-ref)) +trait reference | the name of a trait along with a suitable set of input type/lifetimes ([see more](./traits/goals-and-clauses.html#trait-ref)) token | the smallest unit of parsing. Tokens are produced after lexing ([see more](the-parser.html)). [TLS] | Thread-Local Storage. Variables may be defined so that each thread has its own copy (rather than all threads sharing the variable). This has some interactions with LLVM. Not all platforms support TLS. trans | the code to translate MIR into LLVM IR. trait reference | a trait and values for its type parameters ([see more](ty.html)). ty | the internal representation of a type ([see more](ty.html)). UFCS | Universal Function Call Syntax. An unambiguous syntax for calling a method ([see more](type-checking.html)). -variance | variance determines how changes to a generic type/lifetime parameter affect subtyping; for example, if `T` is a subtype of `U`, then `Vec` is a subtype `Vec` because `Vec` is *covariant* in its generic parameter. See [the background chapter](./appendix-background.html#variance) for a more general explanation. See the [variance chapter](./variance.html) for an explanation of how type checking handles variance. +variance | variance determines how changes to a generic type/lifetime parameter affect subtyping; for example, if `T` is a subtype of `U`, then `Vec` is a subtype `Vec` because `Vec` is *covariant* in its generic parameter. See [the background chapter](./appendix/background.html#variance) for a more general explanation. See the [variance chapter](./variance.html) for an explanation of how type checking handles variance. Wide pointer | a pointer with additional metadata. See "fat pointer" for more. [LLVM]: https://llvm.org/ diff --git a/src/mir/borrowck.md b/src/mir/borrowck.md index 51a6c6cbf..cd83843f8 100644 --- a/src/mir/borrowck.md +++ b/src/mir/borrowck.md @@ -21,7 +21,7 @@ Doing borrow checking on MIR has two key advantages: can see [a list of bugs that the MIR-based borrow checker fixes here][47366].) - Even more importantly, using the MIR enables ["non-lexical lifetimes"][nll], - which are regions derived from the control-flow graph. + which are regions derived from the control-flow graph. [47366]: https://github.com/rust-lang/rust/issues/47366 [nll]: http://rust-lang.github.io/rfcs/2094-nll.html @@ -43,17 +43,17 @@ The overall flow of the borrow checker is as follows: include references to the new regions that we are computing. - We then invoke `nll::replace_regions_in_mir` to modify this copy C. Among other things, this function will replace all of the regions in - the MIR with fresh [inference variables](./appendix-glossary.html). - - (More details can be found in [the regionck section](./mir-regionck.html).) + the MIR with fresh [inference variables](./appendix/glossary.html). + - (More details can be found in [the regionck section](./mir/regionck.html).) - Next, we perform a number of [dataflow - analyses](./appendix-background.html#dataflow) + analyses](./appendix/background.html#dataflow) that compute what data is moved and when. The results of these analyses are needed to do both borrow checking and region inference. - Using the move data, we can then compute the values of all the regions in the MIR. - - (More details can be found in [the NLL section](./mir-regionck.html).) + - (More details can be found in [the NLL section](./mir/regionck.html).) - Finally, the borrow checker itself runs, taking as input (a) the results of move analysis and (b) the regions computed by the region checker. This allows us to figure out which loans are still in scope at any particular point. - + diff --git a/src/mir/index.md b/src/mir/index.md index 81bbf6bd6..838048b81 100644 --- a/src/mir/index.md +++ b/src/mir/index.md @@ -26,7 +26,7 @@ Some of the key characteristics of MIR are: - It does not have nested expressions. - All types in MIR are fully explicit. -[cfg]: ./appendix-background.html#cfg +[cfg]: ./appendix/background.html#cfg ## Key MIR vocabulary @@ -244,4 +244,4 @@ but [you can read about those below](#promoted)). [mir]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir [mirmanip]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir [mir]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir -[newtype'd]: appendix-glossary.html +[newtype'd]: appendix/glossary.html diff --git a/src/mir/passes.md b/src/mir/passes.md index 64e72f06e..a5b5df10a 100644 --- a/src/mir/passes.md +++ b/src/mir/passes.md @@ -174,4 +174,4 @@ alternatives in [rust-lang/rust#41710]. [rust-lang/rust#41710]: https://github.com/rust-lang/rust/issues/41710 [mirtransform]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/transform/ [`NoLandingPads`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/transform/no_landing_pads/struct.NoLandingPads.html -[MIR visitor]: mir-visitor.html +[MIR visitor]: mir/visitor.html diff --git a/src/mir/regionck.md b/src/mir/regionck.md index e67e91138..90f6fa274 100644 --- a/src/mir/regionck.md +++ b/src/mir/regionck.md @@ -35,7 +35,7 @@ The MIR-based region analysis consists of two major functions: - More details to come, though the [NLL RFC] also includes fairly thorough (and hopefully readable) coverage. -[fvb]: appendix-background.html#free-vs-bound +[fvb]: appendix/background.html#free-vs-bound [NLL RFC]: http://rust-lang.github.io/rfcs/2094-nll.html ## Universal regions @@ -131,7 +131,7 @@ the type of `foo` the type `bar` expects We handle this sort of subtyping by taking the variables that are bound in the supertype and **skolemizing** them: this means that we replace them with -[universally quantified](appendix-background.html#quantified) +[universally quantified](appendix/background.html#quantified) representatives, written like `!1`. We call these regions "skolemized regions" -- they represent, basically, "some unknown region". @@ -148,7 +148,7 @@ what we wanted. So let's work through what happens next. To check if two functions are subtypes, we check if their arguments have the desired relationship -(fn arguments are [contravariant](./appendix-background.html#variance), so +(fn arguments are [contravariant](./appendix/background.html#variance), so we swap the left and right here): ```text @@ -187,7 +187,7 @@ Here, the root universe would consist of the lifetimes `'static` and the same concept to types, in which case the types `Foo` and `T` would be in the root universe (along with other global types, like `i32`). Basically, the root universe contains all the names that -[appear free](./appendix-background.html#free-vs-bound) in the body of `bar`. +[appear free](./appendix/background.html#free-vs-bound) in the body of `bar`. Now let's extend `bar` a bit by adding a variable `x`: diff --git a/src/traits/canonicalization.md b/src/traits/canonicalization.md index 37a59026b..f87de5f81 100644 --- a/src/traits/canonicalization.md +++ b/src/traits/canonicalization.md @@ -41,7 +41,7 @@ trait query: `?A: Foo<'static, ?B>`, where `?A` and `?B` are unbound. This query contains two unbound variables, but it also contains the lifetime `'static`. The trait system generally ignores all lifetimes and treats them equally, so when canonicalizing, we will *also* -replace any [free lifetime](./appendix-background.html#free-vs-bound) with a +replace any [free lifetime](./appendix/background.html#free-vs-bound) with a canonical variable. Therefore, we get the following result: ```text diff --git a/src/chalk-overview.md b/src/traits/chalk-overview.md similarity index 95% rename from src/chalk-overview.md rename to src/traits/chalk-overview.md index 76c119a5e..e737edcd9 100644 --- a/src/chalk-overview.md +++ b/src/traits/chalk-overview.md @@ -131,18 +131,18 @@ See [The SLG Solver][slg]. [rustc-issues]: https://github.com/rust-lang-nursery/rustc-guide/issues [chalk]: https://github.com/rust-lang-nursery/chalk -[lowering-to-logic]: traits-lowering-to-logic.html -[lowering-rules]: traits-lowering-rules.html +[lowering-to-logic]: traits/lowering-to-logic.html +[lowering-rules]: traits/lowering-rules.html [ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree [chalk-ast]: https://github.com/rust-lang-nursery/chalk/blob/master/chalk-parse/src/ast.rs [universal quantification]: https://en.wikipedia.org/wiki/Universal_quantification -[lowering-forall]: https://rust-lang-nursery.github.io/rustc-guide/traits-lowering-to-logic.html#type-checking-generic-functions-beyond-horn-clauses +[lowering-forall]: ./traits/lowering-to-logic.html#type-checking-generic-functions-beyond-horn-clauses [programclause]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L721 [clause]: https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md#clause -[goals-and-clauses]: traits-goals-and-clauses.html +[goals-and-clauses]: ./traits/goals-and-clauses.html [well-formedness-checks]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir/lowering.rs#L230-L232 [ir-code]: https://github.com/rust-lang-nursery/chalk/blob/master/src/ir.rs [HIR]: hir.html [binders-struct]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L661 [rules-environment]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/rules.rs#L9 -[slg]: traits-slg.html +[slg]: ./traits/slg.html diff --git a/src/traits/index.md b/src/traits/index.md index 6aad2474d..2dabd0b69 100644 --- a/src/traits/index.md +++ b/src/traits/index.md @@ -4,7 +4,7 @@ [process of being implemented][wg]; this chapter serves as a kind of in-progress design document. If you would prefer to read about how the current trait solver works, check out -[this other chapter](./trait-resolution.html). (By the way, if you +[this other chapter](./traits/resolution.html). (By the way, if you would like to help in hacking on the new solver, you will find instructions for getting involved in the [Traits Working Group tracking issue][wg].) 🚧 diff --git a/src/trait-specialization.md b/src/traits/specialization.md similarity index 100% rename from src/trait-specialization.md rename to src/traits/specialization.md diff --git a/src/type-checking.md b/src/type-checking.md index cb6d346e4..9a161abd2 100644 --- a/src/type-checking.md +++ b/src/type-checking.md @@ -6,7 +6,7 @@ draws heavily on the [type inference] and [trait solving].) [typeck]: https://github.com/rust-lang/rust/tree/master/src/librustc_typeck [type inference]: type-inference.html -[trait solving]: trait-resolution.html +[trait solving]: traits/resolution.html ## Type collection diff --git a/src/variance.md b/src/variance.md index 08399b5b3..05a286c07 100644 --- a/src/variance.md +++ b/src/variance.md @@ -2,7 +2,7 @@ For a more general background on variance, see the [background] appendix. -[background]: ./appendix-background.html +[background]: ./appendix/background.html During type checking we must infer the variance of type and lifetime parameters. The algorithm is taken from Section 4 of the paper ["Taming the From 37c68dbbdd9a1c38ecbba88d5cf5c30e4aba2fb1 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 23 May 2018 12:38:46 -0500 Subject: [PATCH 245/648] allow long relative links --- ci/check_line_lengths.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/check_line_lengths.sh b/ci/check_line_lengths.sh index dc48091c8..91f199b7e 100755 --- a/ci/check_line_lengths.sh +++ b/ci/check_line_lengths.sh @@ -30,7 +30,7 @@ for file in "$@" ; do (( inside_block = !$inside_block )) continue fi - if ! (( $inside_block )) && ! [[ "$line" =~ " | "|"-|-"|"://"|\[\^[^\ ]+\]: ]] && (( "${#line}" > $MAX_LINE_LENGTH )) ; then + if ! (( $inside_block )) && ! [[ "$line" =~ " | "|"-|-"|"://"|"]:"|\[\^[^\ ]+\]: ]] && (( "${#line}" > $MAX_LINE_LENGTH )) ; then (( bad_lines++ )) echo -e "\t$line_no : $line" fi From 4a8412f3af6cdaa5a64781e2a1fdec54ad5fa181 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 4 Jun 2018 19:42:16 -0500 Subject: [PATCH 246/648] add a bunch of type-related terms to glossary --- src/appendix/glossary.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index ad2604e63..a6f4e7b1c 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -12,11 +12,14 @@ bound variable | a "bound variable" is one that is declared within an codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. completeness | completeness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). control-flow graph | a representation of the control-flow of a program; see [the background chapter for more](./appendix/background.html#cfg) +CTFE | Compile-Time Function Evaluation. This is the ability of the compiler to evaluate `const fn`s at compile time. This is part of the compiler's constant evaluation system. ([see more](./const-eval.html)) cx | we tend to use "cx" as an abbrevation for context. See also `tcx`, `infcx`, etc. DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](incremental-compilation.html)) data-flow analysis | a static analysis that figures out what properties are true at each point in the control-flow of a program; see [the background chapter for more](./appendix/background.html#dataflow) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. Double pointer | a pointer with additional metadata. See "fat pointer" for more. +DST | Dynamically-Sized Type. A type for which the compiler cannot statically know the size in memory (e.g. `str` or `[u8]`). Such types don't implement `Sized` and cannot be allocated on the stack. They can only occur as the last field in a struct. They can only be used behind a pointer (e.g. `&str` or `&[u8]`). +empty type | see "uninhabited type". Fat pointer | a two word value carrying the address of some value, along with some further information necessary to put the value to use. Rust includes two kinds of "fat pointers": references to slices, and trait objects. A reference to a slice carries the starting address of the slice and its length. A trait object carries a value's address and a pointer to the trait's implementation appropriate to that value. "Fat pointers" are also known as "wide pointers", and "double pointers". free variable | a "free variable" is one that is not bound within an expression or term; see [the background chapter for more](./appendix/background.html#free-vs-bound) 'gcx | the lifetime of the global arena ([see more](ty.html)) @@ -45,6 +48,7 @@ provider | the function that executes a query ([see more](query. quantified | in math or logic, existential and universal quantification are used to ask questions like "is there any type T for which is true?" or "is this true for all types T?"; see [the background chapter for more](./appendix/background.html#quantified) query | perhaps some sub-computation during compilation ([see more](query.html)) region | another term for "lifetime" often used in the literature and in the borrow checker. +rib | a data structure in the name resolver that keeps track of a single scope for names. ([see more](./name-resolution.html)) sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. sigil | like a keyword but composed entirely of non-alphanumeric tokens. For example, `&` is a sigil for references. @@ -61,8 +65,11 @@ trans | the code to translate MIR into LLVM IR. trait reference | a trait and values for its type parameters ([see more](ty.html)). ty | the internal representation of a type ([see more](ty.html)). UFCS | Universal Function Call Syntax. An unambiguous syntax for calling a method ([see more](type-checking.html)). +uninhabited type | a type which has _no_ values. This is not the same as a ZST, which has exactly 1 value. An example of an uninhabited type is `enum Foo {}`, which has no variants, and so, can never be created. The compiler can treat code that deals with uninhabited types as dead code, since there is no such value to be manipulated. `!` (the never type) is an uninhabited type. Uninhabited types are also called "empty types". +upvar | a variable captured by a closure from outside the closure. variance | variance determines how changes to a generic type/lifetime parameter affect subtyping; for example, if `T` is a subtype of `U`, then `Vec` is a subtype `Vec` because `Vec` is *covariant* in its generic parameter. See [the background chapter](./appendix/background.html#variance) for a more general explanation. See the [variance chapter](./variance.html) for an explanation of how type checking handles variance. Wide pointer | a pointer with additional metadata. See "fat pointer" for more. +ZST | Zero-Sized Type. A type whose values have size 0 bytes. Since `2^0 = 1`, such types can have exactly one value. For example, `()` (unit) is a ZST. `struct Foo;` is also a ZST. The compiler can do some nice optimizations around ZSTs. [LLVM]: https://llvm.org/ [lto]: https://llvm.org/docs/LinkTimeOptimization.html From 809ca6900136db5645fcdfbbf8dd95377230daab Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 4 Jun 2018 20:26:40 -0500 Subject: [PATCH 247/648] A few more data structures to the code index --- src/appendix/code-index.md | 18 ++++++++++++++++-- src/hir.md | 2 ++ src/the-parser.md | 10 +++++----- 3 files changed, 23 insertions(+), 7 deletions(-) diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index 915d00828..62fa6bd93 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -6,24 +6,38 @@ compiler. Item | Kind | Short description | Chapter | Declaration ----------------|----------|-----------------------------|--------------------|------------------- +`BodyId` | struct | One of four types of HIR node identifiers. | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.BodyId.html) `CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser] | [src/libsyntax/codemap.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html) `CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver] | [src/librustc_driver/driver.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/struct.CompileState.html) +`DefId` | struct | One of four types of HIR node identifiers. | [Identifiers in the HIR] | [src/librustc/hir/def_id.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/def_id/struct.DefId.html) `DiagnosticBuilder` | struct | A struct for building up compiler diagnostics, such as errors or lints | [Emitting Diagnostics] | [src/librustc_errors/diagnostic_builder.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.DiagnosticBuilder.html) `DocContext` | struct | A state container used by rustdoc when crawling through a crate to gather its documentation | [Rustdoc] | [src/librustdoc/core.rs](https://github.com/rust-lang/rust/blob/master/src/librustdoc/core.rs) `ast::Crate` | struct | Syntax-level representation of a parsed crate | [The parser] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Crate.html) +`HirId` | struct | One of four types of HIR node identifiers. | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.HirId.html) `hir::Crate` | struct | More abstract, compiler-friendly form of a crate's AST | [The Hir] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Crate.html) -`ParseSess` | struct | This struct contains information about a parsing session | [the Parser] | [src/libsyntax/parse/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/struct.ParseSess.html) -`Session` | struct | The data associated with a compilation session | [the Parser], [The Rustc Driver] | [src/librustc/session/mod.html](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html) +`NodeId` | struct | One of four types of HIR node identifiers. Being phased out. | [Identifiers in the HIR] | [src/libsyntax/ast.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.NodeId.html) +`ParamEnv` | struct | Information about generic parameters or `Self`, useful for working with associated or generic items | [Parameter Environment] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.ParamEnv.html) +`ParseSess` | struct | This struct contains information about a parsing session | [The parser] | [src/libsyntax/parse/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/struct.ParseSess.html) +`Rib` | struct | Represents a single scope of names | [Name resolution] | [src/librustc_resolve/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/struct.Rib.html) +`Session` | struct | The data associated with a compilation session | [The parser], [The Rustc Driver] | [src/librustc/session/mod.html](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html) `Span` | struct | A location in the user's source code, used for error reporting primarily | [Emitting Diagnostics] | [src/libsyntax_pos/span_encoding.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax_pos/struct.Span.html) `StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser] | [src/libsyntax/parse/lexer/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/lexer/struct.StringReader.html) +`syntax::token_stream::TokenStream` | struct | An abstract sequence of tokens, organized into `TokenTree`s | [The parser], [Macro expansion] | [src/libsyntax/tokenstream.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/tokenstream/struct.TokenStream.html) `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules] | [src/librustc/ty/trait_def.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/trait_def/struct.TraitDef.html) +`TraitRef` | struct | Information about a trait reference (e.g. `P0: Trait`) | [Trait Solving: Goals and Clauses], [Trait Solving: Lowering impls] | [src/librustc/ty/sty.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TraitRef.html) `Ty<'tcx>` | struct | This is the internal representation of a type used for type checking | [Type checking] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/type.Ty.html) `TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules] | [src/librustc/ty/context.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html) [The HIR]: hir.html +[Identifiers in the HIR]: hir.html#hir-id [The parser]: the-parser.html [The Rustc Driver]: rustc-driver.html [Type checking]: type-checking.html [The `ty` modules]: ty.html [Rustdoc]: rustdoc.html [Emitting Diagnostics]: diag.html +[Macro expansion]: macro-expansion.html +[Name resolution]: name-resolution.html +[Parameter Environment]: param_env.html +[Trait Solving: Goals and Clauses]: traits/goals-and-clauses.html#domain-goals +[Trait Solving: Lowering impls]: traits/lowering-rules.html#lowering-impls diff --git a/src/hir.md b/src/hir.md index 44c4968b1..2a11531ee 100644 --- a/src/hir.md +++ b/src/hir.md @@ -57,6 +57,8 @@ function to lookup the contents of `bar()` given its id; this gives the compiler a chance to observe that you accessed the data for `bar()`, and then record the dependency. + + ### Identifiers in the HIR Most of the code that has to deal with things in HIR tends not to diff --git a/src/the-parser.md b/src/the-parser.md index 623a38e67..dd451d24f 100644 --- a/src/the-parser.md +++ b/src/the-parser.md @@ -2,7 +2,7 @@ The parser is responsible for converting raw Rust source code into a structured form which is easier for the compiler to work with, usually called an [*Abstract -Syntax Tree*][ast]. An AST mirrors the structure of a Rust program in memory, +Syntax Tree*][ast]. An AST mirrors the structure of a Rust program in memory, using a `Span` to link a particular AST node back to its source text. The bulk of the parser lives in the [libsyntax] crate. @@ -27,16 +27,16 @@ in the [parser module]. They let you do things like turn a filemap into a token stream, create a parser from the token stream, and then execute the parser to get a `Crate` (the root AST node). -To minimise the amount of copying that is done, both the `StringReader` and +To minimise the amount of copying that is done, both the `StringReader` and `Parser` have lifetimes which bind them to the parent `ParseSess`. This contains all the information needed while parsing, as well as the `CodeMap` itself. -[libsyntax]: https://github.com/rust-lang/rust/tree/master/src/libsyntax -[rustc_errors]: https://github.com/rust-lang/rust/tree/master/src/librustc_errors +[libsyntax]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/index.html +[rustc_errors]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html [ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree [`CodeMap`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html [ast module]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/index.html -[parser module]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/parse +[parser module]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/index.html [`Parser`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/parser/struct.Parser.html [`StringReader`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/lexer/struct.StringReader.html [visit module]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/visit/index.html From d0ad0c7bc11ed6969e59c5f7ab00c5309d04ef66 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sun, 3 Jun 2018 22:09:21 -0500 Subject: [PATCH 248/648] add a bit about crater --- src/tests/intro.md | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index e8cf34aff..7c3832ed9 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -170,7 +170,38 @@ communicate with the server to coordinate running tests (see ## Crater -TODO +[Crater](https://github.com/rust-lang-nursery/crater) is a tool for compiling +and running tests for _every_ crate on [crates.io](https://crates.io/). It is +mainly used for checking for extent of breakage when implementing potentially +breaking changes. + +### When to run Crater + +You should request a crater run if your PR makes large changes to the compiler +or could cause breakage. If you are unsure, feel free to ask your PR's reviewer. + +### Requesting Crater Runs + +The rust team maintains a few machines that can be used for running crater runs +on the changes introduced by a PR. If your PR needs a crater run, leave a +comment for the triage team in the PR thread. Your will be enqueued by the +triage team and the results will be posted when they are ready. A crater run +usually takes a few days (as of this writing). + +While crater is really useful, it is also important to be aware of a few caveats: + +- Not all code is on crates.io! There is a lot of code in repos on GitHub and + elsewhere. Also, companies may not wish to publish their code. Thus, a + successful crater run is not a magically green light that there will be no + breakage; you still need to be careful. + +- Crater only runs Linux builds (on x86_64, I believe). Thus, other + architectures and platforms are not tested. Critically, this includes + Windows. + +- Many crates are not tested. This could be for a lot of reasons, including + that the crate doesn't compile any more (e.g. used old nightly features), + has broken or flaky tests, requires network access, or other reasons. ## Further reading From 8ee1d91c3824eb17eccd2b8d6ece9b3620d8f971 Mon Sep 17 00:00:00 2001 From: Mark Date: Sun, 3 Jun 2018 22:40:29 -0500 Subject: [PATCH 249/648] add more on crater + perf runs --- src/tests/intro.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/tests/intro.md b/src/tests/intro.md index 7c3832ed9..034b7aac1 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -203,6 +203,23 @@ While crater is really useful, it is also important to be aware of a few caveats that the crate doesn't compile any more (e.g. used old nightly features), has broken or flaky tests, requires network access, or other reasons. +- Before crater can be run, `@bors try` needs to suceed. This means that if + your code doesn't compile or fails tests, you cannot run crater. + +## Perf runs + +A lot of work is put into improving the performance of the compiler and +preventing performance regressions. A "perf run" is used to compare the +performance of the compiler in different configurations for a large collection +of popular crates. Different configurations include "fresh builds", builds +with incremental compilation, etc. + +The result of a perf run is a comparison between two versions of the +compiler (by their commit hashes). + +You should request a perf run if your PR may affect performance, especially +if it can affect performance adversely. + ## Further reading The following blog posts may also be of interest: From eaa2cc194626b7fa7c2ca778fba23ecc750976d3 Mon Sep 17 00:00:00 2001 From: Mark Date: Sun, 3 Jun 2018 22:46:11 -0500 Subject: [PATCH 250/648] Address review feedback --- src/tests/intro.md | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index 034b7aac1..f4164de5e 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -171,9 +171,10 @@ communicate with the server to coordinate running tests (see ## Crater [Crater](https://github.com/rust-lang-nursery/crater) is a tool for compiling -and running tests for _every_ crate on [crates.io](https://crates.io/). It is -mainly used for checking for extent of breakage when implementing potentially -breaking changes. +and running tests for _every_ crate on [crates.io](https://crates.io/) (and a +few on GitHub). It is mainly used for checking for extent of breakage when +implementing potentially breaking changes and ensuring lack of breakage by +running beta vs stable compiler versions. ### When to run Crater @@ -184,9 +185,16 @@ or could cause breakage. If you are unsure, feel free to ask your PR's reviewer. The rust team maintains a few machines that can be used for running crater runs on the changes introduced by a PR. If your PR needs a crater run, leave a -comment for the triage team in the PR thread. Your will be enqueued by the -triage team and the results will be posted when they are ready. A crater run -usually takes a few days (as of this writing). +comment for the triage team in the PR thread. Please inform the team whether +you require a "check-only" crater run, a "build only" crater run, or a +"build-and-test" crater run. The difference is primarily in time; the +conservative (if you're not sure) option is to go for the build-and-test run. +If making changes that will only have an effect at compile-time (e.g., +implementing a new trait) then you only need a check run. + +Your PR will be enqueued by the triage team and the results will be posted when +they are ready. Check runs will take around ~3-4 days, with the other two +taking 5-6 days on average. While crater is really useful, it is also important to be aware of a few caveats: @@ -195,9 +203,8 @@ While crater is really useful, it is also important to be aware of a few caveats successful crater run is not a magically green light that there will be no breakage; you still need to be careful. -- Crater only runs Linux builds (on x86_64, I believe). Thus, other - architectures and platforms are not tested. Critically, this includes - Windows. +- Crater only runs Linux builds on x86_64. Thus, other architectures and + platforms are not tested. Critically, this includes Windows. - Many crates are not tested. This could be for a lot of reasons, including that the crate doesn't compile any more (e.g. used old nightly features), From 35780c196e280e4a4a525a219c6a043b9f0aef28 Mon Sep 17 00:00:00 2001 From: Mark Date: Sun, 3 Jun 2018 22:48:03 -0500 Subject: [PATCH 251/648] long line --- src/tests/intro.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index f4164de5e..79aa0fd6a 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -196,7 +196,8 @@ Your PR will be enqueued by the triage team and the results will be posted when they are ready. Check runs will take around ~3-4 days, with the other two taking 5-6 days on average. -While crater is really useful, it is also important to be aware of a few caveats: +While crater is really useful, it is also important to be aware of a few +caveats: - Not all code is on crates.io! There is a lot of code in repos on GitHub and elsewhere. Also, companies may not wish to publish their code. Thus, a From 390068501f6b433c55136fc181edab40c6ae0da6 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 4 Jun 2018 19:23:19 -0500 Subject: [PATCH 252/648] tests dont need to pass --- src/tests/intro.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index 79aa0fd6a..bd8a60a88 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -211,8 +211,8 @@ caveats: that the crate doesn't compile any more (e.g. used old nightly features), has broken or flaky tests, requires network access, or other reasons. -- Before crater can be run, `@bors try` needs to suceed. This means that if - your code doesn't compile or fails tests, you cannot run crater. +- Before crater can be run, `@bors try` needs to suceed in building artifacts. + This means that if your code doesn't compile, you cannot run crater. ## Perf runs From 63eaf6b11c343cf71ad01df4f96b28788ef8a229 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 4 Jun 2018 19:46:05 -0500 Subject: [PATCH 253/648] fix typo --- src/tests/intro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index bd8a60a88..bc84c9721 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -211,7 +211,7 @@ caveats: that the crate doesn't compile any more (e.g. used old nightly features), has broken or flaky tests, requires network access, or other reasons. -- Before crater can be run, `@bors try` needs to suceed in building artifacts. +- Before crater can be run, `@bors try` needs to succeed in building artifacts. This means that if your code doesn't compile, you cannot run crater. ## Perf runs From b859b33bbe0df3653e3be7a038752fb6b062783e Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 4 Jun 2018 20:40:23 -0500 Subject: [PATCH 254/648] Add bit about ctags; close #80 --- src/how-to-build-and-run.md | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 66c736548..54e60ddcb 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -75,9 +75,9 @@ internally. The result is the compiling `rustc` is done in stages. For hacking, often building the stage 1 compiler is enough, but for final testing and release, the stage 2 compiler is used. -`./x.py check` is really fast to build the rust compiler. -It is, in particular, very useful when you're doing some kind of -"type-based refactoring", like renaming a method, or changing the +`./x.py check` is really fast to build the rust compiler. +It is, in particular, very useful when you're doing some kind of +"type-based refactoring", like renaming a method, or changing the signature of some function. Once you've created a config.toml, you are now ready to run @@ -155,3 +155,21 @@ in other sections: more details): - `./x.py test --stage 1 src/libstd` – runs the `#[test]` tests from libstd - `./x.py test --stage 1 src/test/run-pass` – runs the `run-pass` test suite + +### ctags + +One of the challenges with rustc is that the RLS can't handle it, making code +navigation difficult. One solution is to use `ctags`. The following script can +be used to set it up: [https://github.com/nikomatsakis/rust-etags][etags]. + +CTAGS integrates into emacs and vim quite easily. The following can then be +used to build and generate tags: + +``` +rust-ctags src/lib* && ./x.py build +``` + +This allows you to do "jump-to-def" with whatever functions were around when +you last built, which is ridiculously useful. + +[etags]: https://github.com/nikomatsakis/rust-etags From b43d9881e2794b3bd18c5bfbd4d5f504e793a8d1 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 4 Jun 2018 23:18:46 -0500 Subject: [PATCH 255/648] fix build --- src/how-to-build-and-run.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 54e60ddcb..557fc9148 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -165,8 +165,8 @@ be used to set it up: [https://github.com/nikomatsakis/rust-etags][etags]. CTAGS integrates into emacs and vim quite easily. The following can then be used to build and generate tags: -``` -rust-ctags src/lib* && ./x.py build +```console +$ rust-ctags src/lib* && ./x.py build ``` This allows you to do "jump-to-def" with whatever functions were around when From 7681b6ccc43863b9d627db5bc1ecc9ec48e9731b Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 8 Jun 2018 20:32:04 -0500 Subject: [PATCH 256/648] mention bless --- src/tests/adding.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/tests/adding.md b/src/tests/adding.md index 28f4c1140..fac13fcac 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -155,6 +155,8 @@ source. lint/warning is generated. Then, it applies the suggestion and compares against `.fixed` (they must match). Finally, the fixed source is compiled, and this compilation is required to succeed. + The `.fixed` file can also be generated automatically with the + `--bless` option, discussed [below](#bless). * `min-{gdb,lldb}-version` * `min-llvm-version` * `compile-pass` for UI tests, indicates that the test is @@ -270,6 +272,8 @@ you can even run the resulting program. Just add one of the following - `// run-pass` -- compilation should succeed and we should run the resulting binary + + ### Editing and updating the reference files If you have changed the compiler's output intentionally, or you are From 460342ca9b9cc0d0a27744af30983a68ab3990f7 Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Sat, 9 Jun 2018 00:32:06 +0100 Subject: [PATCH 257/648] removed whitespace --- book.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book.toml b/book.toml index f3a192c67..ecbe513b7 100644 --- a/book.toml +++ b/book.toml @@ -1,7 +1,7 @@ [book] title = "Guide to Rustc Development" author = "Rustc developers" -description = "A guide to developing rustc " +description = "A guide to developing rustc" [output.html] From 318057e479611b174a38a8d5453c7e71514945c6 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 8 Jun 2018 20:38:38 -0500 Subject: [PATCH 258/648] update trait ref --- src/appendix/code-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index 62fa6bd93..ef7853fc6 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -24,7 +24,7 @@ Item | Kind | Short description | Chapter | `StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser] | [src/libsyntax/parse/lexer/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/lexer/struct.StringReader.html) `syntax::token_stream::TokenStream` | struct | An abstract sequence of tokens, organized into `TokenTree`s | [The parser], [Macro expansion] | [src/libsyntax/tokenstream.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/tokenstream/struct.TokenStream.html) `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules] | [src/librustc/ty/trait_def.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/trait_def/struct.TraitDef.html) -`TraitRef` | struct | Information about a trait reference (e.g. `P0: Trait`) | [Trait Solving: Goals and Clauses], [Trait Solving: Lowering impls] | [src/librustc/ty/sty.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TraitRef.html) +`TraitRef` | struct | The combination of a trait and its input types (e.g. `P0: Trait`) | [Trait Solving: Goals and Clauses], [Trait Solving: Lowering impls] | [src/librustc/ty/sty.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TraitRef.html) `Ty<'tcx>` | struct | This is the internal representation of a type used for type checking | [Type checking] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/type.Ty.html) `TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules] | [src/librustc/ty/context.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html) From 529a75575b84dcf95f4f27287fa63ff49012e160 Mon Sep 17 00:00:00 2001 From: scalexm Date: Thu, 7 Jun 2018 12:13:40 +0200 Subject: [PATCH 259/648] Add chalk rules for type defs --- src/traits/lowering-rules.md | 76 +++++++++++++++++++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-) diff --git a/src/traits/lowering-rules.md b/src/traits/lowering-rules.md index f635ee857..e3febd9f2 100644 --- a/src/traits/lowering-rules.md +++ b/src/traits/lowering-rules.md @@ -174,6 +174,80 @@ we must show that `WellFormed(TraitRef)`. This in turn justifies the implied bounds rules that allow us to extend the set of `FromEnv` items. +## Lowering type definitions + +We also want to have some rules which define when a type is well-formed. +For example, given this type: + +```rust,ignore +struct Set where K: Hash { ... } +``` + +then `Set` is well-formed because `i32` implements `Hash`, but +`Set` would not be well-formed. Basically, a type is well-formed +if its parameters verify the where clauses written on the type definition. + +Hence, for every type definition: + +```rust, ignore +struct Type where WC { ... } +``` + +we produce the following rule: + +```text +// Rule WellFormed-Type +forall { + WellFormed(Type) :- WC +} +``` + +Note that we use `struct` for defining a type, but this should be understood +as a general type definition (it could be e.g. a generic `enum`). + +Conversely, we define rules which say that if we assume that a type is +well-formed, we can also assume that its where clauses hold. That is, +we produce the following family of rules: + +```text +// Rule FromEnv-Type +// +// For each where clause `WC` +forall { + FromEnv(WC) :- FromEnv(Type) +} +``` + +As for the implied bounds RFC, functions will *assume* that their arguments +are well-formed. For example, suppose we have the following bit of code: + +```rust,ignore +trait Hash: Eq { } +struct Set { ... } + +fn foo(collection: Set, x: K, y: K) { + // `x` and `y` can be equalized even if we did not explicitly write + // `where K: Eq` + if x == y { + ... + } +} +``` + +in the `foo` function, we assume that `Set` is well-formed, i.e. we have +`FromEnv(Set)` in our environment. Because of the previous rule, we get + `FromEnv(K: Hash)` without needing an explicit where clause. And because +of the `Hash` trait definition, there also exists a rule which says: + +```text +forall { + FromEnv(K: Eq) :- FromEnv(K: Hash) +} +``` + +which means that we finally get `FromEnv(K: Eq)` and then can compare `x` +and `y` without needing an explicit where clause. + ## Lowering trait items @@ -333,4 +407,4 @@ Chalk didn't model functions and constants, but I would eventually like to treat them exactly like normalization. This presumably involves adding a new kind of parameter (constant), and then having a `NormalizeValue` domain goal. This is *to be written* because the -details are a bit up in the air. +details are a bit up in the air. \ No newline at end of file From 7139188c0780612aefb7b2a114ea32c02ec0a44a Mon Sep 17 00:00:00 2001 From: scalexm Date: Tue, 12 Jun 2018 19:27:04 +0200 Subject: [PATCH 260/648] Capitalize a word --- src/traits/lowering-rules.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/traits/lowering-rules.md b/src/traits/lowering-rules.md index e3febd9f2..d8b739890 100644 --- a/src/traits/lowering-rules.md +++ b/src/traits/lowering-rules.md @@ -234,7 +234,7 @@ fn foo(collection: Set, x: K, y: K) { } ``` -in the `foo` function, we assume that `Set` is well-formed, i.e. we have +In the `foo` function, we assume that `Set` is well-formed, i.e. we have `FromEnv(Set)` in our environment. Because of the previous rule, we get `FromEnv(K: Hash)` without needing an explicit where clause. And because of the `Hash` trait definition, there also exists a rule which says: @@ -407,4 +407,4 @@ Chalk didn't model functions and constants, but I would eventually like to treat them exactly like normalization. This presumably involves adding a new kind of parameter (constant), and then having a `NormalizeValue` domain goal. This is *to be written* because the -details are a bit up in the air. \ No newline at end of file +details are a bit up in the air. From c98cd5a6a843cbbc36f5e2db9fb94fa0fb9efcd2 Mon Sep 17 00:00:00 2001 From: "Zack M. Davis" Date: Sat, 2 Jun 2018 21:58:53 -0700 Subject: [PATCH 261/648] include `./` in example x.py commands for smoother copy-paste experience The current directory is typically not on the user's $PATH. --- src/rustdoc.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/rustdoc.md b/src/rustdoc.md index 36195c3a5..5bdfbeb6a 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -33,13 +33,13 @@ does is call the `main()` that's in this crate's `lib.rs`, though.) ## Cheat sheet -* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable +* Use `./x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable rustdoc you can run on other projects. * Add `src/libtest` to be able to use `rustdoc --test`. * If you've used `rustup toolchain link local /path/to/build/$TARGET/stage1` previously, then after the previous build command, `cargo +local doc` will Just Work. -* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the +* Use `./x.py doc --stage 1 src/libstd` to use this rustdoc to generate the standard library docs. * The completed docs will be available in `build/$TARGET/doc/std`, though the bundle is meant to be used as though you would copy out the `doc` folder to From addc94950a3d70c5d0e96cd7238567fc1f834a5b Mon Sep 17 00:00:00 2001 From: "Zack M. Davis" Date: Sun, 24 Jun 2018 19:08:02 -0700 Subject: [PATCH 262/648] don't mention new rustdoc Steve says it's not ready (https://github.com/rust-lang-nursery/rustc-guide/pull/150#issuecomment-395783504). --- src/rustdoc.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/src/rustdoc.md b/src/rustdoc.md index 5bdfbeb6a..76a8b28ce 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -1,10 +1,7 @@ # The walking tour of rustdoc Rustdoc actually uses the rustc internals directly. It lives in-tree with the -compiler and standard library. This chapter is about how it works. (A new -implementation is also [under way], though). - -[under way]: https://github.com/steveklabnik/rustdoc +compiler and standard library. This chapter is about how it works. Rustdoc is implemented entirely within the crate [`librustdoc`][rd]. It runs the compiler up to the point where we have an internal representation of a From 85d30364b066a198d038ec7c02ecc42fe2f830db Mon Sep 17 00:00:00 2001 From: Eduard-Mihai Burtescu Date: Thu, 14 Jun 2018 20:35:35 +0300 Subject: [PATCH 263/648] rustc: rename ty::maps to ty::query. --- src/high-level-overview.md | 2 +- src/query.md | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/src/high-level-overview.md b/src/high-level-overview.md index be396054b..dce6b4c41 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -85,7 +85,7 @@ input, running the type-checker, and so forth. This on-demand model permits us to do exciting things like only do the minimal amount of work needed to type-check a single function. It also helps with incremental compilation. (For details on defining queries, check out -`src/librustc/ty/maps/README.md`.) +`src/librustc/ty/query/README.md`.) Regardless of the general setup, the basic operations that the compiler must perform are the same. The only thing that changes is diff --git a/src/query.md b/src/query.md index d48b6e7e3..62a627135 100644 --- a/src/query.md +++ b/src/query.md @@ -63,7 +63,7 @@ get to use the nice method-call-style syntax. Instead, you invoke using the `try_get` method, which looks roughly like this: ```rust,ignore -use ty::maps::queries; +use ty::queries; ... match queries::type_of::try_get(tcx, DUMMY_SP, self.did) { Ok(result) => { @@ -215,14 +215,14 @@ Well, defining a query takes place in two steps: To specify the query name and arguments, you simply add an entry to the big macro invocation in -[`src/librustc/ty/maps/mod.rs`][maps-mod]. This will probably have +[`src/librustc/ty/query/mod.rs`][query-mod]. This will probably have changed by the time you read this README, but at present it looks something like: -[maps-mod]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/maps/index.html +[query-mod]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/query/index.html ```rust,ignore -define_maps! { <'tcx> +define_queries! { <'tcx> /// Records the type of every item. [] fn type_of: TypeOfItem(DefId) -> Ty<'tcx>, @@ -250,7 +250,7 @@ Let's go over them one by one: processed. - **Name of query:** the name of the query method (`tcx.type_of(..)`). Also used as the name of a struct - (`ty::maps::queries::type_of`) that will be generated to represent + (`ty::queries::type_of`) that will be generated to represent this query. - **Dep-node constructor:** indicates the constructor function that connects this query to incremental compilation. Typically, this is a @@ -262,7 +262,7 @@ Let's go over them one by one: bottom of the file. This is typically used when the query key is not a def-id, or just not the type that the dep-node expects. - **Query key type:** the type of the argument to this query. - This type must implement the `ty::maps::keys::Key` trait, which + This type must implement the `ty::query::keys::Key` trait, which defines (for example) how to map it to a crate, and so forth. - **Result type of query:** the type produced by this query. This type should (a) not use `RefCell` or other interior mutability and (b) be @@ -277,14 +277,14 @@ Let's go over them one by one: So, to add a query: -- Add an entry to `define_maps!` using the format above. +- Add an entry to `define_queries!` using the format above. - Possibly add a corresponding entry to the dep-node macro. - Link the provider by modifying the appropriate `provide` method; or add a new one if needed and ensure that `rustc_driver` is invoking it. #### Query structs and descriptions -For each kind, the `define_maps` macro will generate a "query struct" +For each kind, the `define_queries` macro will generate a "query struct" named after the query. This struct is a kind of a place-holder describing the query. Each such struct implements the `self::config::QueryConfig` trait, which has associated types for the From b1fa3579b8894dad4568148f2b79cb6e72dd5062 Mon Sep 17 00:00:00 2001 From: Eduard-Mihai Burtescu Date: Mon, 25 Jun 2018 07:07:19 +0300 Subject: [PATCH 264/648] Update high-level-overview.md --- src/high-level-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/high-level-overview.md b/src/high-level-overview.md index dce6b4c41..8e49afaa9 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -85,7 +85,7 @@ input, running the type-checker, and so forth. This on-demand model permits us to do exciting things like only do the minimal amount of work needed to type-check a single function. It also helps with incremental compilation. (For details on defining queries, check out -`src/librustc/ty/query/README.md`.) +the [query model].) Regardless of the general setup, the basic operations that the compiler must perform are the same. The only thing that changes is From 7bf9416221cc6ee2ccea4cb5c04dca1de9609666 Mon Sep 17 00:00:00 2001 From: Eduard-Mihai Burtescu Date: Mon, 25 Jun 2018 07:09:22 +0300 Subject: [PATCH 265/648] Update query.md --- src/query.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/query.md b/src/query.md index 62a627135..7ed299b74 100644 --- a/src/query.md +++ b/src/query.md @@ -215,9 +215,7 @@ Well, defining a query takes place in two steps: To specify the query name and arguments, you simply add an entry to the big macro invocation in -[`src/librustc/ty/query/mod.rs`][query-mod]. This will probably have -changed by the time you read this README, but at present it looks -something like: +[`src/librustc/ty/query/mod.rs`][query-mod], which looks something like: [query-mod]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/query/index.html From ec8d0498d6105451623f493d30deb5f5260c612a Mon Sep 17 00:00:00 2001 From: Alex Kitchens Date: Sun, 24 Jun 2018 21:56:01 -0500 Subject: [PATCH 266/648] Rename trans to codegen This commit is a followup of changes from [b63d7e2b1c4019e40051036bcb1fd5f254a8f6e2](https://github.com/rust-lang/rust/commit/b63d7e2b1c4019e40051036bcb1fd5f254a8f6e2#diff-b433c87466d984aa7eeded378ea6c392) in the Rust source to rename trans to codegen. --- src/SUMMARY.md | 2 +- src/appendix/glossary.md | 4 ++-- src/{trans.md => codegen.md} | 0 src/high-level-overview.md | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-) rename src/{trans.md => codegen.md} (100%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 8efe304b7..f60fee488 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -53,7 +53,7 @@ - [Constant evaluation](./const-eval.md) - [miri const evaluator](./miri.md) - [Parameter Environments](./param_env.md) -- [Generating LLVM IR](./trans.md) +- [Generating LLVM IR](./codegen.md) - [Emitting Diagnostics](./diag.md) --- diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index a6f4e7b1c..c2947090a 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -35,7 +35,7 @@ IR | Intermediate Representation. A general term in compil local crate | the crate currently being compiled. LTO | Link-Time Optimizations. A set of optimizations offered by LLVM that occur just before the final binary is linked. These include optmizations like removing functions that are never used in the final program, for example. _ThinLTO_ is a variant of LTO that aims to be a bit more scalable and efficient, but possibly sacrifices some optimizations. You may also read issues in the Rust repo about "FatLTO", which is the loving nickname given to non-Thin LTO. LLVM documentation: [here][lto] and [here][thinlto] [LLVM] | (actually not an acronym :P) an open-source compiler backend. It accepts LLVM IR and outputs native binaries. Various languages (e.g. Rust) can then implement a compiler front-end that output LLVM IR and use LLVM to compile to all the platforms LLVM supports. -MIR | the Mid-level IR that is created after type-checking for use by borrowck and trans ([see more](./mir/index.html)) +MIR | the Mid-level IR that is created after type-checking for use by borrowck and codegen ([see more](./mir/index.html)) miri | an interpreter for MIR used for constant evaluation ([see more](./miri.html)) normalize | a general term for converting to a more canonical form, but in the case of rustc typically refers to [associated type normalization](./traits/associated-types.html#normalize) newtype | a "newtype" is a wrapper around some other type (e.g., `struct Foo(T)` is a "newtype" for `T`). This is commonly used in Rust to give a stronger type for indices. @@ -61,7 +61,7 @@ tcx | the "typing context", main data structure of the comp trait reference | the name of a trait along with a suitable set of input type/lifetimes ([see more](./traits/goals-and-clauses.html#trait-ref)) token | the smallest unit of parsing. Tokens are produced after lexing ([see more](the-parser.html)). [TLS] | Thread-Local Storage. Variables may be defined so that each thread has its own copy (rather than all threads sharing the variable). This has some interactions with LLVM. Not all platforms support TLS. -trans | the code to translate MIR into LLVM IR. +codegen | the code to translate MIR into LLVM IR. trait reference | a trait and values for its type parameters ([see more](ty.html)). ty | the internal representation of a type ([see more](ty.html)). UFCS | Universal Function Call Syntax. An unambiguous syntax for calling a method ([see more](type-checking.html)). diff --git a/src/trans.md b/src/codegen.md similarity index 100% rename from src/trans.md rename to src/codegen.md diff --git a/src/high-level-overview.md b/src/high-level-overview.md index 8e49afaa9..60b9e80ea 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -8,7 +8,7 @@ standard library and the compiler. This document, of course, focuses on the latter. Rustc consists of a number of crates, including `syntax`, -`rustc`, `rustc_back`, `rustc_trans`, `rustc_driver`, and +`rustc`, `rustc_back`, `rustc_codegen`, `rustc_driver`, and many more. The source for each crate can be found in a directory like `src/libXXX`, where `XXX` is the crate name. @@ -25,7 +25,7 @@ The dependency structure of these crates is roughly a diamond: / | \ / | \ / v \ -rustc_trans rustc_borrowck ... rustc_metadata +rustc_codegen rustc_borrowck ... rustc_metadata \ | / \ | / \ | / From 37872481acc1ee3212782fd54de913607709642c Mon Sep 17 00:00:00 2001 From: Alex Kitchens Date: Tue, 26 Jun 2018 11:28:30 -0500 Subject: [PATCH 267/648] Keep glossary definition of codegen --- src/appendix/glossary.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index c2947090a..bfc2c0d22 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -9,6 +9,7 @@ Term | Meaning AST | the abstract syntax tree produced by the syntax crate; reflects user syntax very closely. binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and \|`a`\|` ...` is a binder for the parameter `a`. See [the background chapter for more](./appendix/background.html#free-vs-bound) bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession \|`a`\|` a * 2`. See [the background chapter for more](./appendix/background.html#free-vs-bound) +codegen | the code to translate MIR into LLVM IR. codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. completeness | completeness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). control-flow graph | a representation of the control-flow of a program; see [the background chapter for more](./appendix/background.html#cfg) @@ -61,7 +62,7 @@ tcx | the "typing context", main data structure of the comp trait reference | the name of a trait along with a suitable set of input type/lifetimes ([see more](./traits/goals-and-clauses.html#trait-ref)) token | the smallest unit of parsing. Tokens are produced after lexing ([see more](the-parser.html)). [TLS] | Thread-Local Storage. Variables may be defined so that each thread has its own copy (rather than all threads sharing the variable). This has some interactions with LLVM. Not all platforms support TLS. -codegen | the code to translate MIR into LLVM IR. +trans | the code to translate MIR into LLVM IR. Renamed to codegen. trait reference | a trait and values for its type parameters ([see more](ty.html)). ty | the internal representation of a type ([see more](ty.html)). UFCS | Universal Function Call Syntax. An unambiguous syntax for calling a method ([see more](type-checking.html)). From 95bdeacaeaa73bb13deeaa25da90b6926fd57dc0 Mon Sep 17 00:00:00 2001 From: "Zack M. Davis" Date: Sat, 30 Jun 2018 08:34:31 -0700 Subject: [PATCH 268/648] add links for `Span`, `CodeMap`, and `rustfix` It's unfortunate that `code-monospaced` links don't render with link colors (such that the reader needs to hover over them just to tell that it is a link), but that's presumably a bug in MdBook, and not something we need concern ourselves with here. --- src/diag.md | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/src/diag.md b/src/diag.md index 6043a79b7..389ba2bbc 100644 --- a/src/diag.md +++ b/src/diag.md @@ -5,14 +5,17 @@ This chapter is about how to emit compile errors and lints from the compiler. ## `Span` -`Span` is the primary data structure in `rustc` used to represent a location in -the code being compiled. `Span`s are attached to most constructs in HIR and MIR, -allowing for easier error reporting whenever an error comes up. +[`Span`][span] is the primary data structure in `rustc` used to represent a +location in the code being compiled. `Span`s are attached to most constructs in +HIR and MIR, allowing for easier error reporting whenever an error comes up. -A `Span` can be looked up in a `CodeMap` to get a "snippet" useful for -displaying errors with [`span_to_snippet` and other similar methods][sptosnip] -on the `CodeMap`. +[span]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.Span.html +A `Span` can be looked up in a [`CodeMap`][codemap] to get a "snippet" useful +for displaying errors with [`span_to_snippet`][sptosnip] and other similar +methods on the `CodeMap`. + +[codemap]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html [sptosnip]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html#method.span_to_snippet ## Error messages @@ -70,14 +73,16 @@ err.emit(); ## Suggestions We would like to make edition transitions as smooth as possible. To that end, -`rustfix` can use compiler suggestions to automatically fix code. For example, -we could use `rustfix` to mechanically apply the `qux` suggestion from the -previous example. However, not all suggestions are mechanically applicable. We -use the [`span_suggestion_with_applicability`][sswa] method of -`DiagnosticBuilder` to inform the emitter of whether a suggestion is -mechanically applicable or not. This information, in turn, is outputed by -rustc when the error format is `json`, which is used by `rustfix`. - +[`rustfix`][rustfix] can use compiler suggestions to automatically fix +code. For example, we could use `rustfix` to mechanically apply the `qux` +suggestion from the previous example. However, not all suggestions are +mechanically applicable. We use the +[`span_suggestion_with_applicability`][sswa] method of `DiagnosticBuilder` to +inform the emitter of whether a suggestion is mechanically applicable or not. +This information, in turn, is outputed by rustc when the error format is +`json`, which is used by `rustfix`. + +[rustfix]: https://github.com/rust-lang-nursery/rustfix/ [sswa]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.DiagnosticBuilder.html#method.span_suggestion_with_applicability For example, to make our `qux` suggestion machine-applicable, we would do: From 14ef432e4829f0bd47a44a6caf9552f50b763bb2 Mon Sep 17 00:00:00 2001 From: "Zack M. Davis" Date: Sat, 30 Jun 2018 08:37:15 -0700 Subject: [PATCH 269/648] `span_to_snippet` return value is a `Result`, not an `Option` --- src/diag.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/diag.md b/src/diag.md index 389ba2bbc..5984e8621 100644 --- a/src/diag.md +++ b/src/diag.md @@ -56,7 +56,7 @@ let mut err = sess.struct_span_err(sp, "oh no! this is an error!"); // In some cases, you might need to check if `sp` is generated by a macro to // avoid printing weird errors about macro-generated code. -if let Some(snippet) = sess.codemap().span_to_snippet(sp) { +if let Ok(snippet) = sess.codemap().span_to_snippet(sp) { // Use the snippet to generate a suggested fix err.span_suggestion(suggestion_sp, "try using a qux here", format!("qux {}", snip)); } else { @@ -90,7 +90,7 @@ For example, to make our `qux` suggestion machine-applicable, we would do: ```rust,ignore let mut err = sess.struct_span_err(sp, "oh no! this is an error!"); -if let Some(snippet) = sess.codemap().span_to_snippet(sp) { +if let Ok(snippet) = sess.codemap().span_to_snippet(sp) { // Add applicability info! err.span_suggestion_with_applicability( suggestion_sp, From c345de354ca53d138d75d976c320dfde8151bde6 Mon Sep 17 00:00:00 2001 From: "Zack M. Davis" Date: Sat, 30 Jun 2018 08:50:28 -0700 Subject: [PATCH 270/648] =?UTF-8?q?"easier"=20=E2=86=92=20"more=20informat?= =?UTF-8?q?ive";=20omit=20needless=20clause?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit We don't want to leave the impression that spans are about making it easier for compiler developers to perform the arduous task of emitting an error; spans are about pointing to particular segments of code in the emitted error messages. Also, we don't need to say "whenever an error comes up"; that's implied by the phrase "error reporting." --- src/diag.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/diag.md b/src/diag.md index 5984e8621..0bef3e274 100644 --- a/src/diag.md +++ b/src/diag.md @@ -7,7 +7,7 @@ This chapter is about how to emit compile errors and lints from the compiler. [`Span`][span] is the primary data structure in `rustc` used to represent a location in the code being compiled. `Span`s are attached to most constructs in -HIR and MIR, allowing for easier error reporting whenever an error comes up. +HIR and MIR, allowing for more informative error reporting. [span]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.Span.html From 2cb56f31fef588e2d7fd5acbbe8b3de6edc48385 Mon Sep 17 00:00:00 2001 From: "Zack M. Davis" Date: Sat, 30 Jun 2018 08:55:04 -0700 Subject: [PATCH 271/648] cautionary parenthetical about failing to emit a `DiagnosticBuilder` --- src/diag.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/diag.md b/src/diag.md index 0bef3e274..26fc200af 100644 --- a/src/diag.md +++ b/src/diag.md @@ -41,13 +41,15 @@ directly and ones that allow finer control over what to emit. For example, [`DiagnosticBuilder`][diagbuild]. `DiagnosticBuilder` allows you to add related notes and suggestions to an error -before emitting it by calling the [`emit`][emit] method. See the +before emitting it by calling the [`emit`][emit] method. (Failing to either +emit or [cancel][cancel] a `DiagnosticBuilder` will result in an ICE.) See the [docs][diagbuild] for more info on what you can do. [spanerr]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html#method.span_err [strspanerr]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html#method.struct_span_err [diagbuild]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/diagnostic_builder/struct.DiagnosticBuilder.html [emit]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/diagnostic_builder/struct.DiagnosticBuilder.html#method.emit +[cancel]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.Diagnostic.html#method.cancel ```rust,ignore // Get a DiagnosticBuilder. This does _not_ emit an error yet. From 85645644db2461375a9239f0fd9a392c54353e26 Mon Sep 17 00:00:00 2001 From: "Zack M. Davis" Date: Sat, 30 Jun 2018 09:03:00 -0700 Subject: [PATCH 272/648] mention lint-level command line flags as well as attributes --- src/diag.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/diag.md b/src/diag.md index 26fc200af..e4c257fd8 100644 --- a/src/diag.md +++ b/src/diag.md @@ -261,4 +261,5 @@ For example, ``` This defines the `nonstandard_style` group which turns on the listed lints. A -user can turn on these lints by using `!#[warn(nonstandard_style)]`. +user can turn on these lints with a `!#[warn(nonstandard_style)]` attribute in +the source code, or by passing `-W nonstandard-style` on the command line. From 196d10ea87c55bda3e5d22313bf62e81b6ad0cfe Mon Sep 17 00:00:00 2001 From: "Zack M. Davis" Date: Sat, 30 Jun 2018 16:55:13 -0700 Subject: [PATCH 273/648] rewrite suggestions intro to be less bizarrely edition/rustfix centric The suggestions API was introduced in April 2015 (rust-lang/rust@906a9728ff), long predating rustfix (initial commit July 2016) or editions (RFC 2052 approved September 2017). --- src/diag.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/src/diag.md b/src/diag.md index e4c257fd8..a0e755718 100644 --- a/src/diag.md +++ b/src/diag.md @@ -74,17 +74,21 @@ err.emit(); ## Suggestions -We would like to make edition transitions as smooth as possible. To that end, -[`rustfix`][rustfix] can use compiler suggestions to automatically fix -code. For example, we could use `rustfix` to mechanically apply the `qux` -suggestion from the previous example. However, not all suggestions are -mechanically applicable. We use the +In addition to telling the user exactly _why_ their code is wrong, it's +oftentimes furthermore possible to tell them how to fix it. To this end, +`DiagnosticBuilder` offers a structured suggestions API, which formats code +suggestions pleasingly in the terminal, or (when the `--error-format json` flag +is passed) as JSON for consumption by tools, most notably the [Rust Language +Server][rls] and [`rustfix`][rustfix]. + +[rls]: https://github.com/rust-lang-nursery/rls +[rustfix]: https://github.com/rust-lang-nursery/rustfix + +Not all suggestions should be applied mechanically. Use the [`span_suggestion_with_applicability`][sswa] method of `DiagnosticBuilder` to -inform the emitter of whether a suggestion is mechanically applicable or not. -This information, in turn, is outputed by rustc when the error format is -`json`, which is used by `rustfix`. +make a suggestion while providing a hint to tools whether the suggestion is +mechanically applicable or not. -[rustfix]: https://github.com/rust-lang-nursery/rustfix/ [sswa]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.DiagnosticBuilder.html#method.span_suggestion_with_applicability For example, to make our `qux` suggestion machine-applicable, we would do: From 79ebdb7b53a21cdc237159e9a9f856ee2b629c5d Mon Sep 17 00:00:00 2001 From: "Zack M. Davis" Date: Sat, 30 Jun 2018 17:03:08 -0700 Subject: [PATCH 274/648] =?UTF-8?q?"and=20has"=20=E2=86=92=20"because=20it?= =?UTF-8?q?=20has"?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- src/diag.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/diag.md b/src/diag.md index a0e755718..67bd4829c 100644 --- a/src/diag.md +++ b/src/diag.md @@ -148,8 +148,9 @@ For more information about this error, try `rustc --explain E0999`. There are a few other [`Applicability`][appl] possibilities: - `MachineApplicable`: Can be applied mechanically. -- `HasPlaceholders`: Cannot be applied mechanically and has placeholder text in - the suggestions. For example, "Try adding a type: \`let x: \\`". +- `HasPlaceholders`: Cannot be applied mechanically because it has placeholder + text in the suggestions. For example, "Try adding a type: \`let x: + \\`". - `MaybeIncorrect`: Cannot be applied mechanically because the suggestion may or may not be a good one. - `Unspecified`: Cannot be applied mechanically because we don't know which From 7fe7f7401c7653b6cdd02a74d967d0745fbbcfee Mon Sep 17 00:00:00 2001 From: mark Date: Wed, 4 Jul 2018 17:17:20 -0500 Subject: [PATCH 275/648] add a link to the rustc docs --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 07d1749e8..5dc64ed4d 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,10 @@ some new part of the compiler that they haven't worked on before. [You can read the latest version of the guide here.](https://rust-lang-nursery.github.io/rustc-guide/) +You may also find the rustdocs [for the compiler itself][rustdocs] useful. + +[rustdocs]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ + The guide can be useful today, but it has a lot of work still go. Once it gets more complete, the plan is probably to move it into the [main Rust repository](https://github.com/rust-lang/rust/). From f394cd109c99f62304766a7895ce79d9334c1fe5 Mon Sep 17 00:00:00 2001 From: Oliver Schneider Date: Fri, 1 Jun 2018 17:34:45 +0200 Subject: [PATCH 276/648] Add some explanation of lowering ids --- src/high-level-overview.md | 3 ++- src/lowering.md | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+), 1 deletion(-) create mode 100644 src/lowering.md diff --git a/src/high-level-overview.md b/src/high-level-overview.md index 60b9e80ea..733c523c9 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -105,7 +105,7 @@ take: 3. **Lowering to HIR** - Once name resolution completes, we convert the AST into the HIR, or "[high-level intermediate representation]". The HIR is defined in - `src/librustc/hir/`; that module also includes the lowering code. + `src/librustc/hir/`; that module also includes the [lowering] code. - The HIR is a lightly desugared variant of the AST. It is more processed than the AST and more suitable for the analyses that follow. It is **not** required to match the syntax of the Rust language. @@ -139,3 +139,4 @@ take: [query model]: query.html [high-level intermediate representation]: hir.html +[lowering]: lowering.html \ No newline at end of file diff --git a/src/lowering.md b/src/lowering.md new file mode 100644 index 000000000..335513342 --- /dev/null +++ b/src/lowering.md @@ -0,0 +1,35 @@ +# Lowering + +The lowering step converts AST to [HIR](hir.html). +This means many structures are removed if they are irrelevant +for type analysis or similar syntax agnostic analyses. Examples +of such structures include but are not limited to + +* Parenthesis + * Removed without replacement, the tree structure makes order explicit +* `for` loops and `while (let)` loops + * Converted to `loop` + `match` and some `let` bindings +* `if let` + * Converted to `match` +* Universal `impl Trait` + * Converted to generic arguments (but with some flags, to know that the user didn't write them) +* Existential `impl Trait` + * Converted to a virtual `existential type` declaration + +Lowering needs to uphold several invariants in order to not trigger the +sanity checks in `src/librustc/hir/map/hir_id_validator.rs`: + +1. A `HirId` must be used if created. So if you use the `lower_node_id`, + you *must* use the resulting `NodeId` or `HirId` (either is fine, since + any `NodeId`s in the `HIR` are checked for existing `HirId`s) +2. Lowering a `HirId` must be done in the scope of the *owning* item. + This means you need to use `with_hir_id_owner` if you are creating parts + of another item than the one being currently lowered. This happens for + example during the lowering of existential `impl Trait` +3. A `NodeId` that will be placed into a HIR structure must be lowered, + even if its `HirId` is unused. Calling + `let _ = self.lower_node_id(node_id);` is perfectly legitimate. +4. If you are creating new nodes that didn't exist in the `AST`, you *must* + create new ids for them. This is done by calling the `next_id` method, + which produces both a new `NodeId` as well as automatically lowering it + for you so you also get the `HirId`. \ No newline at end of file From e8a0108259409b2f4513f9a3b68f5acd18ede4b3 Mon Sep 17 00:00:00 2001 From: Oliver Schneider Date: Thu, 5 Jul 2018 09:30:05 +0200 Subject: [PATCH 277/648] Address review --- src/SUMMARY.md | 1 + src/hir.md | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index f60fee488..a41f78a1a 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -20,6 +20,7 @@ - [Macro expansion](./macro-expansion.md) - [Name resolution](./name-resolution.md) - [The HIR (High-level IR)](./hir.md) + - [Lowering AST to HIR](./lowering.md) - [The `ty` module: representing types](./ty.md) - [Type inference](./type-inference.md) - [Trait solving (old-style)](./traits/resolution.md) diff --git a/src/hir.md b/src/hir.md index 2a11531ee..0d6a6fbff 100644 --- a/src/hir.md +++ b/src/hir.md @@ -3,7 +3,8 @@ The HIR – "High-Level Intermediate Representation" – is the primary IR used in most of rustc. It is a compiler-friendly representation of the abstract syntax tree (AST) that is generated after parsing, macro expansion, and name -resolution. Many parts of HIR resemble Rust surface syntax quite closely, with +resolution (see [Lowering](./lowering.md) for how the HIR is created). +Many parts of HIR resemble Rust surface syntax quite closely, with the exception that some of Rust's expression forms have been desugared away. For example, `for` loops are converted into a `loop` and do not appear in the HIR. This makes HIR more amenable to analysis than a normal AST. From f2e7ef167512f496cce7d30953a629e563a4ccde Mon Sep 17 00:00:00 2001 From: Oliver Schneider Date: Thu, 5 Jul 2018 09:34:37 +0200 Subject: [PATCH 278/648] Add some details about `DefId` creation --- src/lowering.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/src/lowering.md b/src/lowering.md index 335513342..cc2a7b384 100644 --- a/src/lowering.md +++ b/src/lowering.md @@ -32,4 +32,16 @@ sanity checks in `src/librustc/hir/map/hir_id_validator.rs`: 4. If you are creating new nodes that didn't exist in the `AST`, you *must* create new ids for them. This is done by calling the `next_id` method, which produces both a new `NodeId` as well as automatically lowering it - for you so you also get the `HirId`. \ No newline at end of file + for you so you also get the `HirId`. + +If you are creating new `DefId`s, since each `DefId` needs to have a corresponding +`NodeId`, it is adviseable to add these `NodeId`s to the `AST` so you don't have +to generate new ones during lowering. This has the advantage of creating a +way to find the `DefId` of something via its `NodeId`. If lowering needs this +`DefId` in multiple places, you can't generate a new `NodeId` in all those places +because you'd also get a new `DefId` then. With a `NodeId` from the `AST` this is +not an issue. + +Having the `NodeId` also allows the `DefCollector` to generate the `DefId`s instead +of lowering having to do it on the fly. Centralizing the `DefId` generation in one +place makes it easier to refactor and reason about. \ No newline at end of file From 928470c8cc1b0ef15552deeca46e45f5a508abb7 Mon Sep 17 00:00:00 2001 From: Oliver Schneider Date: Fri, 6 Jul 2018 14:25:34 +0200 Subject: [PATCH 279/648] Satisfy tidy checks --- src/hir.md | 12 ++++++------ src/lowering.md | 23 ++++++++++++----------- src/name-resolution.md | 12 ++++++------ 3 files changed, 24 insertions(+), 23 deletions(-) diff --git a/src/hir.md b/src/hir.md index 0d6a6fbff..0fd2fcff5 100644 --- a/src/hir.md +++ b/src/hir.md @@ -1,13 +1,13 @@ # The HIR -The HIR – "High-Level Intermediate Representation" – is the primary IR used in -most of rustc. It is a compiler-friendly representation of the abstract syntax -tree (AST) that is generated after parsing, macro expansion, and name +The HIR – "High-Level Intermediate Representation" – is the primary IR used +in most of rustc. It is a compiler-friendly representation of the abstract +syntax tree (AST) that is generated after parsing, macro expansion, and name resolution (see [Lowering](./lowering.md) for how the HIR is created). Many parts of HIR resemble Rust surface syntax quite closely, with -the exception that some of Rust's expression forms have been desugared away. For -example, `for` loops are converted into a `loop` and do not appear in the HIR. -This makes HIR more amenable to analysis than a normal AST. +the exception that some of Rust's expression forms have been desugared away. +For example, `for` loops are converted into a `loop` and do not appear in +the HIR. This makes HIR more amenable to analysis than a normal AST. This chapter covers the main concepts of the HIR. diff --git a/src/lowering.md b/src/lowering.md index cc2a7b384..eddc00af9 100644 --- a/src/lowering.md +++ b/src/lowering.md @@ -12,7 +12,8 @@ of such structures include but are not limited to * `if let` * Converted to `match` * Universal `impl Trait` - * Converted to generic arguments (but with some flags, to know that the user didn't write them) + * Converted to generic arguments + (but with some flags, to know that the user didn't write them) * Existential `impl Trait` * Converted to a virtual `existential type` declaration @@ -34,14 +35,14 @@ sanity checks in `src/librustc/hir/map/hir_id_validator.rs`: which produces both a new `NodeId` as well as automatically lowering it for you so you also get the `HirId`. -If you are creating new `DefId`s, since each `DefId` needs to have a corresponding -`NodeId`, it is adviseable to add these `NodeId`s to the `AST` so you don't have -to generate new ones during lowering. This has the advantage of creating a -way to find the `DefId` of something via its `NodeId`. If lowering needs this -`DefId` in multiple places, you can't generate a new `NodeId` in all those places -because you'd also get a new `DefId` then. With a `NodeId` from the `AST` this is -not an issue. +If you are creating new `DefId`s, since each `DefId` needs to have a +corresponding `NodeId`, it is adviseable to add these `NodeId`s to the +`AST` so you don't have to generate new ones during lowering. This has +the advantage of creating a way to find the `DefId` of something via its +`NodeId`. If lowering needs this `DefId` in multiple places, you can't +generate a new `NodeId` in all those places because you'd also get a new +`DefId` then. With a `NodeId` from the `AST` this is not an issue. -Having the `NodeId` also allows the `DefCollector` to generate the `DefId`s instead -of lowering having to do it on the fly. Centralizing the `DefId` generation in one -place makes it easier to refactor and reason about. \ No newline at end of file +Having the `NodeId` also allows the `DefCollector` to generate the `DefId`s +instead of lowering having to do it on the fly. Centralizing the `DefId` +generation in one place makes it easier to refactor and reason about. \ No newline at end of file diff --git a/src/name-resolution.md b/src/name-resolution.md index 5095b750a..bba3142fc 100644 --- a/src/name-resolution.md +++ b/src/name-resolution.md @@ -36,9 +36,9 @@ hierarchy, it's types vs. values vs. macros. ## Scopes and ribs A name is visible only in certain area in the source code. This forms a -hierarchical structure, but not necessarily a simple one ‒ if one scope is part -of another, it doesn't mean the name visible in the outer one is also visible in -the inner one, or that it refers to the same thing. +hierarchical structure, but not necessarily a simple one ‒ if one scope is +part of another, it doesn't mean the name visible in the outer one is also +visible in the inner one, or that it refers to the same thing. To cope with that, the compiler introduces the concept of Ribs. This is abstraction of a scope. Every time the set of visible names potentially changes, @@ -54,9 +54,9 @@ example: When searching for a name, the stack of ribs is traversed from the innermost outwards. This helps to find the closest meaning of the name (the one not shadowed by anything else). The transition to outer rib may also change the -rules what names are usable ‒ if there are nested functions (not closures), the -inner one can't access parameters and local bindings of the outer one, even -though they should be visible by ordinary scoping rules. An example: +rules what names are usable ‒ if there are nested functions (not closures), +the inner one can't access parameters and local bindings of the outer one, +even though they should be visible by ordinary scoping rules. An example: ```rust fn do_something(val: T) { // <- New rib in both types and values (1) From c52d026a1f0d6c599be89ef0244dbb70016688b3 Mon Sep 17 00:00:00 2001 From: Oliver Schneider Date: Fri, 6 Jul 2018 14:39:24 +0200 Subject: [PATCH 280/648] md -> html --- src/hir.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/hir.md b/src/hir.md index 0fd2fcff5..40a14dc25 100644 --- a/src/hir.md +++ b/src/hir.md @@ -3,7 +3,7 @@ The HIR – "High-Level Intermediate Representation" – is the primary IR used in most of rustc. It is a compiler-friendly representation of the abstract syntax tree (AST) that is generated after parsing, macro expansion, and name -resolution (see [Lowering](./lowering.md) for how the HIR is created). +resolution (see [Lowering](./lowering.html) for how the HIR is created). Many parts of HIR resemble Rust surface syntax quite closely, with the exception that some of Rust's expression forms have been desugared away. For example, `for` loops are converted into a `loop` and do not appear in From bf2f7c85b55715ef034eff2741719b8413a875cf Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Sun, 8 Jul 2018 18:34:25 -0500 Subject: [PATCH 281/648] Quick fix: copy/paste error --- src/compiler-debugging.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index 4b445c924..c8fb6dd08 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -1,8 +1,3 @@ ---- -layout: default -title: Debugging the Compiler ---- - **Note: This is copied from the [rust-forge](https://github.com/rust-lang-nursery/rust-forge). If anything needs updating, please open an issue or make a PR on the github repo.** From 911231180313389f7552cd80d521b74c939c0b9e Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Mon, 9 Jul 2018 13:28:29 -0500 Subject: [PATCH 282/648] Add the small amount i know about hygiene (#167) * add the small amount i know about hygiene * use en-dash --- src/macro-expansion.md | 50 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/src/macro-expansion.md b/src/macro-expansion.md index df0dc7b1d..bb3116d40 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -143,6 +143,55 @@ in [`src/libsyntax/ext/tt/macro_parser.rs`][code_mp]. ### Hygiene +If you have ever used C/C++ preprocessor macros, you know that there are some +annoying and hard-to-debug gotchas! For example, consider the following C code: + +```c +#define DEFINE_FOO struct Bar {int x;}; struct Foo {Bar bar;}; + +// Then, somewhere else +struct Bar { + ... +}; + +DEFINE_FOO +``` + +Most people avoid writing C like this – and for good reason: it doesn't +compile. The `struct Bar` defined by the macro clashes names with the `struct +Bar` defined in the code. Consider also the following example: + +```c +#define DO_FOO(x) {\ + int y = 0;\ + foo(x, y);\ + } + +// Then elsewhere +int y = 22; +DO_FOO(y); +``` + +Do you see the problem? We wanted to generate a call `foo(22, 0)`, but instead +we got `foo(0, 0)` because the macro defined its own `y`! + +These are both examples of _macro hygiene_ issues. _Hygiene_ relates to how to +handle names defined _within a macro_. In particular, a hygienic macro system +prevents errors due to names introduced within a macro. Rust macros are hygienic +in that they do not allow one to write the sorts of bugs above. + +At a high level, hygiene within the rust compiler is accomplished by keeping +track of the context where a name is introduced and used. We can then +disambiguate names based on that context. Future iterations of the macro system +will allow greater control to the macro author to use that context. For example, +a macro author may want to introduce a new name to the context where the macro +was called. Alternately, the macro author may be defining a variable for use +only within the macro (i.e. it should not be visible outside the macro). + +In rustc, this "context" is tracked via `Span`s. + +TODO: what is call-site hygiene? what is def-site hygiene? + TODO ### Procedural Macros @@ -153,6 +202,7 @@ TODO TODO +TODO: maybe something about macros 2.0? [code_dir]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt From a154eb95492aa5218f947cb2099fed2a67f66716 Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Mon, 9 Jul 2018 05:10:27 -0700 Subject: [PATCH 283/648] Update mdbook Closes #138 Closes #160 --- ci/install.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install.sh b/ci/install.sh index 69b96cc73..29e3a2f9b 100755 --- a/ci/install.sh +++ b/ci/install.sh @@ -20,5 +20,5 @@ function cargo_install() { fi } -cargo_install mdbook 0.1.7 +cargo_install mdbook 0.1.8 cargo_install mdbook-linkcheck 0.1.2 From 37a202cdba5bc9023d7bbe53628126fb0f2eec06 Mon Sep 17 00:00:00 2001 From: mark Date: Sun, 8 Jul 2018 19:49:34 -0500 Subject: [PATCH 284/648] fix all the not-en-dashes --- README.md | 4 +-- src/appendix/background.md | 4 +-- src/compiletest.md | 2 +- src/conventions.md | 6 ++--- src/incrcomp-debugging.md | 2 +- src/mir/borrowck.md | 2 +- src/mir/index.md | 10 +++---- src/mir/passes.md | 6 ++--- src/mir/regionck.md | 36 ++++++++++++------------- src/mir/visitor.md | 2 +- src/query.md | 2 +- src/tests/adding.md | 8 +++--- src/tests/intro.md | 48 ++++++++++++++++----------------- src/tests/running.md | 4 +-- src/traits/associated-types.md | 10 +++---- src/traits/canonical-queries.md | 12 ++++----- src/traits/canonicalization.md | 2 +- src/traits/goals-and-clauses.md | 4 +-- src/traits/index.md | 2 +- src/traits/lowering-module.md | 4 +-- src/traits/lowering-rules.md | 10 +++---- src/traits/lowering-to-logic.md | 10 +++---- src/type-checking.md | 4 +-- src/variance.md | 6 ++--- 24 files changed, 100 insertions(+), 100 deletions(-) diff --git a/README.md b/README.md index 5dc64ed4d..a01758d48 100644 --- a/README.md +++ b/README.md @@ -18,8 +18,8 @@ Once it gets more complete, the plan is probably to move it into the If you'd like to help finish the guide, we'd love to have you! The main tracking issue for the guide [can be found here](https://github.com/rust-lang-nursery/rustc-guide/issues/6). From -there, you can find a list of all the planned chapters and subsections --- if you think something is missing, please open an issue about it! +there, you can find a list of all the planned chapters and subsections. +If you think something is missing, please open an issue about it! Otherwise, find a chapter that sounds interesting to you and then go to its associated issue. There should be a list of things to do. diff --git a/src/appendix/background.md b/src/appendix/background.md index 069131947..efdce4f2a 100644 --- a/src/appendix/background.md +++ b/src/appendix/background.md @@ -15,7 +15,7 @@ exposes the underlying control flow in a very clear way. A control-flow graph is structured as a set of **basic blocks** connected by edges. The key idea of a basic block is that it is a set -of statements that execute "together" -- that is, whenever you branch +of statements that execute "together" – that is, whenever you branch to a basic block, you start at the first statement and then execute all the remainder. Only at the end of the block is there the possibility of branching to more than one place (in MIR, we call that @@ -119,7 +119,7 @@ variables, since that's the thing we're most familiar with. So there you have it: a variable "appears free" in some expression/statement/whatever if it refers to something defined outside of that expressions/statement/whatever. Equivalently, we can -then refer to the "free variables" of an expression -- which is just +then refer to the "free variables" of an expression – which is just the set of variables that "appear free". So what does this have to do with regions? Well, we can apply the diff --git a/src/compiletest.md b/src/compiletest.md index cdcede33a..0837fb819 100644 --- a/src/compiletest.md +++ b/src/compiletest.md @@ -16,7 +16,7 @@ expect, and more. If you are unfamiliar with the compiler testing framework, see [this chapter](./tests/intro.html) for additional background. The tests themselves are typically (but not always) organized into -"suites"--for example, `run-pass`, a folder representing tests that should +"suites" – for example, `run-pass`, a folder representing tests that should succeed, `run-fail`, a folder holding tests that should compile successfully, but return a failure (non-zero status), `compile-fail`, a folder holding tests that should fail to compile, and many more. The various suites are defined in diff --git a/src/conventions.md b/src/conventions.md index 5b2b7b7f6..3f30dac73 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -44,8 +44,8 @@ current year if you like, but you don't have to. Lines should be at most 100 characters. It's even better if you can keep things to 80. -**Ignoring the line length limit.** Sometimes -- in particular for -tests -- it can be necessary to exempt yourself from this limit. In +**Ignoring the line length limit.** Sometimes – in particular for +tests – it can be necessary to exempt yourself from this limit. In that case, you can add a comment towards the top of the file (after the copyright notice) like so: @@ -141,7 +141,7 @@ command like `git rebase -i rust-lang/master` (presuming you use the name `rust-lang` for your remote). **Individual commits do not have to build (but it's nice).** We do not -require that every intermediate commit successfully builds -- we only +require that every intermediate commit successfully builds – we only expect to be able to bisect at a PR level. However, if you *can* make individual commits build, that is always helpful. diff --git a/src/incrcomp-debugging.md b/src/incrcomp-debugging.md index 2488aa320..ecb29d544 100644 --- a/src/incrcomp-debugging.md +++ b/src/incrcomp-debugging.md @@ -90,7 +90,7 @@ path that should not exist, but you will not be quite sure how it came to be. **When the compiler is built with debug assertions,** it can help you track that down. Simply set the `RUST_FORBID_DEP_GRAPH_EDGE` environment variable to a filter. Every edge created in the dep-graph -will be tested against that filter -- if it matches, a `bug!` is +will be tested against that filter – if it matches, a `bug!` is reported, so you can easily see the backtrace (`RUST_BACKTRACE=1`). The syntax for these filters is the same as described in the previous diff --git a/src/mir/borrowck.md b/src/mir/borrowck.md index cd83843f8..d5fea6184 100644 --- a/src/mir/borrowck.md +++ b/src/mir/borrowck.md @@ -1,6 +1,6 @@ # MIR borrow check -The borrow check is Rust's "secret sauce" -- it is tasked with +The borrow check is Rust's "secret sauce" – it is tasked with enforcing a number of properties: - That all variables are initialized before they are used. diff --git a/src/mir/index.md b/src/mir/index.md index 838048b81..ea72decdc 100644 --- a/src/mir/index.md +++ b/src/mir/index.md @@ -3,7 +3,7 @@ MIR is Rust's _Mid-level Intermediate Representation_. It is constructed from [HIR](./hir.html). MIR was introduced in [RFC 1211]. It is a radically simplified form of Rust that is used for -certain flow-sensitive safety checks -- notably the borrow checker! -- +certain flow-sensitive safety checks – notably the borrow checker! – and also for optimization and code generation. If you'd like a very high-level introduction to MIR, as well as some @@ -122,7 +122,7 @@ StorageLive(_1); ``` This statement indicates that the variable `_1` is "live", meaning -that it may be used later -- this will persist until we encounter a +that it may be used later – this will persist until we encounter a `StorageDead(_1)` statement, which indicates that the variable `_1` is done being used. These "storage statements" are used by LLVM to allocate stack space. @@ -134,7 +134,7 @@ _1 = const >::new() -> bb2; ``` Terminators are different from statements because they can have more -than one successor -- that is, control may flow to different +than one successor – that is, control may flow to different places. Function calls like the call to `Vec::new` are always terminators because of the possibility of unwinding, although in the case of `Vec::new` we are able to see that indeed unwinding is not @@ -163,7 +163,7 @@ Assignments in general have the form: = ``` -A place is an expression like `_3`, `_3.f` or `*_3` -- it denotes a +A place is an expression like `_3`, `_3.f` or `*_3` – it denotes a location in memory. An **Rvalue** is an expression that creates a value: in this case, the rvalue is a mutable borrow expression, which looks like `&mut `. So we can kind of define a grammar for @@ -180,7 +180,7 @@ rvalues like so: | move Place ``` -As you can see from this grammar, rvalues cannot be nested -- they can +As you can see from this grammar, rvalues cannot be nested – they can only reference places and constants. Moreover, when you use a place, we indicate whether we are **copying it** (which requires that the place have a type `T` where `T: Copy`) or **moving it** (which works diff --git a/src/mir/passes.md b/src/mir/passes.md index a5b5df10a..c48ef4576 100644 --- a/src/mir/passes.md +++ b/src/mir/passes.md @@ -100,8 +100,8 @@ that appeared within the `main` function.) ### Implementing and registering a pass -A `MirPass` is some bit of code that processes the MIR, typically -- -but not always -- transforming it along the way somehow. For example, +A `MirPass` is some bit of code that processes the MIR, typically – +but not always – transforming it along the way somehow. For example, it might perform an optimization. The `MirPass` trait itself is found in in [the `rustc_mir::transform` module][mirtransform], and it basically consists of one method, `run_pass`, that simply gets an @@ -110,7 +110,7 @@ came from). The MIR is therefore modified in place (which helps to keep things efficient). A good example of a basic MIR pass is [`NoLandingPads`], which walks -the MIR and removes all edges that are due to unwinding -- this is +the MIR and removes all edges that are due to unwinding – this is used when configured with `panic=abort`, which never unwinds. As you can see from its source, a MIR pass is defined by first defining a dummy type, a struct with no fields, something like: diff --git a/src/mir/regionck.md b/src/mir/regionck.md index 90f6fa274..9034af8a8 100644 --- a/src/mir/regionck.md +++ b/src/mir/regionck.md @@ -12,13 +12,13 @@ The MIR-based region analysis consists of two major functions: - `replace_regions_in_mir`, invoked first, has two jobs: - First, it finds the set of regions that appear within the signature of the function (e.g., `'a` in `fn foo<'a>(&'a u32) { - ... }`). These are called the "universal" or "free" regions -- in + ... }`). These are called the "universal" or "free" regions – in particular, they are the regions that [appear free][fvb] in the function body. - Second, it replaces all the regions from the function body with fresh inference variables. This is because (presently) those regions are the results of lexical region inference and hence are - not of much interest. The intention is that -- eventually -- they + not of much interest. The intention is that – eventually – they will be "erased regions" (i.e., no information at all), since we won't be doing lexical region inference at all. - `compute_regions`, invoked second: this is given as argument the @@ -40,11 +40,11 @@ The MIR-based region analysis consists of two major functions: ## Universal regions -*to be written* -- explain the `UniversalRegions` type +*to be written* – explain the `UniversalRegions` type ## Region variables and constraints -*to be written* -- describe the `RegionInferenceContext` and +*to be written* – describe the `RegionInferenceContext` and the role of `liveness_constraints` vs other `constraints`, plus ## Closures @@ -79,13 +79,13 @@ The kinds of region elements are as follows: - Similarly, there is an element denoted `end('static)` corresponding to the remainder of program execution after this function returns. - There is an element `!1` for each skolemized region `!1`. This - corresponds (intuitively) to some unknown set of other elements -- + corresponds (intuitively) to some unknown set of other elements – for details on skolemization, see the section [skolemization and universes](#skol). ## Causal tracking -*to be written* -- describe how we can extend the values of a variable +*to be written* – describe how we can extend the values of a variable with causal tracking etc @@ -133,7 +133,7 @@ bound in the supertype and **skolemizing** them: this means that we replace them with [universally quantified](appendix/background.html#quantified) representatives, written like `!1`. We call these regions "skolemized -regions" -- they represent, basically, "some unknown region". +regions" – they represent, basically, "some unknown region". Once we've done that replacement, we have the following relation: @@ -156,9 +156,9 @@ we swap the left and right here): ``` According to the basic subtyping rules for a reference, this will be -true if `'!1: 'static`. That is -- if "some unknown region `!1`" lives -outlives `'static`. Now, this *might* be true -- after all, `'!1` -could be `'static` -- but we don't *know* that it's true. So this +true if `'!1: 'static`. That is – if "some unknown region `!1`" lives +outlives `'static`. Now, this *might* be true – after all, `'!1` +could be `'static` – but we don't *know* that it's true. So this should yield up an error (eventually). ### What is a universe @@ -238,8 +238,8 @@ not U1. **Giving existential variables a universe.** Now that we have this notion of universes, we can use it to extend our type-checker and things to prevent illegal names from leaking out. The idea is that we -give each inference (existential) variable -- whether it be a type or -a lifetime -- a universe. That variable's value can then only +give each inference (existential) variable – whether it be a type or +a lifetime – a universe. That variable's value can then only reference names visible from that universe. So for example is a lifetime variable is created in U0, then it cannot be assigned a value of `!1` or `!2`, because those names are not visible from the universe @@ -247,7 +247,7 @@ U0. **Representing universes with just a counter.** You might be surprised to see that the compiler doesn't keep track of a full tree of -universes. Instead, it just keeps a counter -- and, to determine if +universes. Instead, it just keeps a counter – and, to determine if one universe can see another one, it just checks if the index is greater. For example, U2 can see U0 because 2 >= 0. But U0 cannot see U2, because 0 >= 2 is false. @@ -323,12 +323,12 @@ Now there are two ways that could happen. First, if `U(V1)` can see the universe `x` (i.e., `x <= U(V1)`), then we can just add `skol(x)` to `value(V1)` and be done. But if not, then we have to approximate: we may not know what set of elements `skol(x)` represents, but we -should be able to compute some sort of **upper bound** B for it -- +should be able to compute some sort of **upper bound** B for it – some region B that outlives `skol(x)`. For now, we'll just use -`'static` for that (since it outlives everything) -- in the future, we +`'static` for that (since it outlives everything) – in the future, we can sometimes be smarter here (and in fact we have code for doing this already in other contexts). Moreover, since `'static` is in the root -universe U0, we know that all variables can see it -- so basically if +universe U0, we know that all variables can see it – so basically if we find that `value(V2)` contains `skol(x)` for some universe `x` that `V1` can't see, then we force `V1` to `'static`. @@ -398,8 +398,8 @@ outlives relationships are satisfied. Then we would go to the "check universal regions" portion of the code, which would test that no universal region grew too large. -In this case, `V1` *did* grow too large -- it is not known to outlive -`end('static)`, nor any of the CFG -- so we would report an error. +In this case, `V1` *did* grow too large – it is not known to outlive +`end('static)`, nor any of the CFG – so we would report an error. ## Another example diff --git a/src/mir/visitor.md b/src/mir/visitor.md index 265769b34..ad00bc3f1 100644 --- a/src/mir/visitor.md +++ b/src/mir/visitor.md @@ -2,7 +2,7 @@ The MIR visitor is a convenient tool for traversing the MIR and either looking for things or making changes to it. The visitor traits are -defined in [the `rustc::mir::visit` module][m-v] -- there are two of +defined in [the `rustc::mir::visit` module][m-v] – there are two of them, generated via a single macro: `Visitor` (which operates on a `&Mir` and gives back shared references) and `MutVisitor` (which operates on a `&mut Mir` and gives back mutable references). diff --git a/src/query.md b/src/query.md index 7ed299b74..a0c0d6282 100644 --- a/src/query.md +++ b/src/query.md @@ -5,7 +5,7 @@ Rust compiler is current transitioning from a traditional "pass-based" setup to a "demand-driven" system. **The Compiler Query System is the key to our new demand-driven organization.** The idea is pretty simple. You have various queries that compute things about the input --- for example, there is a query called `type_of(def_id)` that, given +– for example, there is a query called `type_of(def_id)` that, given the def-id of some item, will compute the type of that item and return it to you. diff --git a/src/tests/adding.md b/src/tests/adding.md index fac13fcac..a9400c591 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -49,8 +49,8 @@ considered an ideal setup. [`src/test/run-pass`]: https://github.com/rust-lang/rust/tree/master/src/test/run-pass/ -For regression tests -- basically, some random snippet of code that -came in from the internet -- we often just name the test after the +For regression tests – basically, some random snippet of code that +came in from the internet – we often just name the test after the issue. For example, `src/test/run-pass/issue-12345.rs`. If possible, though, it is better if you can put the test into a directory that helps identify what piece of code is being tested here (e.g., @@ -267,9 +267,9 @@ can also make UI tests where compilation is expected to succeed, and you can even run the resulting program. Just add one of the following [header commands](#header_commands): -- `// compile-pass` -- compilation should succeed but do +- `// compile-pass` – compilation should succeed but do not run the resulting binary -- `// run-pass` -- compilation should succeed and we should run the +- `// run-pass` – compilation should succeed and we should run the resulting binary diff --git a/src/tests/intro.md b/src/tests/intro.md index bc84c9721..69d9ed057 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -15,7 +15,7 @@ on [how to run tests](./tests/running.html#ui) as well as The compiletest tests are located in the tree in the [`src/test`] directory. Immediately within you will see a series of subdirectories (e.g. `ui`, `run-make`, and so forth). Each of those directories is -called a **test suite** -- they house a group of tests that are run in +called a **test suite** – they house a group of tests that are run in a distinct mode. [`src/test`]: https://github.com/rust-lang/rust/tree/master/src/test @@ -24,31 +24,31 @@ Here is a brief summary of the test suites as of this writing and what they mean. In some cases, the test suites are linked to parts of the manual that give more details. -- [`ui`](./tests/adding.html#ui) -- tests that check the exact +- [`ui`](./tests/adding.html#ui) – tests that check the exact stdout/stderr from compilation and/or running the test -- `run-pass` -- tests that are expected to compile and execute +- `run-pass` – tests that are expected to compile and execute successfully (no panics) - - `run-pass-valgrind` -- tests that ought to run with valgrind -- `run-fail` -- tests that are expected to compile but then panic + - `run-pass-valgrind` – tests that ought to run with valgrind +- `run-fail` – tests that are expected to compile but then panic during execution -- `compile-fail` -- tests that are expected to fail compilation. -- `parse-fail` -- tests that are expected to fail to parse -- `pretty` -- tests targeting the Rust "pretty printer", which +- `compile-fail` – tests that are expected to fail compilation. +- `parse-fail` – tests that are expected to fail to parse +- `pretty` – tests targeting the Rust "pretty printer", which generates valid Rust code from the AST -- `debuginfo` -- tests that run in gdb or lldb and query the debug info -- `codegen` -- tests that compile and then test the generated LLVM +- `debuginfo` – tests that run in gdb or lldb and query the debug info +- `codegen` – tests that compile and then test the generated LLVM code to make sure that the optimizations we want are taking effect. -- `mir-opt` -- tests that check parts of the generated MIR to make +- `mir-opt` – tests that check parts of the generated MIR to make sure we are building things correctly or doing the optimizations we expect. -- `incremental` -- tests for incremental compilation, checking that +- `incremental` – tests for incremental compilation, checking that when certain modifications are performed, we are able to reuse the results from previous compilations. -- `run-make` -- tests that basically just execute a `Makefile`; the +- `run-make` – tests that basically just execute a `Makefile`; the ultimate in flexibility but quite annoying to write. -- `rustdoc` -- tests for rustdoc, making sure that the generated files +- `rustdoc` – tests for rustdoc, making sure that the generated files contain the expected documentation. -- `*-fulldeps` -- same as above, but indicates that the test depends +- `*-fulldeps` – same as above, but indicates that the test depends on things other than `libstd` (and hence those things must be built) ## Other Tests @@ -56,44 +56,44 @@ that give more details. The Rust build system handles running tests for various other things, including: -- **Tidy** -- This is a custom tool used for validating source code +- **Tidy** – This is a custom tool used for validating source code style and formatting conventions, such as rejecting long lines. There is more information in the [section on coding conventions](./conventions.html#formatting). Example: `./x.py test src/tools/tidy` -- **Unittests** -- The Rust standard library and many of the Rust packages +- **Unittests** – The Rust standard library and many of the Rust packages include typical Rust `#[test]` unittests. Under the hood, `x.py` will run `cargo test` on each package to run all the tests. Example: `./x.py test src/libstd` -- **Doctests** -- Example code embedded within Rust documentation is executed +- **Doctests** – Example code embedded within Rust documentation is executed via `rustdoc --test`. Examples: - `./x.py test src/doc` -- Runs `rustdoc --test` for all documentation in + `./x.py test src/doc` – Runs `rustdoc --test` for all documentation in `src/doc`. - `./x.py test --doc src/libstd` -- Runs `rustdoc --test` on the standard + `./x.py test --doc src/libstd` – Runs `rustdoc --test` on the standard library. -- **Linkchecker** -- A small tool for verifying `href` links within +- **Linkchecker** – A small tool for verifying `href` links within documentation. Example: `./x.py test src/tools/linkchecker` -- **Distcheck** -- This verifies that the source distribution tarball created +- **Distcheck** – This verifies that the source distribution tarball created by the build system will unpack, build, and run all tests. Example: `./x.py test distcheck` -- **Tool tests** -- Packages that are included with Rust have all of their +- **Tool tests** – Packages that are included with Rust have all of their tests run as well (typically by running `cargo test` within their directory). This includes things such as cargo, clippy, rustfmt, rls, miri, bootstrap (testing the Rust build system itself), etc. -- **Cargotest** -- This is a small tool which runs `cargo test` on a few +- **Cargotest** – This is a small tool which runs `cargo test` on a few significant projects (such as `servo`, `ripgrep`, `tokei`, etc.) just to ensure there aren't any significant regressions. diff --git a/src/tests/running.md b/src/tests/running.md index 03eb8ccc4..e55b6ec7d 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -1,7 +1,7 @@ # Running tests -You can run the tests using `x.py`. The most basic command -- which -you will almost never want to use! -- is as follows: +You can run the tests using `x.py`. The most basic command – which +you will almost never want to use! – is as follows: ```bash > ./x.py test diff --git a/src/traits/associated-types.md b/src/traits/associated-types.md index ec20985ff..1972a70e6 100644 --- a/src/traits/associated-types.md +++ b/src/traits/associated-types.md @@ -15,7 +15,7 @@ When a trait defines an associated type (e.g., [the `Item` type in the `IntoIterator` trait][intoiter-item]), that type can be referenced by the user using an **associated type projection** like ` as IntoIterator>::Item`. (Often, -though, people will use the shorthand syntax `T::Item` -- presently, +though, people will use the shorthand syntax `T::Item` – presently, that syntax is expanded during ["type collection"](./type-checking.html) into the explicit form, though that is something we may want to change in the future.) @@ -24,8 +24,8 @@ though that is something we may want to change in the future.) -In some cases, associated type projections can be **normalized** -- -that is, simplified -- based on the types given in an impl. So, to +In some cases, associated type projections can be **normalized** – +that is, simplified – based on the types given in an impl. So, to continue with our example, the impl of `IntoIterator` for `Option` declares (among other things) that `Item = T`: @@ -39,9 +39,9 @@ impl IntoIterator for Option { This means we can normalize the projection ` as IntoIterator>::Item` to just `u32`. -In this case, the projection was a "monomorphic" one -- that is, it +In this case, the projection was a "monomorphic" one – that is, it did not have any type parameters. Monomorphic projections are special -because they can **always** be fully normalized -- but often we can +because they can **always** be fully normalized – but often we can normalize other associated type projections as well. For example, ` as IntoIterator>::Item` (where `?T` is an inference variable) can be normalized to just `?T`. diff --git a/src/traits/canonical-queries.md b/src/traits/canonical-queries.md index 56637dace..c58fe3696 100644 --- a/src/traits/canonical-queries.md +++ b/src/traits/canonical-queries.md @@ -1,8 +1,8 @@ # Canonical queries The "start" of the trait system is the **canonical query** (these are -both queries in the more general sense of the word -- something you -would like to know the answer to -- and in the +both queries in the more general sense of the word – something you +would like to know the answer to – and in the [rustc-specific sense](./query.html)). The idea is that the type checker or other parts of the system, may in the course of doing their thing want to know whether some trait is implemented for some type @@ -35,7 +35,7 @@ solver is finding **all possible** instantiations of your query that are true. In this case, if we instantiate `?U = [i32]`, then the query is true (note that a traditional Prolog interface does not, directly, tell us a value for `?U`, but we can infer one by unifying the -response with our original query -- Rust's solver gives back a +response with our original query – Rust's solver gives back a substitution instead). If we were to hit `y`, the solver might then give us another possible answer: @@ -135,7 +135,7 @@ we did find. It consists of four parts: [section on handling regions in traits](./traits/regions.html) for more details. - **Value:** The query result also comes with a value of type `T`. For - some specialized queries -- like normalizing associated types -- + some specialized queries – like normalizing associated types – this is used to carry back an extra result, but it's often just `()`. @@ -187,8 +187,8 @@ for example: Therefore, the result we get back would be as follows (I'm going to ignore region constraints and the "value"): -- Certainty: `Ambiguous` -- we're not sure yet if this holds -- Var values: `[?T = ?T, ?U = ?U]` -- we learned nothing about the values of +- Certainty: `Ambiguous` – we're not sure yet if this holds +- Var values: `[?T = ?T, ?U = ?U]` – we learned nothing about the values of the variables In short, the query result says that it is too soon to say much about diff --git a/src/traits/canonicalization.md b/src/traits/canonicalization.md index f87de5f81..1ef1da717 100644 --- a/src/traits/canonicalization.md +++ b/src/traits/canonicalization.md @@ -24,7 +24,7 @@ form of `X` would be `(?0, ?1)`, where `?0` and `?1` represent these **canonical placeholders**. Note that the type `Y = (?U, ?T)` also canonicalizes to `(?0, ?1)`. But the type `Z = (?T, ?T)` would canonicalize to `(?0, ?0)` (as would `(?U, ?U)`). In other words, the -exact identity of the inference variables is not important -- unless +exact identity of the inference variables is not important – unless they are repeated. We use this to improve caching as well as to detect cycles and other diff --git a/src/traits/goals-and-clauses.md b/src/traits/goals-and-clauses.md index 5844e8d45..93bafe33f 100644 --- a/src/traits/goals-and-clauses.md +++ b/src/traits/goals-and-clauses.md @@ -197,7 +197,7 @@ Implemented(Foo: Send) :- As you can probably imagine, proving that `Option>: Send` is going to wind up circularly requiring us to prove that `Foo: Send` -again. So this would be an example where we wind up in a cycle -- but +again. So this would be an example where we wind up in a cycle – but that's ok, we *do* consider `Foo: Send` to hold, even though it references itself. @@ -219,4 +219,4 @@ as described in the section on [implied bounds]. Some topics yet to be written: - Elaborate on the proof procedure -- SLG solving -- introduce negative reasoning +- SLG solving – introduce negative reasoning diff --git a/src/traits/index.md b/src/traits/index.md index 2dabd0b69..c070d1b88 100644 --- a/src/traits/index.md +++ b/src/traits/index.md @@ -28,7 +28,7 @@ Trait solving is based around a few key ideas: whether types are equal. - [Region constraints](./traits/regions.html), which are accumulated during trait solving but mostly ignored. This means that trait - solving effectively ignores the precise regions involved, always -- + solving effectively ignores the precise regions involved, always – but we still remember the constraints on them so that those constraints can be checked by thet type checker. diff --git a/src/traits/lowering-module.md b/src/traits/lowering-module.md index 205e9a171..030e19860 100644 --- a/src/traits/lowering-module.md +++ b/src/traits/lowering-module.md @@ -8,8 +8,8 @@ created in the [`rustc_traits::lowering`][lowering] module. ## The `program_clauses_for` query -The main entry point is the `program_clauses_for` [query], which -- -given a def-id -- produces a set of Chalk program clauses. These +The main entry point is the `program_clauses_for` [query], which – +given a def-id – produces a set of Chalk program clauses. These queries are tested using a [dedicated unit-testing mechanism, described below](#unit-tests). The query is invoked on a `DefId` that identifies something like a trait, diff --git a/src/traits/lowering-rules.md b/src/traits/lowering-rules.md index d8b739890..e37999e57 100644 --- a/src/traits/lowering-rules.md +++ b/src/traits/lowering-rules.md @@ -52,11 +52,11 @@ from the Rust syntax into goals. In addition, in the rules below, we sometimes do some transformations on the lowered where clauses, as defined here: -- `FromEnv(WC)` -- this indicates that: +- `FromEnv(WC)` – this indicates that: - `Implemented(TraitRef)` becomes `FromEnv(TraitRef)` - `ProjectionEq(Projection = Ty)` becomes `FromEnv(Projection = Ty)` - other where-clauses are left intact -- `WellFormed(WC)` -- this indicates that: +- `WellFormed(WC)` – this indicates that: - `Implemented(TraitRef)` becomes `WellFormed(TraitRef)` - `ProjectionEq(Projection = Ty)` becomes `WellFormed(Projection = Ty)` @@ -121,8 +121,8 @@ trait Eq: PartialEq { ... } In this case, the `PartialEq` supertrait is equivalent to a `where Self: PartialEq` where clause, in our simplified model. The program -clause above therefore states that if we can prove `FromEnv(T: Eq)` -- -e.g., if we are in some function with `T: Eq` in its where clauses -- +clause above therefore states that if we can prove `FromEnv(T: Eq)` – +e.g., if we are in some function with `T: Eq` in its where clauses – then we also know that `FromEnv(T: PartialEq)`. Thus the set of things that follow from the environment are not only the **direct where clauses** but also things that follow from them. @@ -169,7 +169,7 @@ have to prove each one of those: - `WellFormed(T: Foo)` -- cycle, true coinductively This `WellFormed` predicate is only used when proving that impls are -well-formed -- basically, for each impl of some trait ref `TraitRef`, +well-formed – basically, for each impl of some trait ref `TraitRef`, we must show that `WellFormed(TraitRef)`. This in turn justifies the implied bounds rules that allow us to extend the set of `FromEnv` items. diff --git a/src/traits/lowering-to-logic.md b/src/traits/lowering-to-logic.md index 2de28ce4e..cf8c1c25a 100644 --- a/src/traits/lowering-to-logic.md +++ b/src/traits/lowering-to-logic.md @@ -15,8 +15,8 @@ One of the first observations is that the Rust trait system is basically a kind of logic. As such, we can map our struct, trait, and impl declarations into logical inference rules. For the most part, these are basically Horn clauses, though we'll see that to capture the -full richness of Rust -- and in particular to support generic -programming -- we have to go a bit further than standard Horn clauses. +full richness of Rust – and in particular to support generic +programming – we have to go a bit further than standard Horn clauses. To see how this mapping works, let's start with an example. Imagine we declare a trait and a few impls, like so: @@ -38,8 +38,8 @@ Clone(Vec) :- Clone(?T). // Or, put another way, B implies A. ``` -In Prolog terms, we might say that `Clone(Foo)` -- where `Foo` is some -Rust type -- is a *predicate* that represents the idea that the type +In Prolog terms, we might say that `Clone(Foo)` – where `Foo` is some +Rust type – is a *predicate* that represents the idea that the type `Foo` implements `Clone`. These rules are **program clauses**; they state the conditions under which that predicate can be proven (i.e., considered true). So the first rule just says "Clone is implemented @@ -162,7 +162,7 @@ notation but a bit Rustified. Anyway, the problem is that standard Horn clauses don't allow universal quantification (`forall`) or implication (`if`) in goals (though many Prolog engines do support them, as an extension). For this reason, we need to accept something -called "first-order hereditary harrop" (FOHH) clauses -- this long +called "first-order hereditary harrop" (FOHH) clauses – this long name basically means "standard Horn clauses with `forall` and `if` in the body". But it's nice to know the proper name, because there is a lot of work describing how to efficiently handle FOHH clauses; see for diff --git a/src/type-checking.md b/src/type-checking.md index 9a161abd2..d4f4bf110 100644 --- a/src/type-checking.md +++ b/src/type-checking.md @@ -12,7 +12,7 @@ draws heavily on the [type inference] and [trait solving].) Type "collection" is the process of converting the types found in the HIR (`hir::Ty`), which represent the syntactic things that the user wrote, into the -**internal representation** used by the compiler (`Ty<'tcx>`) -- we also do +**internal representation** used by the compiler (`Ty<'tcx>`) – we also do similar conversions for where-clauses and other bits of the function signature. To try and get a sense for the difference, consider this function: @@ -30,7 +30,7 @@ they encode the path somewhat differently. But once they are "collected" into Collection is defined as a bundle of [queries] for computing information about the various functions, traits, and other items in the crate being compiled. -Note that each of these queries is concerned with *interprocedural* things -- +Note that each of these queries is concerned with *interprocedural* things – for example, for a function definition, collection will figure out the type and signature of the function, but it will not visit the *body* of the function in any way, nor examine type annotations on local variables (that's the job of diff --git a/src/variance.md b/src/variance.md index 05a286c07..9fe98b4a1 100644 --- a/src/variance.md +++ b/src/variance.md @@ -167,7 +167,7 @@ one could interpret variance and trait matching. Just as with structs and enums, we can decide the subtyping relationship between two object types `&Trait` and `&Trait` based on the relationship of `A` and `B`. Note that for object -types we ignore the `Self` type parameter -- it is unknown, and +types we ignore the `Self` type parameter – it is unknown, and the nature of dynamic dispatch ensures that we will always call a function that is expected the appropriate `Self` type. However, we must be careful with the other type parameters, or else we could @@ -274,8 +274,8 @@ These conditions are satisfied and so we are happy. ### Variance and associated types -Traits with associated types -- or at minimum projection -expressions -- must be invariant with respect to all of their +Traits with associated types – or at minimum projection +expressions – must be invariant with respect to all of their inputs. To see why this makes sense, consider what subtyping for a trait reference means: From 337c64e4d2d7f7ccb7c817e28d3d5d47e8d99442 Mon Sep 17 00:00:00 2001 From: mark Date: Sun, 8 Jul 2018 20:24:09 -0500 Subject: [PATCH 285/648] fix lacking spaces --- src/tests/intro.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/tests/intro.md b/src/tests/intro.md index 69d9ed057..bfe084618 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -63,13 +63,13 @@ including: Example: `./x.py test src/tools/tidy` -- **Unittests** – The Rust standard library and many of the Rust packages +- **Unit tests** – The Rust standard library and many of the Rust packages include typical Rust `#[test]` unittests. Under the hood, `x.py` will run `cargo test` on each package to run all the tests. Example: `./x.py test src/libstd` -- **Doctests** – Example code embedded within Rust documentation is executed +- **Doc tests** – Example code embedded within Rust documentation is executed via `rustdoc --test`. Examples: `./x.py test src/doc` – Runs `rustdoc --test` for all documentation in @@ -78,12 +78,12 @@ including: `./x.py test --doc src/libstd` – Runs `rustdoc --test` on the standard library. -- **Linkchecker** – A small tool for verifying `href` links within +- **Link checker** – A small tool for verifying `href` links within documentation. Example: `./x.py test src/tools/linkchecker` -- **Distcheck** – This verifies that the source distribution tarball created +- **Dist check** – This verifies that the source distribution tarball created by the build system will unpack, build, and run all tests. Example: `./x.py test distcheck` @@ -93,7 +93,7 @@ including: directory). This includes things such as cargo, clippy, rustfmt, rls, miri, bootstrap (testing the Rust build system itself), etc. -- **Cargotest** – This is a small tool which runs `cargo test` on a few +- **Cargo test** – This is a small tool which runs `cargo test` on a few significant projects (such as `servo`, `ripgrep`, `tokei`, etc.) just to ensure there aren't any significant regressions. From bdcd161d200b8f2103dad541850597c0b4e2ae06 Mon Sep 17 00:00:00 2001 From: mark Date: Sun, 8 Jul 2018 18:44:04 -0500 Subject: [PATCH 286/648] define FileMap; fix #35 --- src/appendix/code-index.md | 5 +++-- src/the-parser.md | 9 +++++---- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index ef7853fc6..d47cde8cd 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -9,12 +9,13 @@ Item | Kind | Short description | Chapter | `BodyId` | struct | One of four types of HIR node identifiers. | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.BodyId.html) `CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser] | [src/libsyntax/codemap.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html) `CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver] | [src/librustc_driver/driver.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/struct.CompileState.html) +`ast::Crate` | struct | Syntax-level representation of a parsed crate | [The parser] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Crate.html) +`hir::Crate` | struct | More abstract, compiler-friendly form of a crate's AST | [The Hir] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Crate.html) `DefId` | struct | One of four types of HIR node identifiers. | [Identifiers in the HIR] | [src/librustc/hir/def_id.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/def_id/struct.DefId.html) `DiagnosticBuilder` | struct | A struct for building up compiler diagnostics, such as errors or lints | [Emitting Diagnostics] | [src/librustc_errors/diagnostic_builder.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.DiagnosticBuilder.html) `DocContext` | struct | A state container used by rustdoc when crawling through a crate to gather its documentation | [Rustdoc] | [src/librustdoc/core.rs](https://github.com/rust-lang/rust/blob/master/src/librustdoc/core.rs) -`ast::Crate` | struct | Syntax-level representation of a parsed crate | [The parser] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Crate.html) +`FileMap` | struct | A single source within a `CodeMap` (e.g. the source code within a single file). | [The parser] | [src/libsyntax_pos/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.FileMap.html) `HirId` | struct | One of four types of HIR node identifiers. | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.HirId.html) -`hir::Crate` | struct | More abstract, compiler-friendly form of a crate's AST | [The Hir] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Crate.html) `NodeId` | struct | One of four types of HIR node identifiers. Being phased out. | [Identifiers in the HIR] | [src/libsyntax/ast.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.NodeId.html) `ParamEnv` | struct | Information about generic parameters or `Self`, useful for working with associated or generic items | [Parameter Environment] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.ParamEnv.html) `ParseSess` | struct | This struct contains information about a parsing session | [The parser] | [src/libsyntax/parse/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/struct.ParseSess.html) diff --git a/src/the-parser.md b/src/the-parser.md index dd451d24f..e89c09e9e 100644 --- a/src/the-parser.md +++ b/src/the-parser.md @@ -22,10 +22,10 @@ The `syntax` crate contains several main players, - and a [visit module] for walking the AST and inspecting or mutating the AST nodes. -The main entrypoint to the parser is via the various `parse_*` functions -in the [parser module]. They let you do things like turn a filemap into a -token stream, create a parser from the token stream, and then execute the -parser to get a `Crate` (the root AST node). +The main entrypoint to the parser is via the various `parse_*` functions in the +[parser module]. They let you do things like turn a [`FileMap`][filemap] (e.g. +the source in a single file) into a token stream, create a parser from the +token stream, and then execute the parser to get a `Crate` (the root AST node). To minimise the amount of copying that is done, both the `StringReader` and `Parser` have lifetimes which bind them to the parent `ParseSess`. This contains @@ -40,3 +40,4 @@ all the information needed while parsing, as well as the `CodeMap` itself. [`Parser`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/parser/struct.Parser.html [`StringReader`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/lexer/struct.StringReader.html [visit module]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/visit/index.html +[filemap]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.FileMap.html From d234211ca1904c1468601c9a547a0f80b8f29288 Mon Sep 17 00:00:00 2001 From: mark Date: Sun, 8 Jul 2018 20:29:58 -0500 Subject: [PATCH 287/648] Attempt to address review comments --- src/appendix/code-index.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index d47cde8cd..1dbbec646 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -6,17 +6,17 @@ compiler. Item | Kind | Short description | Chapter | Declaration ----------------|----------|-----------------------------|--------------------|------------------- -`BodyId` | struct | One of four types of HIR node identifiers. | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.BodyId.html) +`BodyId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.BodyId.html) `CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser] | [src/libsyntax/codemap.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html) `CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver] | [src/librustc_driver/driver.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/struct.CompileState.html) -`ast::Crate` | struct | Syntax-level representation of a parsed crate | [The parser] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Crate.html) -`hir::Crate` | struct | More abstract, compiler-friendly form of a crate's AST | [The Hir] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Crate.html) -`DefId` | struct | One of four types of HIR node identifiers. | [Identifiers in the HIR] | [src/librustc/hir/def_id.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/def_id/struct.DefId.html) +`ast::Crate` | struct | A syntax-level representation of a parsed crate | [The parser] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Crate.html) +`hir::Crate` | struct | A more abstract, compiler-friendly form of a crate's AST | [The Hir] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Crate.html) +`DefId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [src/librustc/hir/def_id.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/def_id/struct.DefId.html) `DiagnosticBuilder` | struct | A struct for building up compiler diagnostics, such as errors or lints | [Emitting Diagnostics] | [src/librustc_errors/diagnostic_builder.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.DiagnosticBuilder.html) `DocContext` | struct | A state container used by rustdoc when crawling through a crate to gather its documentation | [Rustdoc] | [src/librustdoc/core.rs](https://github.com/rust-lang/rust/blob/master/src/librustdoc/core.rs) -`FileMap` | struct | A single source within a `CodeMap` (e.g. the source code within a single file). | [The parser] | [src/libsyntax_pos/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.FileMap.html) -`HirId` | struct | One of four types of HIR node identifiers. | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.HirId.html) -`NodeId` | struct | One of four types of HIR node identifiers. Being phased out. | [Identifiers in the HIR] | [src/libsyntax/ast.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.NodeId.html) +`FileMap` | struct | Part of the `CodeMap`, mapping a single source file | [The parser] | [src/libsyntax_pos/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.FileMap.html) +`HirId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.HirId.html) +`NodeId` | struct | One of four types of HIR node identifiers. Being phased out | [Identifiers in the HIR] | [src/libsyntax/ast.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.NodeId.html) `ParamEnv` | struct | Information about generic parameters or `Self`, useful for working with associated or generic items | [Parameter Environment] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.ParamEnv.html) `ParseSess` | struct | This struct contains information about a parsing session | [The parser] | [src/libsyntax/parse/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/struct.ParseSess.html) `Rib` | struct | Represents a single scope of names | [Name resolution] | [src/librustc_resolve/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/struct.Rib.html) @@ -27,7 +27,7 @@ Item | Kind | Short description | Chapter | `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules] | [src/librustc/ty/trait_def.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/trait_def/struct.TraitDef.html) `TraitRef` | struct | The combination of a trait and its input types (e.g. `P0: Trait`) | [Trait Solving: Goals and Clauses], [Trait Solving: Lowering impls] | [src/librustc/ty/sty.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TraitRef.html) `Ty<'tcx>` | struct | This is the internal representation of a type used for type checking | [Type checking] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/type.Ty.html) -`TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries. | [The `ty` modules] | [src/librustc/ty/context.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html) +`TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries | [The `ty` modules] | [src/librustc/ty/context.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html) [The HIR]: hir.html [Identifiers in the HIR]: hir.html#hir-id From 1a696569a846692c171549537b4726e9fd9fe238 Mon Sep 17 00:00:00 2001 From: mark Date: Tue, 10 Jul 2018 20:52:47 -0500 Subject: [PATCH 288/648] update definitions --- src/appendix/code-index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index 1dbbec646..b6986c5c0 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -7,14 +7,14 @@ compiler. Item | Kind | Short description | Chapter | Declaration ----------------|----------|-----------------------------|--------------------|------------------- `BodyId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.BodyId.html) -`CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser] | [src/libsyntax/codemap.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html) +`CodeMap` | struct | Maps AST nodes to their source code. It is composed of `FileMap`s | [The parser] | [src/libsyntax/codemap.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html) `CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver] | [src/librustc_driver/driver.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/struct.CompileState.html) `ast::Crate` | struct | A syntax-level representation of a parsed crate | [The parser] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Crate.html) `hir::Crate` | struct | A more abstract, compiler-friendly form of a crate's AST | [The Hir] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Crate.html) `DefId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [src/librustc/hir/def_id.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/def_id/struct.DefId.html) `DiagnosticBuilder` | struct | A struct for building up compiler diagnostics, such as errors or lints | [Emitting Diagnostics] | [src/librustc_errors/diagnostic_builder.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.DiagnosticBuilder.html) `DocContext` | struct | A state container used by rustdoc when crawling through a crate to gather its documentation | [Rustdoc] | [src/librustdoc/core.rs](https://github.com/rust-lang/rust/blob/master/src/librustdoc/core.rs) -`FileMap` | struct | Part of the `CodeMap`, mapping a single source file | [The parser] | [src/libsyntax_pos/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.FileMap.html) +`FileMap` | struct | Part of the `CodeMap`. Maps AST nodes to their source code for a single source file | [The parser] | [src/libsyntax_pos/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.FileMap.html) `HirId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.HirId.html) `NodeId` | struct | One of four types of HIR node identifiers. Being phased out | [Identifiers in the HIR] | [src/libsyntax/ast.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.NodeId.html) `ParamEnv` | struct | Information about generic parameters or `Self`, useful for working with associated or generic items | [Parameter Environment] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.ParamEnv.html) From 315ae44507f5a2691083be9f40b28945cffa7762 Mon Sep 17 00:00:00 2001 From: mark Date: Sat, 14 Jul 2018 15:58:17 -0500 Subject: [PATCH 289/648] add a bit about buffered lints --- src/diag.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/src/diag.md b/src/diag.md index 67bd4829c..3690d6152 100644 --- a/src/diag.md +++ b/src/diag.md @@ -268,3 +268,41 @@ For example, This defines the `nonstandard_style` group which turns on the listed lints. A user can turn on these lints with a `!#[warn(nonstandard_style)]` attribute in the source code, or by passing `-W nonstandard-style` on the command line. + +### Linting early in the compiler + +On occasion, you may need to define a lint that runs before the linting system +has been initialized (e.g. during parsing or macro expansion). This is +problematic because we need to have computed lint levels to know whether we +should emit a warning or an error or nothing at all. + +To solve this problem, we buffer the lints until the linting system is +processed. [`Session`][sessbl] and [`ParseSess`][parsebl] both have +`buffer_lint` methods that allow you to buffer a lint for later. The linting +system automatically takes care of handling buffered lints later. + +[sessbl]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html#method.buffer_lint +[parsebl]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/struct.ParseSess.html#method.buffer_lint + +Thus, to define a lint that runs early in the compilation, one defines a lint +like normal but invokes the lint with `buffer_lint`. + +#### Linting even earlier in the compiler + +The parser (`libsyntax`) is interesting in that it cannot have dependencies on +any of the other `librustc*` crates. In particular, it cannot depend on +`librustc::lint` or `librustc_lint`, where all of the compiler linting +infrastructure is defined. That's troublesome! + +To solve this, `libsyntax` defines its own buffered lint type, which +`ParseSess::buffer_lint` uses. After macro expansion, these buffered lints are +then dumped into the `Session::buffered_lints` used by the rest of the compiler. + +Usage for buffered lints in `libsyntax` is pretty much the same as the rest of +the compiler with one exception because we cannot import the `LintId`s for +lints we want to emit. Instead, the [`BufferedEarlyLintId`] type is used. If you +are defining a new lint, you will want to add an entry to this enum. Then, add +an appropriate mapping to the body of [`Lint::from_parser_lint_id`][fplid]. + +[`BufferedEarlyLintId`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/early_buffered_lints/struct.BufferedEarlyLintId.html +[fplid]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/lint/struct.Lint.html#from_parser_lint_id From 57c8403c88fd40053ba4dadc4ec213748786e527 Mon Sep 17 00:00:00 2001 From: mark Date: Sun, 8 Jul 2018 18:28:51 -0500 Subject: [PATCH 290/648] add a bit on llvm --- src/codegen.md | 49 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 48 insertions(+), 1 deletion(-) diff --git a/src/codegen.md b/src/codegen.md index f5894baa7..279875995 100644 --- a/src/codegen.md +++ b/src/codegen.md @@ -1 +1,48 @@ -# Generating LLVM IR +# Code generation + +Code generation or "codegen" is the part of the compiler that actually generates +an executable binary. rustc uses LLVM for code generation. + +## What is LLVM? + +All of the preceeding chapters of this guide have one thing in common: we never +generated any executable machine code at all! With this chapter, all of that +changes. + +Like most compilers, rustc is composed of a "frontend" and a "backend". The +"frontend" is responsible for taking raw source code, checking it for +correctness, and getting it into a format `X` from which we can generate +executable machine code. The "backend" then takes that format `X` and produces +(possibly optimized) executable machine code for some platform. All of the +previous chapters deal with rustc's frontend. + +rustc's backend is [LLVM](https://llvm.org), "a collection of modular and +reusable compiler and toolchain technologies". In particular, the LLVM project +contains a pluggable compiler backend (also called "LLVM"), which is used by +many compiler projects, including the `clang` C compiler and our beloved +`rustc`. + +LLVM's "format `X`" is called LLVM IR. It is basically assembly code with +additional low-level types and annotations added. These annotations are helpful +for doing optimizations on the LLVM IR and outputed machine code. The end result +of all this is (at long last) something executable (e.g. an ELF object or wasm). + +There are a few benefits to using LLVM: + +- We don't have to write a whole compiler backend. This reduces implementation + and maintainance burden. +- We benefit from the large suite of advanced optimizations that the LLVM + project has been collecting. +- We automatically can compile Rust to any of the platforms for which LLVM has + support. For example, as soon as LLVM added support for wasm, voila! rustc, + clang, and a bunch of other languages were able to compile to wasm! (Well, + there was some extra stuff to be done, but we were 90% there anyway). +- We and other compiler projects benefit from each other. For example, when the + [Spectre and Meltdown security vulnerabilities][spectre] were discovered, only LLVM + needed to be patched. + +[spectre]: https://meltdownattack.com/ + +## Generating LLVM IR + +TODO From 894893860dd717d6e81b900c6f196f6c1670098b Mon Sep 17 00:00:00 2001 From: mark Date: Sun, 8 Jul 2018 18:31:56 -0500 Subject: [PATCH 291/648] line length --- src/codegen.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/src/codegen.md b/src/codegen.md index 279875995..cb99e4d38 100644 --- a/src/codegen.md +++ b/src/codegen.md @@ -1,7 +1,7 @@ # Code generation -Code generation or "codegen" is the part of the compiler that actually generates -an executable binary. rustc uses LLVM for code generation. +Code generation or "codegen" is the part of the compiler that actually +generates an executable binary. rustc uses LLVM for code generation. ## What is LLVM? @@ -24,8 +24,9 @@ many compiler projects, including the `clang` C compiler and our beloved LLVM's "format `X`" is called LLVM IR. It is basically assembly code with additional low-level types and annotations added. These annotations are helpful -for doing optimizations on the LLVM IR and outputed machine code. The end result -of all this is (at long last) something executable (e.g. an ELF object or wasm). +for doing optimizations on the LLVM IR and outputed machine code. The end +result of all this is (at long last) something executable (e.g. an ELF object +or wasm). There are a few benefits to using LLVM: @@ -38,8 +39,8 @@ There are a few benefits to using LLVM: clang, and a bunch of other languages were able to compile to wasm! (Well, there was some extra stuff to be done, but we were 90% there anyway). - We and other compiler projects benefit from each other. For example, when the - [Spectre and Meltdown security vulnerabilities][spectre] were discovered, only LLVM - needed to be patched. + [Spectre and Meltdown security vulnerabilities][spectre] were discovered, + only LLVM needed to be patched. [spectre]: https://meltdownattack.com/ From e0d07aad5f6af37df34816fa7f4da057fa5d2bef Mon Sep 17 00:00:00 2001 From: mark Date: Sun, 15 Jul 2018 18:49:03 -0500 Subject: [PATCH 292/648] add notes about generating llvm ir --- src/compiler-debugging.md | 170 ++++++++++++++++++++++++++------------ 1 file changed, 117 insertions(+), 53 deletions(-) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index c8fb6dd08..d1171e0e0 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -1,4 +1,4 @@ -**Note: This is copied from the +**Note: This is copied from the [rust-forge](https://github.com/rust-lang-nursery/rust-forge). If anything needs updating, please open an issue or make a PR on the github repo.** @@ -16,7 +16,7 @@ normal Rust programs. IIRC backtraces **don't work** on Mac and on MinGW, sorry. If you have trouble or the backtraces are full of `unknown`, you might want to find some way to use Linux or MSVC on Windows. -In the default configuration, you don't have line numbers enabled, so the +In the default configuration, you don't have line numbers enabled, so the backtrace looks like this: ```text @@ -36,8 +36,8 @@ stack backtrace: 37: rustc_driver::run_compiler ``` -If you want line numbers for the stack trace, you can enable -`debuginfo-lines=true` or `debuginfo=true` in your config.toml and rebuild the +If you want line numbers for the stack trace, you can enable +`debuginfo-lines=true` or `debuginfo=true` in your config.toml and rebuild the compiler. Then the backtrace will look like this: ```text @@ -110,16 +110,16 @@ note: rustc 1.24.0-dev running on x86_64-unknown-linux-gnu note: run with `RUST_BACKTRACE=1` for a backtrace -thread 'rustc' panicked at 'encountered error with `-Z treat_err_as_bug', +thread 'rustc' panicked at 'encountered error with `-Z treat_err_as_bug', /home/user/rust/src/librustc_errors/lib.rs:411:12 -note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose +note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. stack backtrace: (~~~ IRRELEVANT PART OF BACKTRACE REMOVED BY ME ~~~) - 7: rustc::traits::error_reporting::>::report_selection_error at /home/user/rust/src/librustc/traits/error_reporting.rs:823 - 8: rustc::traits::error_reporting::>::report_fulfillment_errors at /home/user/rust/src/librustc/traits/error_reporting.rs:160 at /home/user/rust/src/librustc/traits/error_reporting.rs:112 @@ -136,7 +136,7 @@ $ # Cool, now I have a backtrace for the error The compiler has a lot of `debug!` calls, which print out logging information at many points. These are very useful to at least narrow down the location of -a bug if not to find it entirely, or just to orient yourself as to why the +a bug if not to find it entirely, or just to orient yourself as to why the compiler is doing a particular thing. To see the logs, you need to set the `RUST_LOG` environment variable to @@ -191,9 +191,9 @@ want to call `x.py clean` to force one. ### Logging etiquette Because calls to `debug!` are removed by default, in most cases, don't worry -about adding "unnecessary" calls to `debug!` and leaving them in code you -commit - they won't slow down the performance of what we ship, and if they -helped you pinning down a bug, they will probably help someone else with a +about adding "unnecessary" calls to `debug!` and leaving them in code you +commit - they won't slow down the performance of what we ship, and if they +helped you pinning down a bug, they will probably help someone else with a different one. However, there are still a few concerns that you might care about: @@ -201,27 +201,27 @@ However, there are still a few concerns that you might care about: ### Expensive operations in logs A note of caution: the expressions *within* the `debug!` call are run -whenever RUST_LOG is set, even if the filter would exclude the log. This means +whenever RUST_LOG is set, even if the filter would exclude the log. This means that if in the module `rustc::foo` you have a statement ```Rust debug!("{:?}", random_operation(tcx)); ``` -Then if someone runs a debug `rustc` with `RUST_LOG=rustc::bar`, then -`random_operation()` will still run - even while it's output will never be +Then if someone runs a debug `rustc` with `RUST_LOG=rustc::bar`, then +`random_operation()` will still run - even while it's output will never be needed! This means that you should not put anything too expensive or likely -to crash there - that would annoy anyone who wants to use logging for their own -module. Note that if `RUST_LOG` is unset (the default), then the code will not -run - this means that if your logging code panics, then no-one will know it +to crash there - that would annoy anyone who wants to use logging for their own +module. Note that if `RUST_LOG` is unset (the default), then the code will not +run - this means that if your logging code panics, then no-one will know it until someone tries to use logging to find *another* bug. -If you *need* to do an expensive operation in a log, be aware that while log -expressions are *evaluated* even if logging is not enabled in your module, -they are not *formatted* unless it *is*. This means you can put your -expensive/crashy operations inside an `fmt::Debug` impl, and they will not be +If you *need* to do an expensive operation in a log, be aware that while log +expressions are *evaluated* even if logging is not enabled in your module, +they are not *formatted* unless it *is*. This means you can put your +expensive/crashy operations inside an `fmt::Debug` impl, and they will not be run unless your log is enabled: ```Rust @@ -246,7 +246,7 @@ debug!("{:?}", ExpensiveOperationContainer { tcx }); ## Formatting Graphviz output (.dot files) [formatting-graphviz-output]: #formatting-graphviz-output -Some compiler options for debugging specific features yield graphviz graphs - +Some compiler options for debugging specific features yield graphviz graphs - e.g. the `#[rustc_mir(borrowck_graphviz_postflow="suffix.dot")]` attribute dumps various borrow-checker dataflow graphs. @@ -261,30 +261,66 @@ $ firefox maybe_init_suffix.pdf # Or your favorite pdf viewer ## Debugging LLVM [debugging-llvm]: #debugging-llvm -LLVM is a big project on its own that probably needs to have its own debugging -document (not that I could find one). But here are some tips that are important -in a rustc context: +> NOTE: If you are looking for info about code generation, please see [this +> chapter][codegen] instead. + +[codegen]: codegen.html + +This section is about debugging compiler bugs in code generation (e.g. why the +compiler generated some piece of code or crashed in LLVM). LLVM is a big +project on its own that probably needs to have its own debugging document (not +that I could find one). But here are some tips that are important in a rustc +context: + +As a general rule, compilers generate lots of information from analyzing code. +Thus, a useful first step is usually to find a minimal example. One way to do +this is to + +1. create a new crate that reproduces the issue (e.g. adding whatever crate is +at fault as a dependency, and using it from there) + +2. minimize the crate by removing external dependencies; that is, moving +everything relevant to the new crate + +3. further minimize the issue by making the code shorter (there are tools that +help with this like `creduce`) The official compilers (including nightlies) have LLVM assertions disabled, which means that LLVM assertion failures can show up as compiler crashes (not ICEs but "real" crashes) and other sorts of weird behavior. If you are encountering these, it is a good idea to try using a compiler with LLVM assertions enabled - either an "alt" nightly or a compiler you build yourself -by setting `[llvm] assertions=true` in your config.toml - and -see whether anything turns up. +by setting `[llvm] assertions=true` in your config.toml - and see whether +anything turns up. -The rustc build process builds the LLVM tools into +The rustc build process builds the LLVM tools into `./build//llvm/bin`. They can be called directly. -The default rustc compilation pipeline has multiple codegen units, which is hard -to replicate manually and means that LLVM is called multiple times in parallel. -If you can get away with it (i.e. if it doesn't make your bug disappear), -passing `-C codegen-units=1` to rustc will make debugging easier. - -If you want to play with the optimization pipeline, you can use the opt tool -from `./build//llvm/bin/` with the the LLVM IR emitted by rustc. -Note that rustc emits different IR depending on whether `-O` is enabled, even -without LLVM's optimizations, so if you want to play with the IR rustc emits, +The default rustc compilation pipeline has multiple codegen units, which is +hard to replicate manually and means that LLVM is called multiple times in +parallel. If you can get away with it (i.e. if it doesn't make your bug +disappear), passing `-C codegen-units=1` to rustc will make debugging easier. + +To rustc to generate LLVM IR, you need to pass the `--emit=llvm-ir` flag. If +you are building via cargo, use the `RUSTFLAGS` environment variable (e.g. +`RUSTFLAGS='--emit=llvm-ir'`). This causes rustc to spit out LLVM IR into the +target directory. + +`cargo llvm-ir [options] path` spits out the LLVM IR for a particular function +at `path`. (`cargo install cargo-asm` installs `cargo asm` and `cargo +llvm-ir`). `--build-type=debug` emits code for debug builds. There are also +other useful options. Also, debug info in LLVM IR can clutter the output a lot: +`RUSTFLAGS="-C debuginfo=0"` is really useful. + +`RUSTFLAGS="-C save-temps"` outputs LLVM bitcode (not the same as IR) at +different stages during compilation, which is sometimes useful. One just needs +to convert the bitcode files to `.ll` files using `llvm-dis` which should be in +the target local compilation of rustc. + +If you want to play with the optimization pipeline, you can use the `opt` tool +from `./build//llvm/bin/` with the LLVM IR emitted by rustc. Note +that rustc emits different IR depending on whether `-O` is enabled, even +without LLVM's optimizations, so if you want to play with the IR rustc emits, you should: ```bash @@ -295,21 +331,21 @@ $ $OPT -S -O2 < my-file.ll > my ``` If you just want to get the LLVM IR during the LLVM pipeline, to e.g. see which -IR causes an optimization-time assertion to fail, or to see when -LLVM performs a particular optimization, you can pass the rustc flag -`-C llvm-args=-print-after-all`, and possibly add -`-C llvm-args='-filter-print-funcs=EXACT_FUNCTION_NAME` (e.g. -`-C llvm-args='-filter-print-funcs=_ZN11collections3str21_$LT$impl$u20$str$GT$\ - 7replace17hbe10ea2e7c809b0bE'`). - -That produces a lot of output into standard error, so you'll want to pipe -that to some file. Also, if you are using neither `-filter-print-funcs` nor -`-C codegen-units=1`, then, because the multiple codegen units run in parallel, -the printouts will mix together and you won't be able to read anything. - -If you want just the IR for a specific function (say, you want to see -why it causes an assertion or doesn't optimize correctly), you can use -`llvm-extract`, e.g. +IR causes an optimization-time assertion to fail, or to see when LLVM performs +a particular optimization, you can pass the rustc flag `-C +llvm-args=-print-after-all`, and possibly add `-C +llvm-args='-filter-print-funcs=EXACT_FUNCTION_NAME` (e.g. `-C +llvm-args='-filter-print-funcs=_ZN11collections3str21_$LT$impl$u20$str$GT$\ +7replace17hbe10ea2e7c809b0bE'`). + +That produces a lot of output into standard error, so you'll want to pipe that +to some file. Also, if you are using neither `-filter-print-funcs` nor `-C +codegen-units=1`, then, because the multiple codegen units run in parallel, the +printouts will mix together and you won't be able to read anything. + +If you want just the IR for a specific function (say, you want to see why it +causes an assertion or doesn't optimize correctly), you can use `llvm-extract`, +e.g. ```bash $ ./build/$TRIPLE/llvm/bin/llvm-extract \ @@ -319,4 +355,32 @@ $ ./build/$TRIPLE/llvm/bin/llvm-extract \ > extracted.ll ``` +### Filing LLVM bug reports + +When filing an LLVM bug report, you will probably want some sort of minimal +working example that demonstrates the problem. The Godbolt compiler explorer is +really helpful for this. + +1. Once you have some LLVM IR for the problematic code (see above), you can +create a minimal working example with Godbolt. Go to +[gcc.godbolt.org](https://gcc.godbolt.org). + +2. Choose `LLVM-IR` as programming language. + +3. Use `llc` to compile the IR to a particular target as is: + - There are some useful flags: `-mattr` enables target features, `-march=` + selects the target, `-mcpu=` selects the CPU, etc. + - Commands like `llc -march=help` output all architectures available, which + is useful because sometimes the Rust arch names and the LLVM names do not + match. + - If you have compiled rustc yourself somewhere, in the target directory + you have binaries for `llc`, `opt`, etc. + +4. If you want to optimize the LLVM-IR, you can use `opt` to see how the LLVM + optimizations transform it. + +5. Once you have a godbolt link demonstrating the issue, it is pretty easy to + fill in an LLVM bug. + + [env-logger]: https://docs.rs/env_logger/0.4.3/env_logger/ From 2a1387faae2ec30e5261f0afe7c5474fb462a6fb Mon Sep 17 00:00:00 2001 From: mark Date: Sun, 15 Jul 2018 18:51:41 -0500 Subject: [PATCH 293/648] minor improvements --- src/SUMMARY.md | 2 +- src/codegen.md | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index a41f78a1a..206bf5d36 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -54,7 +54,7 @@ - [Constant evaluation](./const-eval.md) - [miri const evaluator](./miri.md) - [Parameter Environments](./param_env.md) -- [Generating LLVM IR](./codegen.md) +- [Code Generation](./codegen.md) - [Emitting Diagnostics](./diag.md) --- diff --git a/src/codegen.md b/src/codegen.md index cb99e4d38..6e68db6ea 100644 --- a/src/codegen.md +++ b/src/codegen.md @@ -3,6 +3,11 @@ Code generation or "codegen" is the part of the compiler that actually generates an executable binary. rustc uses LLVM for code generation. +> NOTE: If you are looking for hints on how to debug code generation bugs, +> please see [this section of the debugging chapter][debug]. + +[debug]: compiler-debugging.html#debugging-llvm + ## What is LLVM? All of the preceeding chapters of this guide have one thing in common: we never From 37141c0825b130cf30524d40a8cea20097fb64a4 Mon Sep 17 00:00:00 2001 From: Tom Tromey Date: Thu, 9 Aug 2018 09:05:36 -0600 Subject: [PATCH 294/648] Update compiler test documentation Update the compiler test documentation to document ignore-gdb-version and min-system-llvm-version; and expand the min-gdb-version, min-lldb-version, and min-llvm-version documentation a little. --- src/tests/adding.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/src/tests/adding.md b/src/tests/adding.md index a9400c591..7469755f7 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -129,6 +129,8 @@ be compiled or run. * `ignore-test` always ignores the test * `ignore-lldb` and `ignore-gdb` will skip a debuginfo test on that debugger. +* `ignore-gdb-version` can be used to ignore the test when certain gdb + versions are used Some examples of `X` in `ignore-X`: @@ -157,8 +159,16 @@ source. source is compiled, and this compilation is required to succeed. The `.fixed` file can also be generated automatically with the `--bless` option, discussed [below](#bless). -* `min-{gdb,lldb}-version` -* `min-llvm-version` +* `min-gdb-version` specifies the minimum gdb version required for + this test; see also `ignore-gdb-version` +* `min-lldb-version` specifies the minimum lldb version required for + this test +* `min-llvm-version` specifies the minimum llvm version required for + this test +* `min-system-llvm-version` specifies the minimum system llvm version + required for this test; the test is ignored if the system llvm is in + use and it doesn't meet the minimum version. This is useful when an + llvm feature has been backported to rust-llvm * `compile-pass` for UI tests, indicates that the test is supposed to compile, as opposed to the default where the test is supposed to error out. From c13c5a019faab37ee03b259688bf89e73a2ce4d9 Mon Sep 17 00:00:00 2001 From: John Renner Date: Tue, 24 Jul 2018 08:58:25 -0700 Subject: [PATCH 295/648] Add testing chapter --- src/SUMMARY.md | 1 + src/testing.md | 145 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 src/testing.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 206bf5d36..27a6e38f2 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -17,6 +17,7 @@ - [Incremental compilation](./incremental-compilation.md) - [Debugging and Testing](./incrcomp-debugging.md) - [The parser](./the-parser.md) +- [Testing](./testing.md) - [Macro expansion](./macro-expansion.md) - [Name resolution](./name-resolution.md) - [The HIR (High-level IR)](./hir.md) diff --git a/src/testing.md b/src/testing.md new file mode 100644 index 000000000..abe6849a7 --- /dev/null +++ b/src/testing.md @@ -0,0 +1,145 @@ +### The `#[test]` attribute +Today, rust programmers rely on a built in attribute called `#[test]`. +All you have to do is mark a function as a test and include some asserts like so: + +```rust,ignore +#[test] +fn my_test() { + assert!(2+2 == 4); +} +``` + +When this program is compiled using `rustc --test` or `cargo test`, it will +produce an executable that can run this, and any other test function. This +method of testing allows tests to live alongside code in an organic way. You +can even put tests inside private modules: + +```rust,ignore +mod my_priv_mod { + fn my_priv_func() -> bool {} + + #[test] + fn test_priv_func() { + assert!(my_priv_func()); + } +} +``` +Private items can thus be easily tested without worrying about how to expose +the them to any sort of external testing apparatus. This is key to the +ergonomics of testing in Rust. Semantically, however, it's rather odd. +How does any sort of `main` function invoke these tests if they're not visible? +What exactly is `rustc --test` doing? + +`#[test]` is implemented as a syntactic transformation inside the compiler's +[`libsyntax` crate][libsyntax]. Essentially, it's a fancy macro, that +rewrites the crate in 3 steps: + +#### Step 1: Re-Exporting + +As mentioned earlier, tests can exist inside private modules, so we need a way of +exposing them to the main function, without breaking any existing code. To that end, +`libsyntax` will create local modules called `__test_reexports` that recursively reexport tests. +This expansion translates the above example into: + +```rust,ignore +mod my_priv_mod { + fn my_priv_func() -> bool {} + + pub fn test_priv_func() { + assert!(my_priv_func()); + } + + pub mod __test_reexports { + pub use super::test_priv_func; + } +} +``` + +Now, our test can be accessed as +`my_priv_mod::__test_reexports::test_priv_func`. For deeper module +structures, `__test_reexports` will reexport modules that contain tests, so a +test at `a::b::my_test` becomes +`a::__test_reexports::b::__test_reexports::my_test`. While this process seems +pretty safe, what happens if there is an existing `__test_reexports` module? +The answer: nothing. + +To explain, we need to understand [how the AST represents +identifiers][Ident]. The name of every function, variable, module, etc. is +not stored as a string, but rather as an opaque [Symbol][Symbol] which is +essentially an ID number for each identifier. The compiler keeps a separate +hashtable that allows us to recover the human-readable name of a Symbol when +necessary (such as when printing a syntax error). When the compiler generates +the `__test_reexports` module, it generates a new Symbol for the identifier, +so while the compiler-generated `__test_reexports` may share a name with your +hand-written one, it will not share a Symbol. This technique prevents name +collision during code generation and is the foundation of Rust's macro +hygiene. + +#### Step 2: Harness Generation +Now that our tests are accessible from the root of our crate, we need to do something with them. +`libsyntax` generates a module like so: + +```rust,ignore +pub mod __test { + extern crate test; + const TESTS: &'static [self::test::TestDescAndFn] = &[/*...*/]; + + #[main] + pub fn main() { + self::test::test_static_main(TESTS); + } +} +``` + +While this transformation is simple, it gives us a lot of insight into how tests are actually run. +The tests are aggregated into an array and passed to a test runner called `test_static_main`. +We'll come back to exactly what `TestDescAndFn` is, but for now, the key takeaway is that there is a crate +called [`test`][test] that is part of Rust core, that implements all of the runtime for testing. `test`'s interface is unstable, +so the only stable way to interact with it is through the `#[test]` macro. + +#### Step 3: Test Object Generation +If you've written tests in Rust before, you may be familiar with some of the optional attributes available on test functions. +For example, a test can be annotated with `#[should_panic]` if we expect the test to cause a panic. It looks something like this: + +```rust,ignore +#[test] +#[should_panic] +fn foo() { + panic!("intentional"); +} +``` + +This means our tests are more than just simple functions, they have configuration information as well. `test` encodes this configuration +data into a struct called [`TestDesc`][TestDesc]. For each test function in a crate, `libsyntax` will parse its attributes and generate a `TestDesc` instance. +It then combines the `TestDesc` and test function into the predictably named `TestDescAndFn` struct, that `test_static_main` operates on. +For a given test, the generated `TestDescAndFn` instance looks like so: + +```rust,ignore +self::test::TestDescAndFn{ + desc: self::test::TestDesc{ + name: self::test::StaticTestName("foo"), + ignore: false, + should_panic: self::test::ShouldPanic::Yes, + allow_fail: false, + }, + testfn: self::test::StaticTestFn(|| + self::test::assert_test_result(::crate::__test_reexports::foo())), +} +``` + +Once we've constructed an array of these test objects, they're passed to the +test runner via the harness generated in step 2. + +### Inspecting the generated code +On nightly rust, there's an unstable flag called `unpretty` that you can use to print out the module source after macro expansion: + +```bash +$ rustc my_mod.rs -Z unpretty=hir +``` + +[test]: https://doc.rust-lang.org/test/index.html +[TestDesc]: https://doc.rust-lang.org/test/struct.TestDesc.html +[Symbol]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Ident.html +[Ident]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Ident.html +[eRFC]: https://github.com/rust-lang/rfcs/blob/master/text/2318-custom-test-frameworks.md +[libsyntax]: https://github.com/rust-lang/rust/tree/master/src/libsyntax \ No newline at end of file From 61adf4cb4b90ec74fdddccf425dde4eff88461ac Mon Sep 17 00:00:00 2001 From: John Renner Date: Tue, 24 Jul 2018 10:41:01 -0700 Subject: [PATCH 296/648] Shorten line length --- src/testing.md | 49 +++++++++++++++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 20 deletions(-) diff --git a/src/testing.md b/src/testing.md index abe6849a7..3c93ad619 100644 --- a/src/testing.md +++ b/src/testing.md @@ -1,6 +1,6 @@ ### The `#[test]` attribute -Today, rust programmers rely on a built in attribute called `#[test]`. -All you have to do is mark a function as a test and include some asserts like so: +Today, rust programmers rely on a built in attribute called `#[test]`. All +you have to do is mark a function as a test and include some asserts like so: ```rust,ignore #[test] @@ -36,10 +36,11 @@ rewrites the crate in 3 steps: #### Step 1: Re-Exporting -As mentioned earlier, tests can exist inside private modules, so we need a way of -exposing them to the main function, without breaking any existing code. To that end, -`libsyntax` will create local modules called `__test_reexports` that recursively reexport tests. -This expansion translates the above example into: +As mentioned earlier, tests can exist inside private modules, so we need a +way of exposing them to the main function, without breaking any existing +code. To that end, `libsyntax` will create local modules called +`__test_reexports` that recursively reexport tests. This expansion translates +the above example into: ```rust,ignore mod my_priv_mod { @@ -76,8 +77,8 @@ collision during code generation and is the foundation of Rust's macro hygiene. #### Step 2: Harness Generation -Now that our tests are accessible from the root of our crate, we need to do something with them. -`libsyntax` generates a module like so: +Now that our tests are accessible from the root of our crate, we need to do +something with them. `libsyntax` generates a module like so: ```rust,ignore pub mod __test { @@ -91,15 +92,19 @@ pub mod __test { } ``` -While this transformation is simple, it gives us a lot of insight into how tests are actually run. -The tests are aggregated into an array and passed to a test runner called `test_static_main`. -We'll come back to exactly what `TestDescAndFn` is, but for now, the key takeaway is that there is a crate -called [`test`][test] that is part of Rust core, that implements all of the runtime for testing. `test`'s interface is unstable, -so the only stable way to interact with it is through the `#[test]` macro. +While this transformation is simple, it gives us a lot of insight into how +tests are actually run. The tests are aggregated into an array and passed to +a test runner called `test_static_main`. We'll come back to exactly what +`TestDescAndFn` is, but for now, the key takeaway is that there is a crate +called [`test`][test] that is part of Rust core, that implements all of the +runtime for testing. `test`'s interface is unstable, so the only stable way +to interact with it is through the `#[test]` macro. #### Step 3: Test Object Generation -If you've written tests in Rust before, you may be familiar with some of the optional attributes available on test functions. -For example, a test can be annotated with `#[should_panic]` if we expect the test to cause a panic. It looks something like this: +If you've written tests in Rust before, you may be familiar with some of the +optional attributes available on test functions. For example, a test can be +annotated with `#[should_panic]` if we expect the test to cause a panic. It +looks something like this: ```rust,ignore #[test] @@ -109,10 +114,13 @@ fn foo() { } ``` -This means our tests are more than just simple functions, they have configuration information as well. `test` encodes this configuration -data into a struct called [`TestDesc`][TestDesc]. For each test function in a crate, `libsyntax` will parse its attributes and generate a `TestDesc` instance. -It then combines the `TestDesc` and test function into the predictably named `TestDescAndFn` struct, that `test_static_main` operates on. -For a given test, the generated `TestDescAndFn` instance looks like so: +This means our tests are more than just simple functions, they have +configuration information as well. `test` encodes this configuration data +into a struct called [`TestDesc`][TestDesc]. For each test function in a +crate, `libsyntax` will parse its attributes and generate a `TestDesc` +instance. It then combines the `TestDesc` and test function into the +predictably named `TestDescAndFn` struct, that `test_static_main` operates +on. For a given test, the generated `TestDescAndFn` instance looks like so: ```rust,ignore self::test::TestDescAndFn{ @@ -131,7 +139,8 @@ Once we've constructed an array of these test objects, they're passed to the test runner via the harness generated in step 2. ### Inspecting the generated code -On nightly rust, there's an unstable flag called `unpretty` that you can use to print out the module source after macro expansion: +On nightly rust, there's an unstable flag called `unpretty` that you can use +to print out the module source after macro expansion: ```bash $ rustc my_mod.rs -Z unpretty=hir From 656831723492b241004dcd06df6f2dd619d2e4f3 Mon Sep 17 00:00:00 2001 From: John Renner Date: Tue, 7 Aug 2018 10:59:52 -0700 Subject: [PATCH 297/648] More descriptive name for #[test] implementation --- src/SUMMARY.md | 2 +- src/{testing.md => test-implementation.md} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename src/{testing.md => test-implementation.md} (100%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 27a6e38f2..ca3b4c020 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -17,7 +17,7 @@ - [Incremental compilation](./incremental-compilation.md) - [Debugging and Testing](./incrcomp-debugging.md) - [The parser](./the-parser.md) -- [Testing](./testing.md) +- [`#[test]` Implementation](./test-implementation.md) - [Macro expansion](./macro-expansion.md) - [Name resolution](./name-resolution.md) - [The HIR (High-level IR)](./hir.md) diff --git a/src/testing.md b/src/test-implementation.md similarity index 100% rename from src/testing.md rename to src/test-implementation.md From 1f8e11729293468e206b193f8962eca6d2a0c4ce Mon Sep 17 00:00:00 2001 From: Josh Triplett Date: Sat, 18 Aug 2018 12:54:42 -0700 Subject: [PATCH 298/648] Update copyright guidance to omit notices for new files. Previously approved in https://github.com/rust-lang/rust/pull/43498 ; update the guide to match. --- src/compiletest.md | 10 ---------- src/conventions.md | 23 +++++------------------ src/tests/adding.md | 4 ---- 3 files changed, 5 insertions(+), 32 deletions(-) diff --git a/src/compiletest.md b/src/compiletest.md index 0837fb819..40d06136e 100644 --- a/src/compiletest.md +++ b/src/compiletest.md @@ -63,16 +63,6 @@ header command and the argument list (if present) are typically separated by a colon: ```rust,ignore -// Copyright 2018 The Rust Project Developers. See the COPYRIGHT -// file at the top-level directory of this distribution and at -// http://rust-lang.org/COPYRIGHT. -// -// Licensed under the Apache License, Version 2.0 or the MIT license -// , at your -// option. This file may not be copied, modified, or distributed -// except according to those terms. - // must-compile-successfully // failure-status: 1 diff --git a/src/conventions.md b/src/conventions.md index 3f30dac73..93c5c44be 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -20,24 +20,11 @@ in isolation with `./x.py test src/tools/tidy`. ### Copyright notice -All files must begin with the following copyright notice: - -```rust -// Copyright 2012-2013 The Rust Project Developers. See the COPYRIGHT -// file at the top-level directory of this distribution and at -// http://rust-lang.org/COPYRIGHT. -// -// Licensed under the Apache License, Version 2.0 or the MIT license -// , at your -// option. This file may not be copied, modified, or distributed -// except according to those terms. -``` - -The year at the top is not meaningful: copyright protections are in -fact automatic from the moment of authorship. We do not typically edit -the years on existing files. When creating a new file, you can use the -current year if you like, but you don't have to. +Some existing files begin with a copyright and license notice. Please omit this +notice for new files licensed under the standard terms (dual MIT/Apache-2.0). +For existing files, the year at the top is not meaningful: copyright +protections are in fact automatic from the moment of authorship. We do not +typically edit the years on existing files. ## Line length diff --git a/src/tests/adding.md b/src/tests/adding.md index 7469755f7..ee264747a 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -101,10 +101,6 @@ this test. For example, this test uses the `// compile-flags` command to specify a custom flag to give to rustc when the test is compiled: ```rust,ignore -// Copyright 2017 The Rust Project Developers. blah blah blah. -// ... -// except according to those terms. - // Test the behavior of `0 - 1` when overflow checks are disabled. // compile-flags: -Coverflow-checks=off From 9efa9f13466dc73ac9152ab44ffeeebd930030fb Mon Sep 17 00:00:00 2001 From: Oliver Schneider Date: Fri, 6 Jul 2018 14:53:53 +0200 Subject: [PATCH 299/648] Explain existential types --- src/SUMMARY.md | 1 + src/existential-types.md | 48 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) create mode 100644 src/existential-types.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index ca3b4c020..981da5c1e 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -45,6 +45,7 @@ - [Type checking](./type-checking.md) - [Method Lookup](./method-lookup.md) - [Variance](./variance.md) + - [Existential Types](./existential-types.md) - [The MIR (Mid-level IR)](./mir/index.md) - [MIR construction](./mir/construction.md) - [MIR visitor and traversal](./mir/visitor.md) diff --git a/src/existential-types.md b/src/existential-types.md new file mode 100644 index 000000000..4c5ebb75c --- /dev/null +++ b/src/existential-types.md @@ -0,0 +1,48 @@ +# Existential Types + +Existential types are essentially strong type aliases which only expose +a specific set of traits as their interface and the concrete type in the +background is inferred from a certain set of use sites of the existential +type. + +In the language they are expressed via + +```rust +existential type Foo: Bar; +``` + +This is in existential type named `Foo` which can be interacted with via +the `Bar` trait's interface. + +Since there needs to be a concrete background type, you can currently +express that type by using the existential type in a "defining use site". + +```rust +struct Struct; +impl Bar for Struct { /* stuff */ } +fn foo() -> Foo { + Struct +} +``` + +Any other "defining use site" needs to produce the exact same type. + +## Defining use site(s) + +Currently only the return value of a function inside can +be a defining use site of an existential type (and only if the return +type of that function contains the existential type). + +The defining use of an existential type can be any code *within* the parent +of the existential type definition. This includes any siblings of the +existential type and all children of the siblings. + +The initiative for *"not causing fatal brain damage to developers due to +accidentally running infinite loops in their brain while trying to +comprehend what the type system is doing"* has decided to disallow children +of existential types to be defining use sites. + +### Associated existential types + +Associated existential types can be defined by any other associated item +on the same trait `impl` or a child of these associated items. \ No newline at end of file From 6533ccebba453402192d82af9b44b62de4054e7f Mon Sep 17 00:00:00 2001 From: Oliver Schneider Date: Wed, 1 Aug 2018 18:35:17 +0200 Subject: [PATCH 300/648] Update existential-types.md --- src/existential-types.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/existential-types.md b/src/existential-types.md index 4c5ebb75c..3e4a49210 100644 --- a/src/existential-types.md +++ b/src/existential-types.md @@ -7,7 +7,7 @@ type. In the language they are expressed via -```rust +``` existential type Foo: Bar; ``` @@ -17,7 +17,7 @@ the `Bar` trait's interface. Since there needs to be a concrete background type, you can currently express that type by using the existential type in a "defining use site". -```rust +``` struct Struct; impl Bar for Struct { /* stuff */ } fn foo() -> Foo { @@ -45,4 +45,4 @@ of existential types to be defining use sites. ### Associated existential types Associated existential types can be defined by any other associated item -on the same trait `impl` or a child of these associated items. \ No newline at end of file +on the same trait `impl` or a child of these associated items. From 7836a59e48f33c9f45e93a594ca6c165a78df4c2 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 7 Aug 2018 08:38:21 -0400 Subject: [PATCH 301/648] add rust,ignore to code snippets --- src/existential-types.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/existential-types.md b/src/existential-types.md index 3e4a49210..ef20167c1 100644 --- a/src/existential-types.md +++ b/src/existential-types.md @@ -7,7 +7,7 @@ type. In the language they are expressed via -``` +```rust,ignore existential type Foo: Bar; ``` @@ -17,7 +17,7 @@ the `Bar` trait's interface. Since there needs to be a concrete background type, you can currently express that type by using the existential type in a "defining use site". -``` +```rust,ignore struct Struct; impl Bar for Struct { /* stuff */ } fn foo() -> Foo { From 3aa13477d1b8efab7f46374697231ea523d7965d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?S=C3=A9bastien=20Duquette?= Date: Thu, 23 Aug 2018 00:13:02 -0400 Subject: [PATCH 302/648] Rename CodeMap and FileMap to SourceMap and SourceFile. Those types were renamed in https://github.com/rust-lang/rust/pull/52953 --- src/appendix/code-index.md | 4 ++-- src/diag.md | 16 ++++++++-------- src/rustc-driver.md | 4 ++-- src/the-parser.md | 10 +++++----- 4 files changed, 17 insertions(+), 17 deletions(-) diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index b6986c5c0..140b182b1 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -7,20 +7,20 @@ compiler. Item | Kind | Short description | Chapter | Declaration ----------------|----------|-----------------------------|--------------------|------------------- `BodyId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.BodyId.html) -`CodeMap` | struct | Maps AST nodes to their source code. It is composed of `FileMap`s | [The parser] | [src/libsyntax/codemap.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html) `CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver] | [src/librustc_driver/driver.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/struct.CompileState.html) `ast::Crate` | struct | A syntax-level representation of a parsed crate | [The parser] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Crate.html) `hir::Crate` | struct | A more abstract, compiler-friendly form of a crate's AST | [The Hir] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Crate.html) `DefId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [src/librustc/hir/def_id.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/def_id/struct.DefId.html) `DiagnosticBuilder` | struct | A struct for building up compiler diagnostics, such as errors or lints | [Emitting Diagnostics] | [src/librustc_errors/diagnostic_builder.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.DiagnosticBuilder.html) `DocContext` | struct | A state container used by rustdoc when crawling through a crate to gather its documentation | [Rustdoc] | [src/librustdoc/core.rs](https://github.com/rust-lang/rust/blob/master/src/librustdoc/core.rs) -`FileMap` | struct | Part of the `CodeMap`. Maps AST nodes to their source code for a single source file | [The parser] | [src/libsyntax_pos/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.FileMap.html) `HirId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.HirId.html) `NodeId` | struct | One of four types of HIR node identifiers. Being phased out | [Identifiers in the HIR] | [src/libsyntax/ast.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.NodeId.html) `ParamEnv` | struct | Information about generic parameters or `Self`, useful for working with associated or generic items | [Parameter Environment] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.ParamEnv.html) `ParseSess` | struct | This struct contains information about a parsing session | [The parser] | [src/libsyntax/parse/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/struct.ParseSess.html) `Rib` | struct | Represents a single scope of names | [Name resolution] | [src/librustc_resolve/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/struct.Rib.html) `Session` | struct | The data associated with a compilation session | [The parser], [The Rustc Driver] | [src/librustc/session/mod.html](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html) +`SourceFile` | struct | Part of the `SourceMap`. Maps AST nodes to their source code for a single source file | [The parser] | [src/libsyntax_pos/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceFile.html) +`SourceMap` | struct | Maps AST nodes to their source code. It is composed of `SourceFile`s | [The parser] | [src/libsyntax/source_map.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html) `Span` | struct | A location in the user's source code, used for error reporting primarily | [Emitting Diagnostics] | [src/libsyntax_pos/span_encoding.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax_pos/struct.Span.html) `StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser] | [src/libsyntax/parse/lexer/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/lexer/struct.StringReader.html) `syntax::token_stream::TokenStream` | struct | An abstract sequence of tokens, organized into `TokenTree`s | [The parser], [Macro expansion] | [src/libsyntax/tokenstream.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/tokenstream/struct.TokenStream.html) diff --git a/src/diag.md b/src/diag.md index 3690d6152..3d9a6fc37 100644 --- a/src/diag.md +++ b/src/diag.md @@ -9,14 +9,14 @@ This chapter is about how to emit compile errors and lints from the compiler. location in the code being compiled. `Span`s are attached to most constructs in HIR and MIR, allowing for more informative error reporting. -[span]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.Span.html +[span]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.Span.html -A `Span` can be looked up in a [`CodeMap`][codemap] to get a "snippet" useful +A `Span` can be looked up in a [`SourceMap`][sourcemap] to get a "snippet" useful for displaying errors with [`span_to_snippet`][sptosnip] and other similar -methods on the `CodeMap`. +methods on the `SourceMap`. -[codemap]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html -[sptosnip]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html#method.span_to_snippet +[sourcemap]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html +[sptosnip]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html#method.span_to_snippet ## Error messages @@ -58,7 +58,7 @@ let mut err = sess.struct_span_err(sp, "oh no! this is an error!"); // In some cases, you might need to check if `sp` is generated by a macro to // avoid printing weird errors about macro-generated code. -if let Ok(snippet) = sess.codemap().span_to_snippet(sp) { +if let Ok(snippet) = sess.source_map().span_to_snippet(sp) { // Use the snippet to generate a suggested fix err.span_suggestion(suggestion_sp, "try using a qux here", format!("qux {}", snip)); } else { @@ -96,7 +96,7 @@ For example, to make our `qux` suggestion machine-applicable, we would do: ```rust,ignore let mut err = sess.struct_span_err(sp, "oh no! this is an error!"); -if let Ok(snippet) = sess.codemap().span_to_snippet(sp) { +if let Ok(snippet) = sess.source_map().span_to_snippet(sp) { // Add applicability info! err.span_suggestion_with_applicability( suggestion_sp, @@ -214,7 +214,7 @@ impl<'a, 'tcx> LateLintPass<'a, 'tcx> for WhileTrue { if let ast::LitKind::Bool(true) = lit.node { if lit.span.ctxt() == SyntaxContext::empty() { let msg = "denote infinite loops with `loop { ... }`"; - let condition_span = cx.tcx.sess.codemap().def_span(e.span); + let condition_span = cx.tcx.sess.source_map().def_span(e.span); let mut err = cx.struct_span_lint(WHILE_TRUE, condition_span, msg); err.span_suggestion_short(condition_span, "use `loop`", "loop".to_owned()); err.emit(); diff --git a/src/rustc-driver.md b/src/rustc-driver.md index 1550b14e9..397949836 100644 --- a/src/rustc-driver.md +++ b/src/rustc-driver.md @@ -2,7 +2,7 @@ The [`rustc_driver`] is essentially `rustc`'s `main()` function. It acts as the glue for running the various phases of the compiler in the correct order, -managing state such as the [`CodeMap`] \(maps AST nodes to source code), +managing state such as the [`SourceMap`] \(maps AST nodes to source code), [`Session`] \(general build context and error messaging) and the [`TyCtxt`] \(the "typing context", allowing you to query the type system and other cool stuff). The `rustc_driver` crate also provides external users with a method @@ -71,6 +71,6 @@ thread-locals, although you should rarely need to touch it. [`CompileState`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/struct.CompileState.html [`Session`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html [`TyCtxt`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html -[`CodeMap`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html +[`SourceMap`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html [stupid-stats]: https://github.com/nrc/stupid-stats [Appendix A]: appendix/stupid-stats.html diff --git a/src/the-parser.md b/src/the-parser.md index e89c09e9e..62b43c0ee 100644 --- a/src/the-parser.md +++ b/src/the-parser.md @@ -14,7 +14,7 @@ Like most parsers, the parsing process is composed of two main steps, The `syntax` crate contains several main players, -- a [`CodeMap`] for mapping AST nodes to their source code +- a [`SourceMap`] for mapping AST nodes to their source code - the [ast module] contains types corresponding to each AST node - a [`StringReader`] for lexing source code into tokens - the [parser module] and [`Parser`] struct are in charge of actually parsing @@ -23,21 +23,21 @@ The `syntax` crate contains several main players, nodes. The main entrypoint to the parser is via the various `parse_*` functions in the -[parser module]. They let you do things like turn a [`FileMap`][filemap] (e.g. +[parser module]. They let you do things like turn a [`SourceFile`][sourcefile] (e.g. the source in a single file) into a token stream, create a parser from the token stream, and then execute the parser to get a `Crate` (the root AST node). To minimise the amount of copying that is done, both the `StringReader` and `Parser` have lifetimes which bind them to the parent `ParseSess`. This contains -all the information needed while parsing, as well as the `CodeMap` itself. +all the information needed while parsing, as well as the `SourceMap` itself. [libsyntax]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/index.html [rustc_errors]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html [ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree -[`CodeMap`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.CodeMap.html +[`SourceMap`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html [ast module]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/index.html [parser module]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/index.html [`Parser`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/parser/struct.Parser.html [`StringReader`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/lexer/struct.StringReader.html [visit module]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/visit/index.html -[filemap]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/codemap/struct.FileMap.html +[sourcefile]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceFile.html From a28a4cf252e5dfa273f1af1608ea954d6a5711c3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?S=C3=A9bastien=20Duquette?= Date: Thu, 23 Aug 2018 00:23:03 -0400 Subject: [PATCH 303/648] Fix lines over 80 columns. --- src/diag.md | 6 +++--- src/the-parser.md | 7 ++++--- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/src/diag.md b/src/diag.md index 3d9a6fc37..b30ec2eca 100644 --- a/src/diag.md +++ b/src/diag.md @@ -11,9 +11,9 @@ HIR and MIR, allowing for more informative error reporting. [span]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.Span.html -A `Span` can be looked up in a [`SourceMap`][sourcemap] to get a "snippet" useful -for displaying errors with [`span_to_snippet`][sptosnip] and other similar -methods on the `SourceMap`. +A `Span` can be looked up in a [`SourceMap`][sourcemap] to get a "snippet" +useful for displaying errors with [`span_to_snippet`][sptosnip] and other +similar methods on the `SourceMap`. [sourcemap]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html [sptosnip]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html#method.span_to_snippet diff --git a/src/the-parser.md b/src/the-parser.md index 62b43c0ee..ac902d915 100644 --- a/src/the-parser.md +++ b/src/the-parser.md @@ -23,9 +23,10 @@ The `syntax` crate contains several main players, nodes. The main entrypoint to the parser is via the various `parse_*` functions in the -[parser module]. They let you do things like turn a [`SourceFile`][sourcefile] (e.g. -the source in a single file) into a token stream, create a parser from the -token stream, and then execute the parser to get a `Crate` (the root AST node). +[parser module]. They let you do things like turn a [`SourceFile`][sourcefile] +(e.g. the source in a single file) into a token stream, create a parser from +the token stream, and then execute the parser to get a `Crate` (the root AST +node). To minimise the amount of copying that is done, both the `StringReader` and `Parser` have lifetimes which bind them to the parent `ParseSess`. This contains From 581b5fc9cfb3674e9474abb6a6513320722dda01 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?S=C3=A9bastien=20Duquette?= Date: Thu, 23 Aug 2018 21:29:01 -0400 Subject: [PATCH 304/648] address review comment --- src/appendix/code-index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index 140b182b1..75d35c6c5 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -19,8 +19,8 @@ Item | Kind | Short description | Chapter | `ParseSess` | struct | This struct contains information about a parsing session | [The parser] | [src/libsyntax/parse/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/struct.ParseSess.html) `Rib` | struct | Represents a single scope of names | [Name resolution] | [src/librustc_resolve/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/struct.Rib.html) `Session` | struct | The data associated with a compilation session | [The parser], [The Rustc Driver] | [src/librustc/session/mod.html](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html) -`SourceFile` | struct | Part of the `SourceMap`. Maps AST nodes to their source code for a single source file | [The parser] | [src/libsyntax_pos/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceFile.html) -`SourceMap` | struct | Maps AST nodes to their source code. It is composed of `SourceFile`s | [The parser] | [src/libsyntax/source_map.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html) +`SourceFile` | struct | Part of the `SourceMap`. Maps AST nodes to their source code for a single source file. Was previously called FileMap | [The parser] | [src/libsyntax_pos/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceFile.html) +`SourceMap` | struct | Maps AST nodes to their source code. It is composed of `SourceFile`s. Was previously called CodeMap | [The parser] | [src/libsyntax/source_map.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html) `Span` | struct | A location in the user's source code, used for error reporting primarily | [Emitting Diagnostics] | [src/libsyntax_pos/span_encoding.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax_pos/struct.Span.html) `StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser] | [src/libsyntax/parse/lexer/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/lexer/struct.StringReader.html) `syntax::token_stream::TokenStream` | struct | An abstract sequence of tokens, organized into `TokenTree`s | [The parser], [Macro expansion] | [src/libsyntax/tokenstream.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/tokenstream/struct.TokenStream.html) From f893fe80190fc0fd3fdd66ba683959fe56182c4e Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 09:23:14 -0400 Subject: [PATCH 305/648] sprinkle links to rustdoc through the HIR explanation --- src/hir.md | 87 ++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 58 insertions(+), 29 deletions(-) diff --git a/src/hir.md b/src/hir.md index 40a14dc25..171319011 100644 --- a/src/hir.md +++ b/src/hir.md @@ -20,13 +20,15 @@ You can view the HIR representation of your code by passing the ### Out-of-band storage and the `Crate` type -The top-level data-structure in the HIR is the `Crate`, which stores +The top-level data-structure in the HIR is the [`Crate`], which stores the contents of the crate currently being compiled (we only ever construct HIR for the current crate). Whereas in the AST the crate data structure basically just contains the root module, the HIR `Crate` structure contains a number of maps and other things that serve to organize the content of the crate for easier access. +[`Crate`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Crate.html + For example, the contents of individual items (e.g. modules, functions, traits, impls, etc) in the HIR are not immediately accessible in the parents. So, for example, if there is a module item @@ -38,11 +40,13 @@ mod foo { } ``` -then in the HIR the representation of module `foo` (the `Mod` -stuct) would only have the **`ItemId`** `I` of `bar()`. To get the +then in the HIR the representation of module `foo` (the [`Mod`] +struct) would only have the **`ItemId`** `I` of `bar()`. To get the details of the function `bar()`, we would lookup `I` in the `items` map. +[`Mod`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Mod.html + One nice result from this representation is that one can iterate over all items in the crate by iterating over the key-value pairs in these maps (without the need to trawl through the whole HIR). @@ -51,12 +55,14 @@ as well as "bodies" (explained below). The other reason to set up the representation this way is for better integration with incremental compilation. This way, if you gain access -to an `&hir::Item` (e.g. for the mod `foo`), you do not immediately +to an [`&hir::Item`] (e.g. for the mod `foo`), you do not immediately gain access to the contents of the function `bar()`. Instead, you only gain access to the **id** for `bar()`, and you must invoke some -function to lookup the contents of `bar()` given its id; this gives the -compiler a chance to observe that you accessed the data for `bar()`, -and then record the dependency. +function to lookup the contents of `bar()` given its id; this gives +the compiler a chance to observe that you accessed the data for +`bar()`, and then record the dependency. + +[`&hir::Item`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Item.html @@ -67,63 +73,86 @@ carry around references into the HIR, but rather to carry around *identifier numbers* (or just "ids"). Right now, you will find four sorts of identifiers in active use: -- `DefId`, which primarily names "definitions" or top-level items. - - You can think of a `DefId` as being shorthand for a very explicit +- [`DefId`], which primarily names "definitions" or top-level items. + - You can think of a [`DefId`] as being shorthand for a very explicit and complete path, like `std::collections::HashMap`. However, these paths are able to name things that are not nameable in normal Rust (e.g. impls), and they also include extra information about the crate (such as its version number, as two versions of the same crate can co-exist). - - A `DefId` really consists of two parts, a `CrateNum` (which + - A [`DefId`] really consists of two parts, a `CrateNum` (which identifies the crate) and a `DefIndex` (which indixes into a list of items that is maintained per crate). -- `HirId`, which combines the index of a particular item with an +- [`HirId`], which combines the index of a particular item with an offset within that item. - - the key point of a `HirId` is that it is *relative* to some item - (which is named via a `DefId`). -- `BodyId`, this is an absolute identifier that refers to a specific + - the key point of a [`HirId`] is that it is *relative* to some item + (which is named via a [`DefId`]). +- [`BodyId`], this is an absolute identifier that refers to a specific body (definition of a function or constant) in the crate. It is currently - effectively a "newtype'd" `NodeId`. -- `NodeId`, which is an absolute id that identifies a single node in the HIR + effectively a "newtype'd" [`NodeId`]. +- [`NodeId`], which is an absolute id that identifies a single node in the HIR tree. - While these are still in common use, **they are being slowly phased out**. - Since they are absolute within the crate, adding a new node anywhere in the - tree causes the `NodeId`s of all subsequent code in the crate to change. + tree causes the [`NodeId`]s of all subsequent code in the crate to change. This is terrible for incremental compilation, as you can perhaps imagine. +[`DefId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/def_id/struct.DefId.html +[`HirId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.HirId.html +[`BodyId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.BodyId.html +[`NodeId`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.NodeId.html + ### The HIR Map Most of the time when you are working with the HIR, you will do so via -the **HIR Map**, accessible in the tcx via `tcx.hir` (and defined in -the `hir::map` module). The HIR map contains a number of methods to +the **HIR Map**, accessible in the tcx via [`tcx.hir`] (and defined in +the [`hir::map`] module). The [HIR map] contains a [number of methods] to convert between IDs of various kinds and to lookup data associated with an HIR node. -For example, if you have a `DefId`, and you would like to convert it -to a `NodeId`, you can use `tcx.hir.as_local_node_id(def_id)`. This +[`tcx.hir`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/context/struct.GlobalCtxt.html#structfield.hir +[`hir::map`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/index.html +[HIR map]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/struct.Map.html +[number of methods]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/struct.Map.html#methods + +For example, if you have a [`DefId`], and you would like to convert it +to a [`NodeId`], you can use [`tcx.hir.as_local_node_id(def_id)`][as_local_node_id]. This returns an `Option` – this will be `None` if the def-id refers to something outside of the current crate (since then it has no HIR node), but otherwise returns `Some(n)` where `n` is the node-id of the definition. -Similarly, you can use `tcx.hir.find(n)` to lookup the node for a -`NodeId`. This returns a `Option>`, where `Node` is an enum +[as_local_node_id]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/struct.Map.html#method.as_local_node_id + +Similarly, you can use [`tcx.hir.find(n)`][find] to lookup the node for a +[`NodeId`]. This returns a `Option>`, where [`Node`] is an enum defined in the map; by matching on this you can find out what sort of node the node-id referred to and also get a pointer to the data itself. Often, you know what sort of node `n` is – e.g. if you know that `n` must be some HIR expression, you can do -`tcx.hir.expect_expr(n)`, which will extract and return the -`&hir::Expr`, panicking if `n` is not in fact an expression. +[`tcx.hir.expect_expr(n)`][expect_expr], which will extract and return the +[`&hir::Expr`][Expr], panicking if `n` is not in fact an expression. + +[find]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/struct.Map.html#method.find +[`Node`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/enum.Node.html +[expect_expr]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/struct.Map.html#method.expect_expr +[Expr]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Expr.html Finally, you can use the HIR map to find the parents of nodes, via -calls like `tcx.hir.get_parent_node(n)`. +calls like [`tcx.hir.get_parent_node(n)`][get_parent_node]. + +[get_parent_node]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/struct.Map.html#method.get_parent_node ### HIR Bodies -A **body** represents some kind of executable code, such as the body +A [`hir::Body`] represents some kind of executable code, such as the body of a function/closure or the definition of a constant. Bodies are associated with an **owner**, which is typically some kind of item (e.g. an `fn()` or `const`), but could also be a closure expression (e.g. `|x, y| x + y`). You can use the HIR map to find the body -associated with a given def-id (`maybe_body_owned_by()`) or to find -the owner of a body (`body_owner_def_id()`). +associated with a given def-id ([`maybe_body_owned_by`]) or to find +the owner of a body ([`body_owner_def_id`]). + +[`hir::Body`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Body.html +[`maybe_body_owned_by`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/struct.Map.html#method.maybe_body_owned_by +[`body_owner_def_id`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/struct.Map.html#method.body_owner_def_id From 5321d0059f97fd964d8b6aac3ce3dadab2e7f5cf Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 09:58:32 -0400 Subject: [PATCH 306/648] tweak long line --- src/hir.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/hir.md b/src/hir.md index 171319011..a5e99e8b3 100644 --- a/src/hir.md +++ b/src/hir.md @@ -116,11 +116,12 @@ with an HIR node. [number of methods]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/struct.Map.html#methods For example, if you have a [`DefId`], and you would like to convert it -to a [`NodeId`], you can use [`tcx.hir.as_local_node_id(def_id)`][as_local_node_id]. This -returns an `Option` – this will be `None` if the def-id -refers to something outside of the current crate (since then it has no -HIR node), but otherwise returns `Some(n)` where `n` is the node-id of -the definition. +to a [`NodeId`], you can use +[`tcx.hir.as_local_node_id(def_id)`][as_local_node_id]. This returns +an `Option` – this will be `None` if the def-id refers to +something outside of the current crate (since then it has no HIR +node), but otherwise returns `Some(n)` where `n` is the node-id of the +definition. [as_local_node_id]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/struct.Map.html#method.as_local_node_id From 3d094aa310cc4764a25acf4f95e9d6c4db0a5314 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 12:00:31 -0400 Subject: [PATCH 307/648] add a section about the compiler team --- src/SUMMARY.md | 1 + src/compiler-team.md | 100 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+) create mode 100644 src/compiler-team.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 981da5c1e..8de0e71ca 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -3,6 +3,7 @@ - [About this guide](./about-this-guide.md) - [How to build the compiler and run what you built](./how-to-build-and-run.md) - [Coding conventions](./conventions.md) +- [About the compiler team](./compiler-team.md) - [The compiler testing framework](./tests/intro.md) - [Running tests](./tests/running.md) - [Adding new tests](./tests/adding.md) diff --git a/src/compiler-team.md b/src/compiler-team.md new file mode 100644 index 000000000..89e95e2c3 --- /dev/null +++ b/src/compiler-team.md @@ -0,0 +1,100 @@ +# About the compiler team + +rustc is maintained by the +[Rust compiler team](https://www.rust-lang.org/en-US/team.html). The +people who belong to this team collectively work to track regressions +and implement new features. Members of the Rust compiler team are +people who have made significant contributions to rustc and its +design. + +## Rust compiler meeting + +The compiler team has a weekly meeting where we do triage and try to generally +stay on top of new bugs, regressions, and other things. This general plan for +this meeting can be found in [the rust-compiler-meeting etherpad][etherpad]. It works +roughly as follows: + +- **Review P-high bugs:** P-high bugs are those that are sufficiently important for us + to actively track progress. P-high bugs should ideally always have an assignee. +- **Look over new regressions:** we then look for new cases where the + compiler broke previously working code in the wild. Regressions are + almost always marked as P-high; the major exception would be bug + fixes (though even there we often + [aim to give warnings first][procedure]). +- **Check I-nominated issues:** These are issues where feedback from the team is desired. +- **Check for beta nominations:** These are nominations of things to backport to beta. + +The meeting currently takes place on Thursdays at 10am Boston time +(UTC-4 typically, but daylight savings time sometimes makes things +complicated). + +The meeting is held over a "chat medium" -- it used to be IRC, but we +are currently in the process of evaluating other alternatives. Check +[the etherpad] to find the current home (and see +[this internals thread][thread] for some ongoing discussion). + +[etherpad]: https://public.etherpad-mozilla.org/p/rust-compiler-meeting +[thread]: https://internals.rust-lang.org/t/where-should-the-compiler-team-and-perhaps-working-groups-chat/7894 +[procedure]: https://forge.rust-lang.org/rustc-bug-fix-procedure.html + +## Team membership + +Membership in the Rust team is typically offered when someone has been +making significant contributions to the compiler for some +time. Membership is both a recognition but also an obligation: +compiler team members are generally expected to help with upkeep as +well as doing reviews and other work. + +If you are interested in becoming a compiler team member, the first +thing to do is to start fixing some bugs, or get involved in a working +group. + +### r+ rights + +Once you have made a number of individual PRs to rustc, we will often +offer r+ privileges. This means that you have the right to instruct +"bors" (the robot that manages which PRs get landed into rustc) to +merge a PR +([here are some instructions for how to talk to bors][homu-guide]). + +[homu-guide]: https://buildbot2.rust-lang.org/homu/ + +The guidelines for reviewers are as follows: + +- You are always welcome to review any PR, regardless of who it is assigned to. + However, do not r+ PRs unless: + - You are confident in that part of the code. + - You are confident that nobody else wants to review it first. + - For example, sometimes people will express a desire to review a PR before it lands, + perhaps because it touches a particularly sensitive part of the code. +- Always be polite when reviewing: you are a representative of the Rust project, + so it is expected that you will go above and beyond when it comes to the [Code of Conduct]. + +[Code of Conduct]: https://www.rust-lang.org/en-US/conduct.html + +### high-five + +Once you have r+ rights, you can also be added to the high-five +rotation. high-five is the bot that assigns incoming PRs to +reviewers. If you are added, you will be randomly selected to review +PRs. If you find you are assigned a PR that you don't feel comfortable +reviewing, you can also leave a comment like `r? @so-and-so` to assign +to someone else -- if you don't know who to request, just write `r? +@nikomatsakis for reassignment` and @nikomatsakis will pick someone +for you. + +[hi5]: https://github.com/rust-highfive + +Getting on the high-five list is much appreciated as it lowers the +review burden for all of us! However, if you don't have time to give +people timely feedback on their PRs, it may be better that you don't +get on the list. + +### Team membership + +Full team membership is typically extended once someone made many +contributions to the Rust compiler over time, ideally (but not +necessarily) to multiple areas. Sometimes this might be implementing a +new feature, but it is also important -- perhaps more important! -- to +have time and willingness to help out with general upkeep such as +bugfixes, tracking regressions, and other less glamorous work. From 7c200dffd7adcd4c8b870e0378d00d260b1d1f35 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 12:59:29 -0400 Subject: [PATCH 308/648] adjust long lines --- src/compiler-team.md | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-) diff --git a/src/compiler-team.md b/src/compiler-team.md index 89e95e2c3..d7ee98d30 100644 --- a/src/compiler-team.md +++ b/src/compiler-team.md @@ -9,20 +9,24 @@ design. ## Rust compiler meeting -The compiler team has a weekly meeting where we do triage and try to generally -stay on top of new bugs, regressions, and other things. This general plan for -this meeting can be found in [the rust-compiler-meeting etherpad][etherpad]. It works -roughly as follows: - -- **Review P-high bugs:** P-high bugs are those that are sufficiently important for us - to actively track progress. P-high bugs should ideally always have an assignee. +The compiler team has a weekly meeting where we do triage and try to +generally stay on top of new bugs, regressions, and other things. This +general plan for this meeting can be found in +[the rust-compiler-meeting etherpad][etherpad]. It works roughly as +follows: + +- **Review P-high bugs:** P-high bugs are those that are sufficiently + important for us to actively track progress. P-high bugs should + ideally always have an assignee. - **Look over new regressions:** we then look for new cases where the compiler broke previously working code in the wild. Regressions are almost always marked as P-high; the major exception would be bug fixes (though even there we often [aim to give warnings first][procedure]). -- **Check I-nominated issues:** These are issues where feedback from the team is desired. -- **Check for beta nominations:** These are nominations of things to backport to beta. +- **Check I-nominated issues:** These are issues where feedback from + the team is desired. +- **Check for beta nominations:** These are nominations of things to + backport to beta. The meeting currently takes place on Thursdays at 10am Boston time (UTC-4 typically, but daylight savings time sometimes makes things @@ -61,14 +65,16 @@ merge a PR The guidelines for reviewers are as follows: -- You are always welcome to review any PR, regardless of who it is assigned to. - However, do not r+ PRs unless: +- You are always welcome to review any PR, regardless of who it is + assigned to. However, do not r+ PRs unless: - You are confident in that part of the code. - You are confident that nobody else wants to review it first. - - For example, sometimes people will express a desire to review a PR before it lands, - perhaps because it touches a particularly sensitive part of the code. -- Always be polite when reviewing: you are a representative of the Rust project, - so it is expected that you will go above and beyond when it comes to the [Code of Conduct]. + - For example, sometimes people will express a desire to review a + PR before it lands, perhaps because it touches a particularly + sensitive part of the code. +- Always be polite when reviewing: you are a representative of the + Rust project, so it is expected that you will go above and beyond + when it comes to the [Code of Conduct]. [Code of Conduct]: https://www.rust-lang.org/en-US/conduct.html From 207e669c1f7e45a8cc2fd761567d9b6f24e49d3a Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 14:46:03 -0400 Subject: [PATCH 309/648] fix various minor points --- src/SUMMARY.md | 2 +- src/compiler-team.md | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 8de0e71ca..1cb9ea599 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -1,9 +1,9 @@ # Summary - [About this guide](./about-this-guide.md) +- [About the compiler team](./compiler-team.md) - [How to build the compiler and run what you built](./how-to-build-and-run.md) - [Coding conventions](./conventions.md) -- [About the compiler team](./compiler-team.md) - [The compiler testing framework](./tests/intro.md) - [Running tests](./tests/running.md) - [Adding new tests](./tests/adding.md) diff --git a/src/compiler-team.md b/src/compiler-team.md index d7ee98d30..ad8ac292f 100644 --- a/src/compiler-team.md +++ b/src/compiler-team.md @@ -32,9 +32,9 @@ The meeting currently takes place on Thursdays at 10am Boston time (UTC-4 typically, but daylight savings time sometimes makes things complicated). -The meeting is held over a "chat medium" -- it used to be IRC, but we +The meeting is held over a "chat medium" — it used to be IRC, but we are currently in the process of evaluating other alternatives. Check -[the etherpad] to find the current home (and see +the [etherpad] to find the current home (and see [this internals thread][thread] for some ongoing discussion). [etherpad]: https://public.etherpad-mozilla.org/p/rust-compiler-meeting @@ -85,7 +85,7 @@ rotation. high-five is the bot that assigns incoming PRs to reviewers. If you are added, you will be randomly selected to review PRs. If you find you are assigned a PR that you don't feel comfortable reviewing, you can also leave a comment like `r? @so-and-so` to assign -to someone else -- if you don't know who to request, just write `r? +to someone else — if you don't know who to request, just write `r? @nikomatsakis for reassignment` and @nikomatsakis will pick someone for you. @@ -96,11 +96,11 @@ review burden for all of us! However, if you don't have time to give people timely feedback on their PRs, it may be better that you don't get on the list. -### Team membership +### Full team membership Full team membership is typically extended once someone made many contributions to the Rust compiler over time, ideally (but not necessarily) to multiple areas. Sometimes this might be implementing a -new feature, but it is also important -- perhaps more important! -- to +new feature, but it is also important — perhaps more important! — to have time and willingness to help out with general upkeep such as bugfixes, tracking regressions, and other less glamorous work. From e295ad412e02c6770bbacdd153fef379f1d12efc Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 14:48:16 -0400 Subject: [PATCH 310/648] includ some notes on finding bugs --- src/compiler-team.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/compiler-team.md b/src/compiler-team.md index ad8ac292f..45883e9d1 100644 --- a/src/compiler-team.md +++ b/src/compiler-team.md @@ -51,7 +51,10 @@ well as doing reviews and other work. If you are interested in becoming a compiler team member, the first thing to do is to start fixing some bugs, or get involved in a working -group. +group. One good way to find bugs is to look for +[open issues tagged with E-easy](https://github.com/rust-lang/rust/issues?q=is%3Aopen+is%3Aissue+label%3AE-easy) +or +[E-mentor](https://github.com/rust-lang/rust/issues?q=is%3Aopen+is%3Aissue+label%3AE-mentor). ### r+ rights From 81fd1c02f2e7db7570b87e4dff2270f25339ba7b Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 14:50:29 -0400 Subject: [PATCH 311/648] note about discussion channels --- src/compiler-team.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/compiler-team.md b/src/compiler-team.md index 45883e9d1..a6328198c 100644 --- a/src/compiler-team.md +++ b/src/compiler-team.md @@ -7,6 +7,16 @@ and implement new features. Members of the Rust compiler team are people who have made significant contributions to rustc and its design. +## Discussion + +Currently the compiler team chats in a number of places. There is an +ongoing [thread] on the internals board about trying to find a permanent +home. In any case, you can find people in one of three places at the moment: + +- The `#rustc` channel on mozilla's IRC (`irc.mozilla.org`) +- The `t-compiler` stream on [the Zulip instance](https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler) +- The `compiler` channel on the [rust-lang discord](https://discord.gg/rust-lang) + ## Rust compiler meeting The compiler team has a weekly meeting where we do triage and try to From e731de57556dadd62b3148aaad5d83652a86eff7 Mon Sep 17 00:00:00 2001 From: Dale Wijnand <344610+dwijnand@users.noreply.github.com> Date: Fri, 7 Sep 2018 11:46:53 +0100 Subject: [PATCH 312/648] No copyright notices on new tests --- src/tests/adding.md | 1 - 1 file changed, 1 deletion(-) diff --git a/src/tests/adding.md b/src/tests/adding.md index ee264747a..dca224fff 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -9,7 +9,6 @@ To add a new test, the first thing you generally do is to create a file, typically a Rust source file. Test files have a particular structure: -- They always begin with the [copyright notice](./conventions.html#copyright); - then they should have some kind of [comment explaining what the test is about](#explanatory_comment); - next, they can have one or more [header commands](#header_commands), which From ad03ec4ee8e1f1579018f5ed879e06d881b2ea31 Mon Sep 17 00:00:00 2001 From: Dale Wijnand <344610+dwijnand@users.noreply.github.com> Date: Fri, 7 Sep 2018 11:47:56 +0100 Subject: [PATCH 313/648] Write in English --- src/tests/adding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/tests/adding.md b/src/tests/adding.md index dca224fff..b9f05b9dd 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -9,7 +9,7 @@ To add a new test, the first thing you generally do is to create a file, typically a Rust source file. Test files have a particular structure: -- then they should have some kind of +- They should have some kind of [comment explaining what the test is about](#explanatory_comment); - next, they can have one or more [header commands](#header_commands), which are special comments that the test interpreter knows how to interpret. From 44c05c72be6f4a5f6bf6295fc7f6ece9566c7f2e Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 12:57:51 -0400 Subject: [PATCH 314/648] add a section about profiling with perf --- src/SUMMARY.md | 4 +- src/profiling.md | 9 ++ src/profiling/with_perf.md | 294 +++++++++++++++++++++++++++++++++++++ 3 files changed, 306 insertions(+), 1 deletion(-) create mode 100644 src/profiling.md create mode 100644 src/profiling/with_perf.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 1cb9ea599..ad8a82623 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -4,13 +4,15 @@ - [About the compiler team](./compiler-team.md) - [How to build the compiler and run what you built](./how-to-build-and-run.md) - [Coding conventions](./conventions.md) +- [Walkthrough: a typical contribution](./walkthrough.md) - [The compiler testing framework](./tests/intro.md) - [Running tests](./tests/running.md) - [Adding new tests](./tests/adding.md) - [Using `compiletest` + commands to control test execution](./compiletest.md) - [Debugging the Compiler](./compiler-debugging.md) -- [Walkthrough: a typical contribution](./walkthrough.md) +- [Profiling the compiler](./profiling.md) + - [with the linux perf tool](./profiling/with_perf.md) - [High-level overview of the compiler source](./high-level-overview.md) - [The Rustc Driver](./rustc-driver.md) - [Rustdoc](./rustdoc.md) diff --git a/src/profiling.md b/src/profiling.md new file mode 100644 index 000000000..f6771b07b --- /dev/null +++ b/src/profiling.md @@ -0,0 +1,9 @@ +# Profiling the compiler + +This discussion talks about how profile the compiler and find out +where it spends its time. If you just want to get a general overview, +it is often a good idea to just add `-Zself-profile` option to the +rustc command line. This will break down time spent into various +categories. But if you want a more detailed look, you probably want +to break out a custom profiler. + diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md new file mode 100644 index 000000000..dc767c9a5 --- /dev/null +++ b/src/profiling/with_perf.md @@ -0,0 +1,294 @@ +# Profiling with perf + +sThis is a guide for how to profile rustc with perf. + +## Initial steps + +- Get a clean checkout of rust-lang/master, or whatever it is you want to profile. +- Set the following settings in your `config.toml`: + - `debuginfo-lines = true` + - `use-jemalloc = false` -- lets you do memory use profiling with valgrind + - leave everything else the defaults +- Run `./x.py build` to get a full build +- Make a rustup toolchain (let's call it `rust-prof`) pointing to that result + - `rustup toolchain link` XXX + +## Gathering a perf profile + +perf is an excellent tool on linux that can be used to gather and +analyze all kinds of information. Mostly it is used to figure out +where a program spends its time. It can also be used for other sorts +of events, though, like cache misses and so forth. + +### The basics + +The basic `perf` command is this: + +``` +perf record -F99 --call-graph dwarf XXX +``` + +The `-F99` tells perf to sample at 99 Hz, which avoids generating too +much data for longer runs. The `--call-graph dwarf` tells perf to get +call-graph information from debuginfo, which is accurate. The `XXX` is +the command you want to profile. So, for example, you might do: + +``` +perf record -F99 --call-graph dwarf cargo +rust-prof rustc +``` + +to run `cargo`. But there are some things to be aware of: + +- You probably don't want to profile the time spend building + dependencies. So something like `cargo build; cargo clean -p $C` may + be helpful (where `$C` is the crate name) +- You probably don't want incremental messing about with your + profile. So something like `CARGO_INCREMENTAL=0` can be helpful. + +### Gathering a perf profile from a `perf.rust-lang.org` test + +Often we want to analyze a specific test from `perf.rust-lang.org`. To +do that, the first step is to clone +[the rustc-perf repository][rustc-perf-gh]: + +```bash +> git clone https://github.com/rust-lang-nursery/rustc-perf +``` + +[rustc-perf-gh]: https://github.com/rust-lang-nursery/rustc-perf + +This repo contains a bunch of stuff, but the sources for the tests are +found in [the `collector/benchmarks` directory][dir]. So let's go into +the directory of a specific test; we'll use `clap-rs` as an example: + +[dir]: https://github.com/rust-lang-nursery/rustc-perf/tree/master/collector/benchmarks + +```bash +cd collector/benchmarks/clap-rs +``` + +In this case, let's say we want to profile the `cargo check` +performance. In that case, I would first run some basic commands to +build the dependencies: + +```bash +# Setup: first clean out any old results and build the dependencies: +cargo +rust-prof clean +CARGO_INCREMENTAL=0 cargo +rust-prof check +``` + +Next: we want record the execution time for *just* the clap-rs crate, +running cargo check. I tend to use `cargo rustc` for this, since it +also allows me to add explicit flags, which we'll do later on. + +```bash +touch src/lib.rs +CARGO_INCREMENTAL=0 perf record -F99 --call-graph dwarf cargo rustc --profile check --lib +``` + +Note that final command: it's a doozy! It uses the `cargo rustc` +command, which executes rustc with (potentially) additional options; +the `--profile check` and `--lib` options specify that we are doing a +`cargo check` execution, and that this is a library (not an +execution). + +At this point, we can use `perf` tooling to analyze the results. For example: + +```bash +> perf report +``` + +will open up an interactive TUI program. In simple cases, that can be +helpful. For more detailed examination, the [`perf-focus` tool][pf] +can be helpful; it is covered below. + +**A note of caution.** Each of the rustc-perf tests is its own special + snowflake. In particular, some of them are not libraries, in which + case you would want to do `touch src/main.rs` and avoid passing + `--lib`. I'm not sure how best to tell which test is which to be + honest. + +### Gathering NLL data + +If you want to profile an NLL run, you can just pass extra options to the `cargo rustc` command. The actual perf site just uses `-Zborrowck=mir`, which we can simulate like so: + +```bash +touch src/lib.rs +CARGO_INCREMENTAL=0 perf record -F99 --call-graph dwarf cargo rustc --profile check --lib -- -Zborrowck=mir +``` + +[pf]: https://github.com/nikomatsakis/perf-focus + +## Analyzing a perf profile with `perf focus` + +Once you've gathered a perf profile, we want to get some information +about it. For this, I personally use [perf focus][pf]. It's a kind of +simple but useful tool that lets you answer queries like: + +- "how much time was spent in function F" (no matter where it was called from) +- "how much time was spent in function F when it was called from G" +- "how much time was spent in function F *excluding* time spent in G" +- "what fns does F call and how much time does it spend in them" + +To understand how it works, you have to know just a bit about +perf. Basically, perf works by *sampling* your process on a regular +basis (or whenever some event occurs). For each sample, perf gathers a +backtrace. `perf focus` lets you write a regular expression that tests +which fns appear in that backtrace, and then tells you which +percentage of samples had a backtrace that met the regular +expression. It's probably easiest to explain by walking through how I +would analyze NLL performance. + +## Installing `perf-focus` + +You can install perf-focus using `cargo install`: + +``` +cargo install perf-focus +``` + +## Example: How much time is spent in MIR borrowck? + +Let's say we've gathered the NLL data for a test. We'd like to know +how much time it is spending in the MIR borrow-checker. The "main" +function of the MIR borrowck is called `do_mir_borrowck`, so we can do +this command: + +```bash +> perf focus '{do_mir_borrowck}' +Matcher : {do_mir_borrowck} +Matches : 228 +Not Matches: 542 +Percentage : 29% +``` + +The `'{do_mir_borrowck}'` argument is called the **matcher**. It +specifies the test to be applied on the backtrace. In this case, the +`{X}` indicates that there must be *some* function on the backtrace +that meets the regular expression `X`. In this case, that regex is +just the name of the fn we want (in fact, it's a subset of the name; +the full name includes a bunch of other stuff, like the module +path). In this mode, perf-focus just prints out the percentage of +samples where `do_mir_borrowck` was on the stack: in this case, 29%. + +**A note about c++filt.** To get the data from `perf`, `perf focus` + currently executes `perf script` (perhaps there is a better + way...). I've sometimes found that `perf script` outputs C++ mangled + names. This is annoying. You can tell by running `perf script | + head` yourself -- if you see named like `5rustc6middle` instead of + `rustc::middle`, then you have the same problem. You can solve this + by doing: + +```bash +> perf script | c++filt | perf focus --from-stdin ... +``` + +This will pipe the output from `perf script` through `c++filt` and +should mostly convert those names into a more friendly format. The +`--from-stdin` flag to `perf focus` tells it to get its data from +stdin, rather than executing `perf focus`. We should make this more +convenient (at worst, maybe add a `c++filt` option to `perf focus`, or +just always use it -- it's pretty harmless). + +## Example: How much time does MIR borrowck spend solving traits? + +Perhaps we'd like to know how much time MIR borrowck spends in the +trait checker. We can ask this using a more complex regex: + +```bash +> perf focus '{do_mir_borrowck}..{^rustc::traits}' +Matcher : {do_mir_borrowck},..{^rustc::traits} +Matches : 12 +Not Matches: 1311 +Percentage : 0% +``` + +Here we used the `..` operator to ask "how often do we have +`do_mir_borrowck` on the stack and then, later, some fn whose name +begins with `rusc::traits`?" (basically, code in that module). It +turns out the answer is "almost never" -- only 12 samples fit that +description (if you ever see *no* samples, that often indicates your +query is messed up). + +If you're curious, you can find out exactly which samples by using the +`--print-match` option. This will print out the full backtrace for +each sample. The `|` at the front of the line indicates the part that +the regular expression matched. + +## Example: Where does MIR borrowck spend its time? + +Often we want to do a more "explorational" queries. Like, we know that +MIR borrowck is 29% of the time, but where does that time get spent? +For that, the `--tree-callees` option is often the best tool. You +usually also want to give `--tree-min-percent` or +`--tree-max-depth`. The result looks like this: + +```bash +> perf focus '{do_mir_borrowck}' --tree-callees --tree-min-percent 3 +Matcher : {do_mir_borrowck} +Matches : 577 +Not Matches: 746 +Percentage : 43% + +Tree +| matched `{do_mir_borrowck}` (43% total, 0% self) +: | rustc_mir::borrow_check::nll::compute_regions (20% total, 0% self) +: : | rustc_mir::borrow_check::nll::type_check::type_check_internal (13% total, 0% self) +: : : | core::ops::function::FnOnce::call_once (5% total, 0% self) +: : : : | rustc_mir::borrow_check::nll::type_check::liveness::generate (5% total, 3% self) +: : : | as rustc::mir::visit::Visitor<'tcx>>::visit_mir (3% total, 0% self) +: | rustc::mir::visit::Visitor::visit_mir (8% total, 6% self) +: | as rustc_mir::dataflow::DataflowResultsConsumer<'cx, 'tcx>>::visit_statement_entry (5% total, 0% self) +: | rustc_mir::dataflow::do_dataflow (3% total, 0% self) +``` + +What happens with `--tree-callees` is that + +- we find each sample matching the regular expression +- we look at the code that is occurs *after* the regex match and try to build up a call tree + +The `--tree-min-percent 3` option says "only show me things that take +more than 3% of the time. Without this, the tree often gets really +noisy and includes random stuff like the innards of +malloc. `--tree-max-depth` can be useful too, it just limits how many +levels we print. + +For each line, we display the percent of time in that function +altogether ("total") and the percent of time spent in **just that +function and not some callee of that function** (self). Usually +"total" is the more interesting number, but not always. + +### Absolute vs relative percentages + +By default, all in perf-focus are relative to the **total program +execution**. This is useful to help you keep perspective -- often as +we drill down to find hot spots, we can lose sight of the fact that, +in terms of overall program execution, this "hot spot" is actually not +important. It also ensures that percentages between different queries +are easily compared against one another. + +That said, sometimes it's useful to get relative percentages, so `perf +focus` offers a `--relative` option. In this case, the percentages are +listed only for samples that match (vs all samples). So for example we +could find out get our percentages relative to the borrowck itself +like so: + +```bash +> perf focus '{do_mir_borrowck}' --tree-callees --relative --tree-max-depth 1 --tree-min-percent 5 +Matcher : {do_mir_borrowck} +Matches : 577 +Not Matches: 746 +Percentage : 100% + +Tree +| matched `{do_mir_borrowck}` (100% total, 0% self) +: | rustc_mir::borrow_check::nll::compute_regions (47% total, 0% self) [...] +: | rustc::mir::visit::Visitor::visit_mir (19% total, 15% self) [...] +: | as rustc_mir::dataflow::DataflowResultsConsumer<'cx, 'tcx>>::visit_statement_entry (13% total, 0% self) [...] +: | rustc_mir::dataflow::do_dataflow (8% total, 1% self) [...] +``` + +Here you see that `compute_regions` came up as "47% total" -- that +means that 47% of `do_mir_borrowck` is spent in that function. Before, +we saw 20% -- that's because `do_mir_borrowck` itself is only 43% of +the total time (and `.47 * .43 = .20`). From 5571290aac90206fbcec777a02cf2c273f6024f7 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 13:46:13 -0400 Subject: [PATCH 315/648] various nits from mark-i-m --- src/profiling/with_perf.md | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index dc767c9a5..94ce6f92f 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -1,6 +1,6 @@ # Profiling with perf -sThis is a guide for how to profile rustc with perf. +This is a guide for how to profile rustc with [perf](https://perf.wiki.kernel.org/index.php/Main_Page). ## Initial steps @@ -11,7 +11,7 @@ sThis is a guide for how to profile rustc with perf. - leave everything else the defaults - Run `./x.py build` to get a full build - Make a rustup toolchain (let's call it `rust-prof`) pointing to that result - - `rustup toolchain link` XXX + - `rustup toolchain link ` ## Gathering a perf profile @@ -29,9 +29,11 @@ perf record -F99 --call-graph dwarf XXX ``` The `-F99` tells perf to sample at 99 Hz, which avoids generating too -much data for longer runs. The `--call-graph dwarf` tells perf to get -call-graph information from debuginfo, which is accurate. The `XXX` is -the command you want to profile. So, for example, you might do: +much data for longer runs (why 99 Hz you ask? No particular reason, it +just seems to work well for me). The `--call-graph dwarf` tells perf +to get call-graph information from debuginfo, which is accurate. The +`XXX` is the command you want to profile. So, for example, you might +do: ``` perf record -F99 --call-graph dwarf cargo +rust-prof rustc @@ -42,6 +44,7 @@ to run `cargo`. But there are some things to be aware of: - You probably don't want to profile the time spend building dependencies. So something like `cargo build; cargo clean -p $C` may be helpful (where `$C` is the crate name) + - Though usually I just do `touch src/lib.rs` and rebuild instead. =) - You probably don't want incremental messing about with your profile. So something like `CARGO_INCREMENTAL=0` can be helpful. @@ -89,8 +92,7 @@ CARGO_INCREMENTAL=0 perf record -F99 --call-graph dwarf cargo rustc --profile ch Note that final command: it's a doozy! It uses the `cargo rustc` command, which executes rustc with (potentially) additional options; the `--profile check` and `--lib` options specify that we are doing a -`cargo check` execution, and that this is a library (not an -execution). +`cargo check` execution, and that this is a library (not a binary). At this point, we can use `perf` tooling to analyze the results. For example: @@ -110,7 +112,8 @@ can be helpful; it is covered below. ### Gathering NLL data -If you want to profile an NLL run, you can just pass extra options to the `cargo rustc` command. The actual perf site just uses `-Zborrowck=mir`, which we can simulate like so: +If you want to profile an NLL run, you can just pass extra options to +the `cargo rustc` command, like so: ```bash touch src/lib.rs @@ -128,18 +131,18 @@ simple but useful tool that lets you answer queries like: - "how much time was spent in function F" (no matter where it was called from) - "how much time was spent in function F when it was called from G" - "how much time was spent in function F *excluding* time spent in G" -- "what fns does F call and how much time does it spend in them" +- "what functions does F call and how much time does it spend in them" To understand how it works, you have to know just a bit about perf. Basically, perf works by *sampling* your process on a regular basis (or whenever some event occurs). For each sample, perf gathers a backtrace. `perf focus` lets you write a regular expression that tests -which fns appear in that backtrace, and then tells you which +which functions appear in that backtrace, and then tells you which percentage of samples had a backtrace that met the regular expression. It's probably easiest to explain by walking through how I would analyze NLL performance. -## Installing `perf-focus` +### Installing `perf-focus` You can install perf-focus using `cargo install`: @@ -147,7 +150,7 @@ You can install perf-focus using `cargo install`: cargo install perf-focus ``` -## Example: How much time is spent in MIR borrowck? +### Example: How much time is spent in MIR borrowck? Let's say we've gathered the NLL data for a test. We'd like to know how much time it is spending in the MIR borrow-checker. The "main" @@ -175,7 +178,7 @@ samples where `do_mir_borrowck` was on the stack: in this case, 29%. currently executes `perf script` (perhaps there is a better way...). I've sometimes found that `perf script` outputs C++ mangled names. This is annoying. You can tell by running `perf script | - head` yourself -- if you see named like `5rustc6middle` instead of + head` yourself -- if you see names like `5rustc6middle` instead of `rustc::middle`, then you have the same problem. You can solve this by doing: @@ -190,7 +193,7 @@ stdin, rather than executing `perf focus`. We should make this more convenient (at worst, maybe add a `c++filt` option to `perf focus`, or just always use it -- it's pretty harmless). -## Example: How much time does MIR borrowck spend solving traits? +### Example: How much time does MIR borrowck spend solving traits? Perhaps we'd like to know how much time MIR borrowck spends in the trait checker. We can ask this using a more complex regex: @@ -215,7 +218,7 @@ If you're curious, you can find out exactly which samples by using the each sample. The `|` at the front of the line indicates the part that the regular expression matched. -## Example: Where does MIR borrowck spend its time? +### Example: Where does MIR borrowck spend its time? Often we want to do a more "explorational" queries. Like, we know that MIR borrowck is 29% of the time, but where does that time get spent? @@ -258,7 +261,7 @@ altogether ("total") and the percent of time spent in **just that function and not some callee of that function** (self). Usually "total" is the more interesting number, but not always. -### Absolute vs relative percentages +### Relative percentages By default, all in perf-focus are relative to the **total program execution**. This is useful to help you keep perspective -- often as @@ -270,8 +273,7 @@ are easily compared against one another. That said, sometimes it's useful to get relative percentages, so `perf focus` offers a `--relative` option. In this case, the percentages are listed only for samples that match (vs all samples). So for example we -could find out get our percentages relative to the borrowck itself -like so: +could get our percentages relative to the borrowck itself like so: ```bash > perf focus '{do_mir_borrowck}' --tree-callees --relative --tree-max-depth 1 --tree-min-percent 5 From f1e087db37d38534ee751f3e9fe19b46c50526ee Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 13:49:29 -0400 Subject: [PATCH 316/648] switch to em-dash --- src/profiling/with_perf.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index 94ce6f92f..9f0554e5e 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -7,7 +7,7 @@ This is a guide for how to profile rustc with [perf](https://perf.wiki.kernel.or - Get a clean checkout of rust-lang/master, or whatever it is you want to profile. - Set the following settings in your `config.toml`: - `debuginfo-lines = true` - - `use-jemalloc = false` -- lets you do memory use profiling with valgrind + - `use-jemalloc = false` — lets you do memory use profiling with valgrind - leave everything else the defaults - Run `./x.py build` to get a full build - Make a rustup toolchain (let's call it `rust-prof`) pointing to that result @@ -117,7 +117,7 @@ the `cargo rustc` command, like so: ```bash touch src/lib.rs -CARGO_INCREMENTAL=0 perf record -F99 --call-graph dwarf cargo rustc --profile check --lib -- -Zborrowck=mir +CARGO_INCREMENTAL=0 perf record -F99 --call-graph dwarf cargo rustc --profile check --lib — -Zborrowck=mir ``` [pf]: https://github.com/nikomatsakis/perf-focus @@ -178,7 +178,7 @@ samples where `do_mir_borrowck` was on the stack: in this case, 29%. currently executes `perf script` (perhaps there is a better way...). I've sometimes found that `perf script` outputs C++ mangled names. This is annoying. You can tell by running `perf script | - head` yourself -- if you see names like `5rustc6middle` instead of + head` yourself — if you see names like `5rustc6middle` instead of `rustc::middle`, then you have the same problem. You can solve this by doing: @@ -191,7 +191,7 @@ should mostly convert those names into a more friendly format. The `--from-stdin` flag to `perf focus` tells it to get its data from stdin, rather than executing `perf focus`. We should make this more convenient (at worst, maybe add a `c++filt` option to `perf focus`, or -just always use it -- it's pretty harmless). +just always use it — it's pretty harmless). ### Example: How much time does MIR borrowck spend solving traits? @@ -209,7 +209,7 @@ Percentage : 0% Here we used the `..` operator to ask "how often do we have `do_mir_borrowck` on the stack and then, later, some fn whose name begins with `rusc::traits`?" (basically, code in that module). It -turns out the answer is "almost never" -- only 12 samples fit that +turns out the answer is "almost never" — only 12 samples fit that description (if you ever see *no* samples, that often indicates your query is messed up). @@ -264,7 +264,7 @@ function and not some callee of that function** (self). Usually ### Relative percentages By default, all in perf-focus are relative to the **total program -execution**. This is useful to help you keep perspective -- often as +execution**. This is useful to help you keep perspective — often as we drill down to find hot spots, we can lose sight of the fact that, in terms of overall program execution, this "hot spot" is actually not important. It also ensures that percentages between different queries @@ -273,7 +273,8 @@ are easily compared against one another. That said, sometimes it's useful to get relative percentages, so `perf focus` offers a `--relative` option. In this case, the percentages are listed only for samples that match (vs all samples). So for example we -could get our percentages relative to the borrowck itself like so: +could get our percentages relative to the borrowck itself +like so: ```bash > perf focus '{do_mir_borrowck}' --tree-callees --relative --tree-max-depth 1 --tree-min-percent 5 @@ -290,7 +291,7 @@ Tree : | rustc_mir::dataflow::do_dataflow (8% total, 1% self) [...] ``` -Here you see that `compute_regions` came up as "47% total" -- that +Here you see that `compute_regions` came up as "47% total" — that means that 47% of `do_mir_borrowck` is spent in that function. Before, -we saw 20% -- that's because `do_mir_borrowck` itself is only 43% of +we saw 20% — that's because `do_mir_borrowck` itself is only 43% of the total time (and `.47 * .43 = .20`). From 4e5ab2ac25c023dbc38c748fd953fb11a2c0b4d1 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 13:50:06 -0400 Subject: [PATCH 317/648] fix long lines --- src/profiling/with_perf.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index 9f0554e5e..85dc8efd6 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -4,7 +4,8 @@ This is a guide for how to profile rustc with [perf](https://perf.wiki.kernel.or ## Initial steps -- Get a clean checkout of rust-lang/master, or whatever it is you want to profile. +- Get a clean checkout of rust-lang/master, or whatever it is you want + to profile. - Set the following settings in your `config.toml`: - `debuginfo-lines = true` - `use-jemalloc = false` — lets you do memory use profiling with valgrind @@ -248,7 +249,8 @@ Tree What happens with `--tree-callees` is that - we find each sample matching the regular expression -- we look at the code that is occurs *after* the regex match and try to build up a call tree +- we look at the code that is occurs *after* the regex match and try + to build up a call tree The `--tree-min-percent 3` option says "only show me things that take more than 3% of the time. Without this, the tree often gets really From fb59bf5383e5c5f4e596f50f6056d4e9b08a1201 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 10:03:56 -0400 Subject: [PATCH 318/648] be consistent about bash snippets --- src/profiling/with_perf.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index 85dc8efd6..61720c3b4 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -68,7 +68,7 @@ the directory of a specific test; we'll use `clap-rs` as an example: [dir]: https://github.com/rust-lang-nursery/rustc-perf/tree/master/collector/benchmarks ```bash -cd collector/benchmarks/clap-rs +> cd collector/benchmarks/clap-rs ``` In this case, let's say we want to profile the `cargo check` @@ -77,8 +77,8 @@ build the dependencies: ```bash # Setup: first clean out any old results and build the dependencies: -cargo +rust-prof clean -CARGO_INCREMENTAL=0 cargo +rust-prof check +> cargo +rust-prof clean +> CARGO_INCREMENTAL=0 cargo +rust-prof check ``` Next: we want record the execution time for *just* the clap-rs crate, @@ -86,8 +86,8 @@ running cargo check. I tend to use `cargo rustc` for this, since it also allows me to add explicit flags, which we'll do later on. ```bash -touch src/lib.rs -CARGO_INCREMENTAL=0 perf record -F99 --call-graph dwarf cargo rustc --profile check --lib +> touch src/lib.rs +> CARGO_INCREMENTAL=0 perf record -F99 --call-graph dwarf cargo rustc --profile check --lib ``` Note that final command: it's a doozy! It uses the `cargo rustc` @@ -117,8 +117,8 @@ If you want to profile an NLL run, you can just pass extra options to the `cargo rustc` command, like so: ```bash -touch src/lib.rs -CARGO_INCREMENTAL=0 perf record -F99 --call-graph dwarf cargo rustc --profile check --lib — -Zborrowck=mir +> touch src/lib.rs +> CARGO_INCREMENTAL=0 perf record -F99 --call-graph dwarf cargo rustc --profile check --lib — -Zborrowck=mir ``` [pf]: https://github.com/nikomatsakis/perf-focus From cdc688451e4196ee426f939e69a27828a4bad645 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 10:55:46 -0400 Subject: [PATCH 319/648] say function, not fn --- src/profiling/with_perf.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index 61720c3b4..f0f6aada9 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -170,7 +170,7 @@ The `'{do_mir_borrowck}'` argument is called the **matcher**. It specifies the test to be applied on the backtrace. In this case, the `{X}` indicates that there must be *some* function on the backtrace that meets the regular expression `X`. In this case, that regex is -just the name of the fn we want (in fact, it's a subset of the name; +just the name of the function we want (in fact, it's a subset of the name; the full name includes a bunch of other stuff, like the module path). In this mode, perf-focus just prints out the percentage of samples where `do_mir_borrowck` was on the stack: in this case, 29%. @@ -208,8 +208,8 @@ Percentage : 0% ``` Here we used the `..` operator to ask "how often do we have -`do_mir_borrowck` on the stack and then, later, some fn whose name -begins with `rusc::traits`?" (basically, code in that module). It +`do_mir_borrowck` on the stack and then, later, some function whose +name begins with `rusc::traits`?" (basically, code in that module). It turns out the answer is "almost never" — only 12 samples fit that description (if you ever see *no* samples, that often indicates your query is messed up). From 37aa15e93fab0c65e3d37282e838dbc93072d8d9 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 10:59:05 -0400 Subject: [PATCH 320/648] link to build-and-run, fix em-dash --- src/how-to-build-and-run.md | 8 ++++++++ src/profiling/with_perf.md | 4 ++-- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 557fc9148..3ad9a52ae 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -110,6 +110,8 @@ This is just a subset of the full rustc build. The **full** rustc build - Build libstd with stage2 compiler. - Build librustdoc and a bunch of other things. + + ### Creating a rustup toolchain Once you have successfully built rustc, you will have created a bunch @@ -125,6 +127,12 @@ to run the entire test suite). > rustup toolchain link stage2 build//stage2 ``` +The `` would typically be one of the following: + +- Linux: `x86_64-unknown-linux-gnu` +- Mac: `x86_64-apple-darwin` +- Windows: `x86_64-pc-windows-msvc` + Now you can run the rustc you built with. If you run with `-vV`, you should see a version number ending in `-dev`, indicating a build from your local environment: diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index f0f6aada9..740b0a263 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -12,7 +12,7 @@ This is a guide for how to profile rustc with [perf](https://perf.wiki.kernel.or - leave everything else the defaults - Run `./x.py build` to get a full build - Make a rustup toolchain (let's call it `rust-prof`) pointing to that result - - `rustup toolchain link ` + - see [the "build and run" section for instructions](../how-to-build-and-run.html#toolchain) ## Gathering a perf profile @@ -118,7 +118,7 @@ the `cargo rustc` command, like so: ```bash > touch src/lib.rs -> CARGO_INCREMENTAL=0 perf record -F99 --call-graph dwarf cargo rustc --profile check --lib — -Zborrowck=mir +> CARGO_INCREMENTAL=0 perf record -F99 --call-graph dwarf cargo rustc --profile check --lib -- -Zborrowck=mir ``` [pf]: https://github.com/nikomatsakis/perf-focus From 9a463df8332c8b4f4afa269d1d16edd63c0ec3ce Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 11:00:09 -0400 Subject: [PATCH 321/648] don't write `rust-prof` bur rather ` --- src/profiling/with_perf.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index 740b0a263..683b61d2e 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -11,7 +11,7 @@ This is a guide for how to profile rustc with [perf](https://perf.wiki.kernel.or - `use-jemalloc = false` — lets you do memory use profiling with valgrind - leave everything else the defaults - Run `./x.py build` to get a full build -- Make a rustup toolchain (let's call it `rust-prof`) pointing to that result +- Make a rustup toolchain pointing to that result - see [the "build and run" section for instructions](../how-to-build-and-run.html#toolchain) ## Gathering a perf profile @@ -37,10 +37,11 @@ to get call-graph information from debuginfo, which is accurate. The do: ``` -perf record -F99 --call-graph dwarf cargo +rust-prof rustc +perf record -F99 --call-graph dwarf cargo + rustc ``` -to run `cargo`. But there are some things to be aware of: +to run `cargo` -- here `` should be the name of the toolchain +you made in the beginning. But there are some things to be aware of: - You probably don't want to profile the time spend building dependencies. So something like `cargo build; cargo clean -p $C` may @@ -77,10 +78,13 @@ build the dependencies: ```bash # Setup: first clean out any old results and build the dependencies: -> cargo +rust-prof clean -> CARGO_INCREMENTAL=0 cargo +rust-prof check +> cargo + clean +> CARGO_INCREMENTAL=0 cargo + check ``` +(Again, `` should be replaced with the name of the +toolchain we made in the first step.) + Next: we want record the execution time for *just* the clap-rs crate, running cargo check. I tend to use `cargo rustc` for this, since it also allows me to add explicit flags, which we'll do later on. From 85ed2e32de2151adae306651757c544ca41482b6 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 11:01:29 -0400 Subject: [PATCH 322/648] update with njn's insight --- src/profiling/with_perf.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index 683b61d2e..cec17d344 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -30,11 +30,11 @@ perf record -F99 --call-graph dwarf XXX ``` The `-F99` tells perf to sample at 99 Hz, which avoids generating too -much data for longer runs (why 99 Hz you ask? No particular reason, it -just seems to work well for me). The `--call-graph dwarf` tells perf -to get call-graph information from debuginfo, which is accurate. The -`XXX` is the command you want to profile. So, for example, you might -do: +much data for longer runs (why 99 Hz you ask? It is often chosen +because it is unlikely to be in lockstep with other periodic +activity). The `--call-graph dwarf` tells perf to get call-graph +information from debuginfo, which is accurate. The `XXX` is the +command you want to profile. So, for example, you might do: ``` perf record -F99 --call-graph dwarf cargo + rustc From fcbd9b907e37065c8b8f7d5ded1005b29295d149 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 11:07:56 -0400 Subject: [PATCH 323/648] add a note about the collector executable --- src/profiling/with_perf.md | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index cec17d344..498dc1396 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -62,9 +62,35 @@ do that, the first step is to clone [rustc-perf-gh]: https://github.com/rust-lang-nursery/rustc-perf -This repo contains a bunch of stuff, but the sources for the tests are -found in [the `collector/benchmarks` directory][dir]. So let's go into -the directory of a specific test; we'll use `clap-rs` as an example: +#### Doing it the easy way + +Once you've cloned the repo, you can use the `collector` executable to +do profiling for you! You can find +[instructions in the rustc-perf readme][rustc-perf-readme]. + +[rustc-perf-readme]: https://github.com/rust-lang-nursery/rustc-perf/blob/master/collector/README.md#profiling + +For example, to measure the clap-rs test, you might do: + +``` +> ./target/release/collector \ + --output-repo /path/to/place/output \ + profile perf-record + --rustc /path/to/rustc/executable/from/your/build/directory + --cargo `which cargo` + --filter clap-rs + --builds Check +``` + +You can also use that same command to use cachegrind or other profiling tools. + +#### Doing it the hard way + +If you prefer to run things manually, that is also possible. You first +need to find the source for the test you want. Sources for the tests +are found in [the `collector/benchmarks` directory][dir]. So let's go +into the directory of a specific test; we'll use `clap-rs` as an +example: [dir]: https://github.com/rust-lang-nursery/rustc-perf/tree/master/collector/benchmarks From 694a90c19de298bbd7188a2502bac966f7d27f6d Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 11:29:45 -0400 Subject: [PATCH 324/648] fix link --- src/profiling/with_perf.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index 498dc1396..be6a8a477 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -12,7 +12,9 @@ This is a guide for how to profile rustc with [perf](https://perf.wiki.kernel.or - leave everything else the defaults - Run `./x.py build` to get a full build - Make a rustup toolchain pointing to that result - - see [the "build and run" section for instructions](../how-to-build-and-run.html#toolchain) + - see [the "build and run" section for instructions][b-a-r] + +[b-a-r]: ../how-to-build-and-run.html#toolchain ## Gathering a perf profile From 5665ed6a7ac8f860908c54478c5ac26854365f53 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 17:21:35 -0400 Subject: [PATCH 325/648] links are always relative to root or what? --- src/profiling/with_perf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index be6a8a477..c851d1a23 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -14,7 +14,7 @@ This is a guide for how to profile rustc with [perf](https://perf.wiki.kernel.or - Make a rustup toolchain pointing to that result - see [the "build and run" section for instructions][b-a-r] -[b-a-r]: ../how-to-build-and-run.html#toolchain +[b-a-r]: ./how-to-build-and-run.html#toolchain ## Gathering a perf profile From f72ba710ab0ad5a4eed18ec2c5a56405b7f19e1c Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 10 Sep 2018 14:38:11 -0400 Subject: [PATCH 326/648] fix various bash snippets --- src/profiling/with_perf.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index c851d1a23..7d8276ced 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -27,8 +27,8 @@ of events, though, like cache misses and so forth. The basic `perf` command is this: -``` -perf record -F99 --call-graph dwarf XXX +```bash +> perf record -F99 --call-graph dwarf XXX ``` The `-F99` tells perf to sample at 99 Hz, which avoids generating too @@ -38,8 +38,8 @@ activity). The `--call-graph dwarf` tells perf to get call-graph information from debuginfo, which is accurate. The `XXX` is the command you want to profile. So, for example, you might do: -``` -perf record -F99 --call-graph dwarf cargo + rustc +```bash +> perf record -F99 --call-graph dwarf cargo + rustc ``` to run `cargo` -- here `` should be the name of the toolchain @@ -74,9 +74,9 @@ do profiling for you! You can find For example, to measure the clap-rs test, you might do: -``` -> ./target/release/collector \ - --output-repo /path/to/place/output \ +```bash +> ./target/release/collector + --output-repo /path/to/place/output profile perf-record --rustc /path/to/rustc/executable/from/your/build/directory --cargo `which cargo` @@ -179,8 +179,8 @@ would analyze NLL performance. You can install perf-focus using `cargo install`: -``` -cargo install perf-focus +```bash +> cargo install perf-focus ``` ### Example: How much time is spent in MIR borrowck? From 732dc4782ed8d9bd771744688fe0da584cf5a6e7 Mon Sep 17 00:00:00 2001 From: Philipp Hansch Date: Mon, 10 Sep 2018 23:43:51 +0200 Subject: [PATCH 327/648] Mention incremental compilation in testing chapter The `--incremental` flag was only mentioned in [how_to_build_and_run] so far. Faster testing means faster iteration and perhaps even more added tests. [how_to_build_and_run]: https://rust-lang-nursery.github.io/rustc-guide/how-to-build-and-run.html --- src/tests/running.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/src/tests/running.md b/src/tests/running.md index e55b6ec7d..fdc603a54 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -59,7 +59,27 @@ Under the hood, the test runner invokes the standard rust test runner (the same one you get with `#[test]`), so this command would wind up filtering for tests that include "issue-1234" in the name. -Often, though, it's easier to just run the test by hand. Most tests are +## Using incremental compilation + +You can further enable the `--incremental` flag to save additional time in subsequent rebuilds: + +```bash +> ./x.py test --stage 1 src/test/ui --incremental --test-args issue-1234 +``` + +If you don't want to include the flag with every command, you can enable it in the `config.toml`, too: + +```toml +# Whether to always use incremental compilation when building rustc +incremental = true +``` + +Note that incremental compilation will use more disk space than usual. If disk space is a +concern for you, you might want to check the size of the `build` directory from time to time. + +## Running tests manually + +Sometimes it's easier and faster to just run the test by hand. Most tests are just `rs` files, so you can do something like ```bash From a2ad0a8eb276916b0760243a7e2a88c0c8b3e0fb Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 11:35:37 -0400 Subject: [PATCH 328/648] document keep-stage1 --- src/how-to-build-and-run.md | 134 +++++++++++++++++++++++++++--------- 1 file changed, 100 insertions(+), 34 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 3ad9a52ae..749340d7b 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -53,24 +53,21 @@ One thing to keep in mind is that `rustc` is a _bootstrapping_ compiler. That is, since `rustc` is written in Rust, we need to use an older version of the compiler to compile the newer version. In particular, the newer version of the compiler, `libstd`, and other tooling may use some unstable features -internally. The result is the compiling `rustc` is done in stages. - -- **Stage 0:** the stage0 compiler can be your existing - (perhaps older version of) - Rust compiler, the current _beta_ compiler or you may download the binary - from the internet. -- **Stage 1:** the code in your clone (for new version) - is then compiled with the stage0 - compiler to produce the stage1 compiler. - However, it was built with an older compiler (stage0), - so to optimize the stage1 compiler we go to next stage. -- **Stage 2:** we rebuild our stage1 compiler with itself - to produce the stage2 compiler (i.e. it builds - itself) to have all the _latest optimizations_. -- _(Optional)_ **Stage 3**: to sanity check of our new compiler, - we can build it again - with stage2 compiler which must be identical to itself, - unless something has broken. +internally. The result is the compiling `rustc` is done in stages: + +- **Stage 0:** the stage0 compiler is usually the current _beta_ compiler + (`x.py` will download it for you); you can configure `x.py` to use something + else, though. +- **Stage 1:** the code in your clone (for new version) is then + compiled with the stage0 compiler to produce the stage1 compiler. + However, it was built with an older compiler (stage0), so to + optimize the stage1 compiler we go to next stage. +- **Stage 2:** we rebuild our stage1 compiler with itself to produce + the stage2 compiler (i.e. it builds itself) to have all the _latest + optimizations_. +- _(Optional)_ **Stage 3**: to sanity check of our new compiler, we + can build it again with stage2 compiler which must be identical to + itself, unless something has broken. For hacking, often building the stage 1 compiler is enough, but for final testing and release, the stage 2 compiler is used. @@ -80,6 +77,8 @@ It is, in particular, very useful when you're doing some kind of "type-based refactoring", like renaming a method, or changing the signature of some function. + + Once you've created a config.toml, you are now ready to run `x.py`. There are a lot of options here, but let's start with what is probably the best "go to" command for building a local rust: @@ -88,27 +87,39 @@ probably the best "go to" command for building a local rust: > ./x.py build -i --stage 1 src/libstd ``` -What this command will do is the following: +This may *look* like it only builds libstd, but that is not the case. +What this command does is the following: + +- Build libstd using the stage0 compiler (using incremental) +- Build librustc using the stage0 compiler (using incremental) + - This produces the stage1 compiler +- Build libstd using the stage1 compiler (cannot use incremental) + +This final product (stage1 compiler + libs build using that compiler) +is what you need to build other rust programs. -- Using the beta compiler (also called stage 0), it will build the - standard library and rustc from the `src` directory. The resulting - compiler is called the "stage 1" compiler. - - During this build, the `-i` (or `--incremental`) switch enables incremental - compilation, so that if you later rebuild after editing things in - `src`, you can save a bit of time. -- Using this stage 1 compiler, it will build the standard library. - (this is what the `src/libstd`) means. +Note that the command includes the `-i` switch. This enables incremental +compilation. This will be used to speed up the first two steps of the process: +in particular, if you make a small change, we ought to be able to use your old +results to make producing the stage1 **compiler** faster. -This is just a subset of the full rustc build. The **full** rustc build -(what you get if you just say `./x.py build`) has quite a few more steps: +Unfortunately, incremental cannot be used to speed up making the +stage1 libraries. This is because incremental only works when you run +the *same compiler* twice in a row. In this case, we are building a +*new stage1 compiler* every time. Therefore, the old incremental +results may not apply. **As a result, you will probably find that +building the stage1 libstd is a bottleneck for you** -- but fear not, +there is a (hacky) workaround. See [the section on "recommended +workflows"](#workflow) below. -- Build stage1 rustc with stage0 compiler. -- Build libstd with stage1 compiler (up to here is the same). -- Build rustc from `src` again, this time with the stage1 compiler - (this part is new). +Note that this whole command just gives you a subset of the full rustc +build. The **full** rustc build (what you get if you just say `./x.py +build`) has quite a few more steps: + +- Build librustc rustc with the stage1 compiler. - The resulting compiler here is called the "stage2" compiler. - Build libstd with stage2 compiler. -- Build librustdoc and a bunch of other things. +- Build librustdoc and a bunch of other things with the stage2 compiler. @@ -148,6 +159,61 @@ release: 1.25.0-dev LLVM version: 4.0 ``` + + +### Suggested workflows for faster builds of the compiler + +There are two workflows that are useful for faster builders of the +compiler. + +**Check, check, and check again.** The first workflow, which is useful when doing +simple refactorings, is to run `./x.py check` continuously. Here you +are just checking that the compiler can **build**, but often that is +all you need (e.g., when renaming a method). You can then run `./x.py build` +when you acqtually need to run tests. + +In fact, it is eomtimes useful to put off tests even when you are not +100% sure the code will work. You can then keep building up +refactoring commits and only run the tests at some later time. You can +then use `git bisect` to track down **precisely** which commit caused +the problem. A nice side-effect of this style is that you are left +with a fairly fine-grained set of commits at the end, all of which +build and pass testes. This often helps reviewing. + +**Incremental builds with `--keep-stage`.** Sometimes just checking +whether the compiler builds is not enough. A common example is that +you need to add a `debug!` statement to inspect the value of some +state or better understand the problem. In that case, you really need +a full build. By leveraging incremental, though, you can often get +these builds to complete very fast (e.g., around 30 seconds): the only +catch is this requires a bit of fudging and may produce compilers that +don't work (but that is easily detected and fixed). + +The sequence of commands you want is as follows: + +- Initial build: `./x.py build -i --stage 1 src/libstd` + - As [documented above](#command), this will build a functional stage1 compiler +- Subsequent builds: `./x.py build -i --stage 1 src/libstd --keep-stage 1` + - Note that we added the `--keep-stage 1` flag here + +The effect of `--keep-stage1` is that we just *assume* that the old +standard library can be re-used. If you are editing the compiler, this +is almost always true: you haven't changed the standard library, after +all. But sometimes, it's not true: for example, if you are editing +the "metadata" part of the compiler, which controls how the compiler +encodes types and other states into the `rlib` files, or if you are editing +things that wind up in the metadata (such as the definition of the MIR). + +**The TL;DR is that you might get weird behavior from a compile when +using `--keep-stage 1`** -- for example, strange ICEs or other +panics. In that case, you should simply remove the `--keep-stage 1` +from the command and rebuild. That ought to fix the problem. + +Note: you can also use `--keep-stage 1` when running tests. Something like this: + +- Initial test run: `./x.py test -i --stage 1 src/test/ui` +- Subsequent test run: `./x.py test -i --stage 1 src/test/ui --keep-stage 1` + ### Other x.py commands Here are a few other useful x.py commands. We'll cover some of them in detail From 7d87e3cab294d564c26386bb5e579444c7369dc9 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 12:46:10 -0400 Subject: [PATCH 329/648] wrap long lines --- src/how-to-build-and-run.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 749340d7b..59c31491e 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -166,11 +166,12 @@ LLVM version: 4.0 There are two workflows that are useful for faster builders of the compiler. -**Check, check, and check again.** The first workflow, which is useful when doing -simple refactorings, is to run `./x.py check` continuously. Here you -are just checking that the compiler can **build**, but often that is -all you need (e.g., when renaming a method). You can then run `./x.py build` -when you acqtually need to run tests. +**Check, check, and check again.** The first workflow, which is useful +when doing simple refactorings, is to run `./x.py check` +continuously. Here you are just checking that the compiler can +**build**, but often that is all you need (e.g., when renaming a +method). You can then run `./x.py build` when you acqtually need to +run tests. In fact, it is eomtimes useful to put off tests even when you are not 100% sure the code will work. You can then keep building up @@ -192,7 +193,8 @@ don't work (but that is easily detected and fixed). The sequence of commands you want is as follows: - Initial build: `./x.py build -i --stage 1 src/libstd` - - As [documented above](#command), this will build a functional stage1 compiler + - As [documented above](#command), this will build a functional + stage1 compiler - Subsequent builds: `./x.py build -i --stage 1 src/libstd --keep-stage 1` - Note that we added the `--keep-stage 1` flag here From 4ab5fec4c65a4ac832b3baa541e3ef8c71d81d11 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 12:48:54 -0400 Subject: [PATCH 330/648] note about why stage1/stage2 are different --- src/how-to-build-and-run.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 59c31491e..f6c26ddf1 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -62,6 +62,15 @@ internally. The result is the compiling `rustc` is done in stages: compiled with the stage0 compiler to produce the stage1 compiler. However, it was built with an older compiler (stage0), so to optimize the stage1 compiler we go to next stage. + - (In theory, the stage1 compiler is functionally identical to the + stage2 compiler, but in practice there are subtle differences. In + particular, the stage1 compiler itself was built by stage0 and + hence not by the source in your working directory: this means that + the symbol names used in the compiler source may not match the + symbol names that would have been made by the stage1 compiler. + This can be important when using dynamic linking (e.g., with + derives. Sometimes this means that some tests don't work when run + with stage1.) - **Stage 2:** we rebuild our stage1 compiler with itself to produce the stage2 compiler (i.e. it builds itself) to have all the _latest optimizations_. From 4a6c34a1689d481c248b910141d2a3dfbd8730a2 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 12:49:16 -0400 Subject: [PATCH 331/648] fix typos --- src/how-to-build-and-run.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index f6c26ddf1..aef5b66b8 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -179,10 +179,10 @@ compiler. when doing simple refactorings, is to run `./x.py check` continuously. Here you are just checking that the compiler can **build**, but often that is all you need (e.g., when renaming a -method). You can then run `./x.py build` when you acqtually need to +method). You can then run `./x.py build` when you actually need to run tests. -In fact, it is eomtimes useful to put off tests even when you are not +In fact, it is sometimes useful to put off tests even when you are not 100% sure the code will work. You can then keep building up refactoring commits and only run the tests at some later time. You can then use `git bisect` to track down **precisely** which commit caused From 8cbf9d0b01e57689489c4641865ea27ab9e14b07 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 11:13:59 -0400 Subject: [PATCH 332/648] fix typo, add link --- src/how-to-build-and-run.md | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index aef5b66b8..2285b2685 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -49,11 +49,12 @@ use-jemalloc = false ### Running x.py and building a stage1 compiler -One thing to keep in mind is that `rustc` is a _bootstrapping_ compiler. That -is, since `rustc` is written in Rust, we need to use an older version of the -compiler to compile the newer version. In particular, the newer version of the -compiler, `libstd`, and other tooling may use some unstable features -internally. The result is the compiling `rustc` is done in stages: +One thing to keep in mind is that `rustc` is a _bootstrapping_ +compiler. That is, since `rustc` is written in Rust, we need to use an +older version of the compiler to compile the newer version. In +particular, the newer version of the compiler, `libstd`, and other +tooling may use some unstable features internally. The result is that +compiling `rustc` is done in stages: - **Stage 0:** the stage0 compiler is usually the current _beta_ compiler (`x.py` will download it for you); you can configure `x.py` to use something @@ -73,10 +74,11 @@ internally. The result is the compiling `rustc` is done in stages: with stage1.) - **Stage 2:** we rebuild our stage1 compiler with itself to produce the stage2 compiler (i.e. it builds itself) to have all the _latest - optimizations_. + optimizations_. (By default, we copy the stage1 libraries for use by + the stage2 compiler, since they ought to be identical.) - _(Optional)_ **Stage 3**: to sanity check of our new compiler, we - can build it again with stage2 compiler which must be identical to - itself, unless something has broken. + can build the libraries with the stage2 compiler. The result ought + to be identical to before, unless something has broken. For hacking, often building the stage 1 compiler is enough, but for final testing and release, the stage 2 compiler is used. @@ -86,7 +88,7 @@ It is, in particular, very useful when you're doing some kind of "type-based refactoring", like renaming a method, or changing the signature of some function. - + Once you've created a config.toml, you are now ready to run `x.py`. There are a lot of options here, but let's start with what is @@ -104,8 +106,8 @@ What this command does is the following: - This produces the stage1 compiler - Build libstd using the stage1 compiler (cannot use incremental) -This final product (stage1 compiler + libs build using that compiler) -is what you need to build other rust programs. +This final product (stage1 compiler + libs built using that compiler) +is what you need to build other rust programs. Note that the command includes the `-i` switch. This enables incremental compilation. This will be used to speed up the first two steps of the process: @@ -125,7 +127,7 @@ Note that this whole command just gives you a subset of the full rustc build. The **full** rustc build (what you get if you just say `./x.py build`) has quite a few more steps: -- Build librustc rustc with the stage1 compiler. +- Build librustc and rustc with the stage1 compiler. - The resulting compiler here is called the "stage2" compiler. - Build libstd with stage2 compiler. - Build librustdoc and a bunch of other things with the stage2 compiler. @@ -168,7 +170,7 @@ release: 1.25.0-dev LLVM version: 4.0 ``` - + ### Suggested workflows for faster builds of the compiler @@ -188,7 +190,7 @@ refactoring commits and only run the tests at some later time. You can then use `git bisect` to track down **precisely** which commit caused the problem. A nice side-effect of this style is that you are left with a fairly fine-grained set of commits at the end, all of which -build and pass testes. This often helps reviewing. +build and pass tests. This often helps reviewing. **Incremental builds with `--keep-stage`.** Sometimes just checking whether the compiler builds is not enough. A common example is that @@ -216,7 +218,7 @@ encodes types and other states into the `rlib` files, or if you are editing things that wind up in the metadata (such as the definition of the MIR). **The TL;DR is that you might get weird behavior from a compile when -using `--keep-stage 1`** -- for example, strange ICEs or other +using `--keep-stage 1`** -- for example, strange [ICEs](appendix/glossary.html) or other panics. In that case, you should simply remove the `--keep-stage 1` from the command and rebuild. That ought to fix the problem. From 576581da0dd81b6d8e6e3996bc98dba455a97040 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 11:15:17 -0400 Subject: [PATCH 333/648] fix some more typos --- src/how-to-build-and-run.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 2285b2685..17c883ffc 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -209,13 +209,14 @@ The sequence of commands you want is as follows: - Subsequent builds: `./x.py build -i --stage 1 src/libstd --keep-stage 1` - Note that we added the `--keep-stage 1` flag here -The effect of `--keep-stage1` is that we just *assume* that the old +The effect of `--keep-stage 1` is that we just *assume* that the old standard library can be re-used. If you are editing the compiler, this is almost always true: you haven't changed the standard library, after all. But sometimes, it's not true: for example, if you are editing the "metadata" part of the compiler, which controls how the compiler -encodes types and other states into the `rlib` files, or if you are editing -things that wind up in the metadata (such as the definition of the MIR). +encodes types and other states into the `rlib` files, or if you are +editing things that wind up in the metadata (such as the definition of +the MIR). **The TL;DR is that you might get weird behavior from a compile when using `--keep-stage 1`** -- for example, strange [ICEs](appendix/glossary.html) or other From 4490acdbe6eeda52ae6148631798c79caa4fa20d Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 11:16:20 -0400 Subject: [PATCH 334/648] remove silly `note:` --- src/how-to-build-and-run.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 17c883ffc..a4e6cc0f0 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -223,7 +223,8 @@ using `--keep-stage 1`** -- for example, strange [ICEs](appendix/glossary.html) panics. In that case, you should simply remove the `--keep-stage 1` from the command and rebuild. That ought to fix the problem. -Note: you can also use `--keep-stage 1` when running tests. Something like this: +You can also use `--keep-stage 1` when running tests. Something like +this: - Initial test run: `./x.py test -i --stage 1 src/test/ui` - Subsequent test run: `./x.py test -i --stage 1 src/test/ui --keep-stage 1` From 69cdc0a77e0d1fa6c436b5173d68c05a76ab6572 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 11 Sep 2018 12:02:46 -0400 Subject: [PATCH 335/648] wrap long lines --- src/how-to-build-and-run.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index a4e6cc0f0..66ba8efda 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -219,9 +219,10 @@ editing things that wind up in the metadata (such as the definition of the MIR). **The TL;DR is that you might get weird behavior from a compile when -using `--keep-stage 1`** -- for example, strange [ICEs](appendix/glossary.html) or other -panics. In that case, you should simply remove the `--keep-stage 1` -from the command and rebuild. That ought to fix the problem. +using `--keep-stage 1`** -- for example, strange +[ICEs](appendix/glossary.html) or other panics. In that case, you +should simply remove the `--keep-stage 1` from the command and +rebuild. That ought to fix the problem. You can also use `--keep-stage 1` when running tests. Something like this: From b655878d2ca05c95fe2b9b49279f4f33a1bd2214 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 11 Sep 2018 16:22:15 -0400 Subject: [PATCH 336/648] wrap lines in `running.md` --- src/tests/running.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/src/tests/running.md b/src/tests/running.md index fdc603a54..d410b79c3 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -61,21 +61,24 @@ filtering for tests that include "issue-1234" in the name. ## Using incremental compilation -You can further enable the `--incremental` flag to save additional time in subsequent rebuilds: +You can further enable the `--incremental` flag to save additional +time in subsequent rebuilds: ```bash > ./x.py test --stage 1 src/test/ui --incremental --test-args issue-1234 ``` -If you don't want to include the flag with every command, you can enable it in the `config.toml`, too: +If you don't want to include the flag with every command, you can +enable it in the `config.toml`, too: ```toml # Whether to always use incremental compilation when building rustc incremental = true ``` -Note that incremental compilation will use more disk space than usual. If disk space is a -concern for you, you might want to check the size of the `build` directory from time to time. +Note that incremental compilation will use more disk space than +usual. If disk space is a concern for you, you might want to check the +size of the `build` directory from time to time. ## Running tests manually From 0ab21a2aa5bb9789ed21801026806e1a01bc3ee6 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 09:29:38 -0400 Subject: [PATCH 337/648] rearrange to promote the borrow checker into its own section --- src/SUMMARY.md | 4 ++-- src/{mir/borrowck.md => borrow_check.md} | 0 src/{mir/regionck.md => borrow_check/region_inference.md} | 0 3 files changed, 2 insertions(+), 2 deletions(-) rename src/{mir/borrowck.md => borrow_check.md} (100%) rename src/{mir/regionck.md => borrow_check/region_inference.md} (100%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index ad8a82623..e9e2ffaea 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -53,9 +53,9 @@ - [MIR construction](./mir/construction.md) - [MIR visitor and traversal](./mir/visitor.md) - [MIR passes: getting the MIR for a function](./mir/passes.md) - - [MIR borrowck](./mir/borrowck.md) - - [MIR-based region checking (NLL)](./mir/regionck.md) - [MIR optimizations](./mir/optimizations.md) +- [The borrow checker](./borrow_check.md) + - [Region inference](./borrow_check/region_inference.md) - [Constant evaluation](./const-eval.md) - [miri const evaluator](./miri.md) - [Parameter Environments](./param_env.md) diff --git a/src/mir/borrowck.md b/src/borrow_check.md similarity index 100% rename from src/mir/borrowck.md rename to src/borrow_check.md diff --git a/src/mir/regionck.md b/src/borrow_check/region_inference.md similarity index 100% rename from src/mir/regionck.md rename to src/borrow_check/region_inference.md From d453faee7a05fb4b8ea18842cbd520abcfb46dce Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 09:57:35 -0400 Subject: [PATCH 338/648] add content --- src/SUMMARY.md | 3 + src/borrow_check.md | 48 ++++--- src/borrow_check/moves_and_initialization.md | 50 +++++++ .../moves_and_initialization/move_paths.md | 126 ++++++++++++++++++ src/borrow_check/region_inference.md | 4 +- src/borrow_check/type_check.md | 11 ++ 6 files changed, 218 insertions(+), 24 deletions(-) create mode 100644 src/borrow_check/moves_and_initialization.md create mode 100644 src/borrow_check/moves_and_initialization/move_paths.md create mode 100644 src/borrow_check/type_check.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index e9e2ffaea..338cb7fe1 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -55,6 +55,9 @@ - [MIR passes: getting the MIR for a function](./mir/passes.md) - [MIR optimizations](./mir/optimizations.md) - [The borrow checker](./borrow_check.md) + - [Tracking moves and initialization](./borrow_check/moves_and_initialization.md) + - [Move paths](./borrow_check/moves_and_initialization/move_paths.md) + - [MIR type checker](./borrow_check/type_check.md) - [Region inference](./borrow_check/region_inference.md) - [Constant evaluation](./const-eval.md) - [miri const evaluator](./miri.md) diff --git a/src/borrow_check.md b/src/borrow_check.md index d5fea6184..07095f20a 100644 --- a/src/borrow_check.md +++ b/src/borrow_check.md @@ -14,7 +14,10 @@ enforcing a number of properties: At the time of this writing, the code is in a state of transition. The "main" borrow checker still works by processing [the HIR](hir.html), but that is being phased out in favor of the MIR-based borrow checker. -Doing borrow checking on MIR has two key advantages: +Accordingly, this documentation focuses on the new, MIR-based borrow +checker. + +Doing borrow checking on MIR has several advantages: - The MIR is *far* less complex than the HIR; the radical desugaring helps prevent bugs in the borrow checker. (If you're curious, you @@ -30,30 +33,31 @@ Doing borrow checking on MIR has two key advantages: The borrow checker source is found in [the `rustc_mir::borrow_check` module][b_c]. The main entry point is -the `mir_borrowck` query. At the time of this writing, MIR borrowck can operate -in several modes, but this text will describe only the mode when NLL is enabled -(what you get with `#![feature(nll)]`). - -[b_c]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir/borrow_check +the [`mir_borrowck`] query. -The overall flow of the borrow checker is as follows: +[b_c]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/index.html +[`mir_borrowck`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/fn.mir_borrowck.html - We first create a **local copy** C of the MIR. In the coming steps, we will modify this copy in place to modify the types and things to include references to the new regions that we are computing. -- We then invoke `nll::replace_regions_in_mir` to modify this copy C. - Among other things, this function will replace all of the regions in +- We then invoke [`replace_regions_in_mir`] to modify this copy C. + Among other things, this function will replace all of the [regions](./appendix/glossary.html) in the MIR with fresh [inference variables](./appendix/glossary.html). - - (More details can be found in [the regionck section](./mir/regionck.html).) -- Next, we perform a number of [dataflow - analyses](./appendix/background.html#dataflow) - that compute what data is moved and when. The results of these analyses - are needed to do both borrow checking and region inference. -- Using the move data, we can then compute the values of all the regions in the - MIR. - - (More details can be found in [the NLL section](./mir/regionck.html).) -- Finally, the borrow checker itself runs, taking as input (a) the - results of move analysis and (b) the regions computed by the region - checker. This allows us to figure out which loans are still in scope - at any particular point. - +- Next, we perform a number of + [dataflow analyses](./appendix/background.html#dataflow) that + compute what data is moved and when. +- We then do a [second type check](borrow_check/type_check.html) across the MIR: + the purpose of this type check is to determine all of the constraints between + different regions. +- Next, we do [region inference](borrow_check/region_inference.html), which computes + the values of each region -- basically, points in the control-flow graph. +- At this point, we can compute the "borrows in scope" at each point. +- Finally, we do a second walk over the MIR, looking at the actions it + does and reporting errors. For example, if we see a statement like + `*a + 1`, then we would check that the variable `a` is initialized + and that it is not mutably borrowed, as either of those would + require an error to be reported. + - Doing this check requires the results of all the previous analyses. + +[`replace_regions_in_mir`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/fn.replace_regions_in_mir.html diff --git a/src/borrow_check/moves_and_initialization.md b/src/borrow_check/moves_and_initialization.md new file mode 100644 index 000000000..6bc8cc476 --- /dev/null +++ b/src/borrow_check/moves_and_initialization.md @@ -0,0 +1,50 @@ +# Tracking moves and initialization + +Part of the borrow checker's job is to track which variables are +"initialized" at any given point in time -- this also requires +figuring out where moves occur and tracking those. + +## Initialization and moves + +From a user's perspective, initialization -- giving a variable some +value -- and moves -- transfering ownership to another place -- might +seem like distinct topics. Indeed, our borrow checker error messages +often talk about them differently. But **within the borrow checker**, +they are not nearly as separate. Roughly speaking, the borrow checker +tracks the set of "initialized places" at any point in time. Assigning +to a previously uninitialized local variable adds it to that set; +moving from a local variable removes it from that set. + +Consider this example: + +```rust +fn foo() { + let a: Vec; + + // a is not initialized yet + + a = vec![22]; + + // a is initialized here + + std::mem::drop(a); // a is moved here + + // a is no longer initialized here + + let l = a.len(); //~ ERROR +} +``` + +Here you can see that `a` starts off as uninitialized; once it is +assigned, it becomes initialized. But when `drop(a)` is called, it +becomes uninitialized again. + +## Subsections + +To make it easier to peruse, this section is broken into a number of +subsections: + +- [Move paths](./moves_and_initialization/move_paths.html the + *move path* concept that we use to track which local variables (or parts of + local variables, in some cases) are initialized. +- *Rest not yet written* =) diff --git a/src/borrow_check/moves_and_initialization/move_paths.md b/src/borrow_check/moves_and_initialization/move_paths.md new file mode 100644 index 000000000..9d5dc3ed9 --- /dev/null +++ b/src/borrow_check/moves_and_initialization/move_paths.md @@ -0,0 +1,126 @@ +# Move paths + +In reality, it's not enough to track initialization at the granularity +of local variables. Sometimes we need to track, e.g., individual fields: + +```rust +fn foo() { + let a: (Vec, Vec) = (vec![22], vec![44]); + + // a.0 and a.1 are both initialized + + let b = a.0; // moves a.0 + + // a.0 is not initializd, but a.1 still is + + let c = a.0; // ERROR + let d = a.1; // OK +} +``` + +To handle this, we track initialization at the granularity of a **move +path**. A [`MovePath`] represents some location that the user can +initialize, move, etc. So e.g. there is a move-path representing the +local variable `a`, and there is a move-path representing `a.0`. Move +paths roughly correspond to the concept of a [`Place`] from MIR, but +they are indexed in ways that enable us to do move analysis more +efficiently. + +[`MovePath`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MovePath.html +[`Place`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/enum.Place.html + +## Move path indices + +Although there is a [`MovePath`] data structure, they are never +referenced directly. Instead, all the code passes around *indices* of +type +[`MovePathIndex`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/indexes/struct.MovePathIndex.html). If +you need to get information about a move path, you use this index with +the [`move_paths` field of the `MoveData`][move_paths]. For example, +to convert a [`MovePathIndex`] `mpi` into a MIR [`Place`], you might +access the [`MovePath::place`] field like so: + +```rust +move_data.move_paths[mpi].place +``` + +[move_paths]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MoveData.html#structfield.move_paths +[`MovePath::place`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MovePath.html#structfield.place + +## Building move paths + +One of the first things we do in the MIR borrow check is to construct +the set of move paths. This is done as part of the +[`MoveData::gather_moves`] function. This function uses a MIR visitor +called [`Gatherer`] to walk the MIR and look at how each [`Place`] +within is accessed. For each such [`Place`], it constructs a +corresponding [`MovePathIndex`]. It also records when/where that +particular move path is moved/initialized, but we'll get to that in a +later section. + +[`Gatherer`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/builder/struct.Gatherer.html +[`MoveData::gather_moves`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MoveData.html#method.gather_moves + +### Illegal move paths + +We don't actually move-paths for **every** [`Place`] that gets used. +In particular, if it is illegal to move from a [`Place`], then there +is no need for a [`MovePathIndex`]. Some examples: + +- You cannot move from a static variable, so we do not create a [`MovePathIndex`] + for static variables. +- You cannot move an individual element of an array, so if we have e.g. `foo: [String; 3]`, + there would be no move-path for `foo[1]`. +- You cannot move from inside of a borrowed reference, so if we have e.g. `foo: &String`, + there would be no move-path for `*foo`. + +These rules are enforced by the [`move_path_for`] function, which +converts a [`Place`] into a [`MovePathIndex`] -- in error cases like +those just discussed, the function returns an `Err`. This in turn +means we don't have to bother tracking whether those places are +initialized (which lowers overhead). + +[`move_path_for`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/builder/struct.Gatherer.html#method.move_path_for + +## Looking up a move-path + +If you have a [`Place`] and you would like to convert it to a [`MovePathIndex`], you +can do that using the [`MovePathLookup`] structure found in the [`rev_lookup`] field +of [`MoveData`]. There are two different methods: + +[`MovePathLookup`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MovePathLookup.html +[`rev_lookup`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MoveData.html#structfield.rev_lookup + +- [`find_local`], which takes a [`mir::Local`] representing a local + variable. This is the easier method, because we **always** create a + [`MovePathIndex`] for every local variable. +- [`find`], which takes an arbitrary [`Place`]. This method is a bit + more annoying to use, precisely because we don't have a + [`MovePathIndex`] for **every** [`Place`] (as we just discussed in + the "illegal move paths" section). Therefore, [`find`] returns a + [`LookupResult`] indicating the closest path it was able to find + that exists (e.g., for `foo[1]`, it might return just the path for + `foo`). + +[`find`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MovePathLookup.html#method.find +[`find_local`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MovePathLookup.html#method.find_local +[`mir::Local`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/struct.Local.html +[`LookupResult`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/enum.LookupResult.html + +## Cross-references + +As we noted above, move-paths are stored in a big vector and +referenced via their [`MovePathIndex`]. However, within this vector, +they are also structured into a tree. So for example if you have the +[`MovePathIndex`] for `a.b.c`, you can go to its parent move-path +`a.b`. You can also iterate over all children paths: so, from `a.b`, +you might iterate to find the path `a.b.c` (here you are iterating +just over the paths that the user **actually referenced**, not all +**possible** paths the user could have done). These references are +used for example in the [`has_any_child_of`] function, which checks +whether the dataflow results contain a value for the given move-path +(e.g., `a.b`) or any child of that move-path (e.g., `a.b.c`). + +[`Place`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/enum.Place.html +[`has_any_child_of`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/at_location/struct.FlowAtLocation.html#method.has_any_child_of + diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index 9034af8a8..47b21b0d2 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -1,11 +1,11 @@ -# MIR-based region checking (NLL) +# Region inference (NLL) The MIR-based region checking code is located in [the `rustc_mir::borrow_check::nll` module][nll]. (NLL, of course, stands for "non-lexical lifetimes", a term that will hopefully be deprecated once they become the standard kind of lifetime.) -[nll]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir/borrow_check/nll +[nll]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/index.html The MIR-based region analysis consists of two major functions: diff --git a/src/borrow_check/type_check.md b/src/borrow_check/type_check.md new file mode 100644 index 000000000..10d961554 --- /dev/null +++ b/src/borrow_check/type_check.md @@ -0,0 +1,11 @@ +# The MIR type-check + +A key component of the borrow check is the +[MIR type-check](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/type_check/index.html). +This check walks the MIR and does a complete "type check" -- the same +kind you might find in any other language. In the process of doing +this type-check, we also uncover the region constraints that apply to +the program. + + + From 87382baca61ac42528a0d475b45fb700eee6378f Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 11:24:24 -0400 Subject: [PATCH 339/648] address nits --- src/borrow_check.md | 6 +++--- src/borrow_check/moves_and_initialization.md | 12 +++++------ .../moves_and_initialization/move_paths.md | 20 ++++++++++--------- src/borrow_check/type_check.md | 3 +-- 4 files changed, 21 insertions(+), 20 deletions(-) diff --git a/src/borrow_check.md b/src/borrow_check.md index 07095f20a..40858b1b4 100644 --- a/src/borrow_check.md +++ b/src/borrow_check.md @@ -38,10 +38,10 @@ the [`mir_borrowck`] query. [b_c]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/index.html [`mir_borrowck`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/fn.mir_borrowck.html -- We first create a **local copy** C of the MIR. In the coming steps, +- We first create a **local copy** of the MIR. In the coming steps, we will modify this copy in place to modify the types and things to include references to the new regions that we are computing. -- We then invoke [`replace_regions_in_mir`] to modify this copy C. +- We then invoke [`replace_regions_in_mir`] to modify our local MIR. Among other things, this function will replace all of the [regions](./appendix/glossary.html) in the MIR with fresh [inference variables](./appendix/glossary.html). - Next, we perform a number of @@ -51,7 +51,7 @@ the [`mir_borrowck`] query. the purpose of this type check is to determine all of the constraints between different regions. - Next, we do [region inference](borrow_check/region_inference.html), which computes - the values of each region -- basically, points in the control-flow graph. + the values of each region — basically, points in the control-flow graph. - At this point, we can compute the "borrows in scope" at each point. - Finally, we do a second walk over the MIR, looking at the actions it does and reporting errors. For example, if we see a statement like diff --git a/src/borrow_check/moves_and_initialization.md b/src/borrow_check/moves_and_initialization.md index 6bc8cc476..3801a5b8c 100644 --- a/src/borrow_check/moves_and_initialization.md +++ b/src/borrow_check/moves_and_initialization.md @@ -11,9 +11,9 @@ value -- and moves -- transfering ownership to another place -- might seem like distinct topics. Indeed, our borrow checker error messages often talk about them differently. But **within the borrow checker**, they are not nearly as separate. Roughly speaking, the borrow checker -tracks the set of "initialized places" at any point in time. Assigning -to a previously uninitialized local variable adds it to that set; -moving from a local variable removes it from that set. +tracks the set of "initialized places" at any point in the source +code. Assigning to a previously uninitialized local variable adds it +to that set; moving from a local variable removes it from that set. Consider this example: @@ -36,8 +36,8 @@ fn foo() { ``` Here you can see that `a` starts off as uninitialized; once it is -assigned, it becomes initialized. But when `drop(a)` is called, it -becomes uninitialized again. +assigned, it becomes initialized. But when `drop(a)` is called, that +moves `a` into the call, and hence it becomes uninitialized again. ## Subsections @@ -47,4 +47,4 @@ subsections: - [Move paths](./moves_and_initialization/move_paths.html the *move path* concept that we use to track which local variables (or parts of local variables, in some cases) are initialized. -- *Rest not yet written* =) +- TODO *Rest not yet written* =) diff --git a/src/borrow_check/moves_and_initialization/move_paths.md b/src/borrow_check/moves_and_initialization/move_paths.md index 9d5dc3ed9..7a1239db1 100644 --- a/src/borrow_check/moves_and_initialization/move_paths.md +++ b/src/borrow_check/moves_and_initialization/move_paths.md @@ -1,7 +1,8 @@ # Move paths In reality, it's not enough to track initialization at the granularity -of local variables. Sometimes we need to track, e.g., individual fields: +of local variables. Rust also allows us to do moves and initialization +at the field granularity: ```rust fn foo() { @@ -63,9 +64,9 @@ later section. ### Illegal move paths -We don't actually move-paths for **every** [`Place`] that gets used. -In particular, if it is illegal to move from a [`Place`], then there -is no need for a [`MovePathIndex`]. Some examples: +We don't actually create a move-path for **every** [`Place`] that gets +used. In particular, if it is illegal to move from a [`Place`], then +there is no need for a [`MovePathIndex`]. Some examples: - You cannot move from a static variable, so we do not create a [`MovePathIndex`] for static variables. @@ -115,11 +116,12 @@ they are also structured into a tree. So for example if you have the [`MovePathIndex`] for `a.b.c`, you can go to its parent move-path `a.b`. You can also iterate over all children paths: so, from `a.b`, you might iterate to find the path `a.b.c` (here you are iterating -just over the paths that the user **actually referenced**, not all -**possible** paths the user could have done). These references are -used for example in the [`has_any_child_of`] function, which checks -whether the dataflow results contain a value for the given move-path -(e.g., `a.b`) or any child of that move-path (e.g., `a.b.c`). +just over the paths that are **actually referenced** in the source, +not all **possible** paths that could have been referenced). These +references are used for example in the [`has_any_child_of`] function, +which checks whether the dataflow results contain a value for the +given move-path (e.g., `a.b`) or any child of that move-path (e.g., +`a.b.c`). [`Place`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/enum.Place.html [`has_any_child_of`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/at_location/struct.FlowAtLocation.html#method.has_any_child_of diff --git a/src/borrow_check/type_check.md b/src/borrow_check/type_check.md index 10d961554..ee955d971 100644 --- a/src/borrow_check/type_check.md +++ b/src/borrow_check/type_check.md @@ -7,5 +7,4 @@ kind you might find in any other language. In the process of doing this type-check, we also uncover the region constraints that apply to the program. - - +TODO -- elaborate further? Maybe? :) From d2cecbf3cd162fbc842dc3c66c8bec702f2f648f Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 17:23:47 -0400 Subject: [PATCH 340/648] fix glossary links --- src/appendix/glossary.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index bfc2c0d22..42315536a 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -40,7 +40,7 @@ MIR | the Mid-level IR that is created after type-checking miri | an interpreter for MIR used for constant evaluation ([see more](./miri.html)) normalize | a general term for converting to a more canonical form, but in the case of rustc typically refers to [associated type normalization](./traits/associated-types.html#normalize) newtype | a "newtype" is a wrapper around some other type (e.g., `struct Foo(T)` is a "newtype" for `T`). This is commonly used in Rust to give a stronger type for indices. -NLL | [non-lexical lifetimes](./mir/regionck.html), an extension to Rust's borrowing system to make it be based on the control-flow graph. +NLL | [non-lexical lifetimes](./borrow_check/region_inference.html), an extension to Rust's borrowing system to make it be based on the control-flow graph. node-id or NodeId | an index identifying a particular node in the AST or HIR; gradually being phased out and replaced with `HirId`. obligation | something that must be proven by the trait system ([see more](traits/resolution.html)) projection | a general term for a "relative path", e.g. `x.f` is a "field projection", and `T::Item` is an ["associated type projection"](./traits/goals-and-clauses.html#trait-ref) @@ -53,7 +53,7 @@ rib | a data structure in the name resolver that keeps trac sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. sigil | like a keyword but composed entirely of non-alphanumeric tokens. For example, `&` is a sigil for references. -skolemization | a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)`) as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on skolemization and universes](./mir/regionck.html#skol) for more details. +skolemization | a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)`) as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on skolemization and universes](./borrow_check/region_inference.html#skol) for more details. soundness | soundness is a technical term in type theory. Roughly, if a type system is sound, then if a program type-checks, it is type-safe; i.e. I can never (in safe rust) force a value into a variable of the wrong type. (see "completeness"). span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. substs | the substitutions for a given generic type or item (e.g. the `i32`, `u32` in `HashMap`) From a72a0687227651f8eb90e4c996535e90f849ebd2 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 10 Sep 2018 14:34:48 -0400 Subject: [PATCH 341/648] bump max line length to 100 otherwise our summary.md doesn't fit --- .travis.yml | 2 +- ci/check_line_lengths.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index d7f8f8f42..55ea20587 100644 --- a/.travis.yml +++ b/.travis.yml @@ -3,7 +3,7 @@ cache: - cargo before_install: - shopt -s globstar -- MAX_LINE_LENGTH=80 bash ci/check_line_lengths.sh src/**/*.md +- MAX_LINE_LENGTH=100 bash ci/check_line_lengths.sh src/**/*.md install: - source ~/.cargo/env || true - bash ci/install.sh diff --git a/ci/check_line_lengths.sh b/ci/check_line_lengths.sh index 91f199b7e..5b7b12d3e 100755 --- a/ci/check_line_lengths.sh +++ b/ci/check_line_lengths.sh @@ -2,7 +2,7 @@ if [ "$1" == "--help" ]; then echo 'Usage:' - echo ' MAX_LINE_LENGTH=80' "$0" 'src/**/*.md' + echo ' MAX_LINE_LENGTH=100' "$0" 'src/**/*.md' exit 1 fi From a6284183f8b69094513da8ac7528268b4eb8255d Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 11 Sep 2018 16:43:36 -0400 Subject: [PATCH 342/648] ignore various sample tests --- src/borrow_check/moves_and_initialization.md | 2 +- src/borrow_check/moves_and_initialization/move_paths.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/borrow_check/moves_and_initialization.md b/src/borrow_check/moves_and_initialization.md index 3801a5b8c..d1cd41e0f 100644 --- a/src/borrow_check/moves_and_initialization.md +++ b/src/borrow_check/moves_and_initialization.md @@ -17,7 +17,7 @@ to that set; moving from a local variable removes it from that set. Consider this example: -```rust +```rust,ignore fn foo() { let a: Vec; diff --git a/src/borrow_check/moves_and_initialization/move_paths.md b/src/borrow_check/moves_and_initialization/move_paths.md index 7a1239db1..c9e22a81c 100644 --- a/src/borrow_check/moves_and_initialization/move_paths.md +++ b/src/borrow_check/moves_and_initialization/move_paths.md @@ -4,7 +4,7 @@ In reality, it's not enough to track initialization at the granularity of local variables. Rust also allows us to do moves and initialization at the field granularity: -```rust +```rust,ignore fn foo() { let a: (Vec, Vec) = (vec![22], vec![44]); @@ -41,7 +41,7 @@ the [`move_paths` field of the `MoveData`][move_paths]. For example, to convert a [`MovePathIndex`] `mpi` into a MIR [`Place`], you might access the [`MovePath::place`] field like so: -```rust +```rust,ignore move_data.move_paths[mpi].place ``` From c4ac71e1d057ee6f65484aa516e4b9ddf59e3f9a Mon Sep 17 00:00:00 2001 From: Philipp Hansch Date: Tue, 11 Sep 2018 07:36:49 +0200 Subject: [PATCH 343/648] Fix typos These were fixed using [codespell](https://github.com/codespell-project/codespell). --- src/appendix/glossary.md | 4 ++-- src/borrow_check/region_inference.md | 4 ++-- src/codegen.md | 6 +++--- src/compiletest.md | 2 +- src/lowering.md | 2 +- src/macro-expansion.md | 6 +++--- src/rustdoc.md | 2 +- src/traits/canonicalization.md | 2 +- src/traits/lowering-rules.md | 2 +- src/traits/resolution.md | 2 +- 10 files changed, 16 insertions(+), 16 deletions(-) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 42315536a..44ac26ca3 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -14,7 +14,7 @@ codegen unit | when we produce LLVM IR, we group the Rust code into completeness | completeness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). control-flow graph | a representation of the control-flow of a program; see [the background chapter for more](./appendix/background.html#cfg) CTFE | Compile-Time Function Evaluation. This is the ability of the compiler to evaluate `const fn`s at compile time. This is part of the compiler's constant evaluation system. ([see more](./const-eval.html)) -cx | we tend to use "cx" as an abbrevation for context. See also `tcx`, `infcx`, etc. +cx | we tend to use "cx" as an abbreviation for context. See also `tcx`, `infcx`, etc. DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](incremental-compilation.html)) data-flow analysis | a static analysis that figures out what properties are true at each point in the control-flow of a program; see [the background chapter for more](./appendix/background.html#dataflow) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. @@ -34,7 +34,7 @@ inference variable | when doing type or region inference, an "inference va infcx | the inference context (see `librustc/infer`) IR | Intermediate Representation. A general term in compilers. During compilation, the code is transformed from raw source (ASCII text) to various IRs. In Rust, these are primarily HIR, MIR, and LLVM IR. Each IR is well-suited for some set of computations. For example, MIR is well-suited for the borrow checker, and LLVM IR is well-suited for codegen because LLVM accepts it. local crate | the crate currently being compiled. -LTO | Link-Time Optimizations. A set of optimizations offered by LLVM that occur just before the final binary is linked. These include optmizations like removing functions that are never used in the final program, for example. _ThinLTO_ is a variant of LTO that aims to be a bit more scalable and efficient, but possibly sacrifices some optimizations. You may also read issues in the Rust repo about "FatLTO", which is the loving nickname given to non-Thin LTO. LLVM documentation: [here][lto] and [here][thinlto] +LTO | Link-Time Optimizations. A set of optimizations offered by LLVM that occur just before the final binary is linked. These include optimizations like removing functions that are never used in the final program, for example. _ThinLTO_ is a variant of LTO that aims to be a bit more scalable and efficient, but possibly sacrifices some optimizations. You may also read issues in the Rust repo about "FatLTO", which is the loving nickname given to non-Thin LTO. LLVM documentation: [here][lto] and [here][thinlto] [LLVM] | (actually not an acronym :P) an open-source compiler backend. It accepts LLVM IR and outputs native binaries. Various languages (e.g. Rust) can then implement a compiler front-end that output LLVM IR and use LLVM to compile to all the platforms LLVM supports. MIR | the Mid-level IR that is created after type-checking for use by borrowck and codegen ([see more](./mir/index.html)) miri | an interpreter for MIR used for constant evaluation ([see more](./miri.html)) diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index 47b21b0d2..2cd6066ec 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -23,7 +23,7 @@ The MIR-based region analysis consists of two major functions: won't be doing lexical region inference at all. - `compute_regions`, invoked second: this is given as argument the results of move analysis. It has the job of computing values for all - the inference variabes that `replace_regions_in_mir` introduced. + the inference variables that `replace_regions_in_mir` introduced. - To do that, it first runs the [MIR type checker](#mirtypeck). This is basically a normal type-checker but specialized to MIR, which is much simpler than full Rust of course. Running the MIR type @@ -531,7 +531,7 @@ then we have this constraint `V2: V3`, so we wind up having to enlarge V2 in U2 = {skol(1), skol(2)} ``` -Now contraint propagation is done, but when we check the outlives +Now constraint propagation is done, but when we check the outlives relationships, we find that `V2` includes this new element `skol(1)`, so we report an error. diff --git a/src/codegen.md b/src/codegen.md index 6e68db6ea..766dfff59 100644 --- a/src/codegen.md +++ b/src/codegen.md @@ -10,7 +10,7 @@ generates an executable binary. rustc uses LLVM for code generation. ## What is LLVM? -All of the preceeding chapters of this guide have one thing in common: we never +All of the preceding chapters of this guide have one thing in common: we never generated any executable machine code at all! With this chapter, all of that changes. @@ -29,14 +29,14 @@ many compiler projects, including the `clang` C compiler and our beloved LLVM's "format `X`" is called LLVM IR. It is basically assembly code with additional low-level types and annotations added. These annotations are helpful -for doing optimizations on the LLVM IR and outputed machine code. The end +for doing optimizations on the LLVM IR and outputted machine code. The end result of all this is (at long last) something executable (e.g. an ELF object or wasm). There are a few benefits to using LLVM: - We don't have to write a whole compiler backend. This reduces implementation - and maintainance burden. + and maintenance burden. - We benefit from the large suite of advanced optimizations that the LLVM project has been collecting. - We automatically can compile Rust to any of the platforms for which LLVM has diff --git a/src/compiletest.md b/src/compiletest.md index 40d06136e..3cfb943b0 100644 --- a/src/compiletest.md +++ b/src/compiletest.md @@ -115,7 +115,7 @@ which make it simple to parse common patterns like simple presence or not attribute is defined (`has_cfg_prefix()`) and many more. The low-level parsers are found near the end of the `impl Config` block; be sure to look through them and their associated parsers immediately above to see how they are used to -avoid writing additional parsing code unneccessarily. +avoid writing additional parsing code unnecessarily. As a concrete example, here is the implementation for the `parse_failure_status()` parser, in diff --git a/src/lowering.md b/src/lowering.md index eddc00af9..2504ed491 100644 --- a/src/lowering.md +++ b/src/lowering.md @@ -36,7 +36,7 @@ sanity checks in `src/librustc/hir/map/hir_id_validator.rs`: for you so you also get the `HirId`. If you are creating new `DefId`s, since each `DefId` needs to have a -corresponding `NodeId`, it is adviseable to add these `NodeId`s to the +corresponding `NodeId`, it is advisable to add these `NodeId`s to the `AST` so you don't have to generate new ones during lowering. This has the advantage of creating a way to find the `DefId` of something via its `NodeId`. If lowering needs this `DefId` in multiple places, you can't diff --git a/src/macro-expansion.md b/src/macro-expansion.md index bb3116d40..b03a218c2 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -90,13 +90,13 @@ tokens containing the inside of the example invocation `print foo`, while `ms` might be the sequence of token (trees) `print $mvar:ident`. The output of the parser is a `NamedParseResult`, which indicates which of -three cases has occured: +three cases has occurred: - Success: `tts` matches the given matcher `ms`, and we have produced a binding from metavariables to the corresponding token trees. - Failure: `tts` does not match `ms`. This results in an error message such as "No rule expected token _blah_". -- Error: some fatal error has occured _in the parser_. For example, this happens +- Error: some fatal error has occurred _in the parser_. For example, this happens if there are more than one pattern match, since that indicates the macro is ambiguous. @@ -112,7 +112,7 @@ the macro parser. This is extremely non-intuitive and self-referential. The code to parse macro _definitions_ is in [`src/libsyntax/ext/tt/macro_rules.rs`][code_mr]. It defines the pattern for matching for a macro definition as `$( $lhs:tt => $rhs:tt );+`. In other words, -a `macro_rules` defintion should have in its body at least one occurence of a +a `macro_rules` definition should have in its body at least one occurrence of a token tree followed by `=>` followed by another token tree. When the compiler comes to a `macro_rules` definition, it uses this pattern to match the two token trees per rule in the definition of the macro _using the macro parser itself_. diff --git a/src/rustdoc.md b/src/rustdoc.md index 76a8b28ce..318a8e2e1 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -30,7 +30,7 @@ does is call the `main()` that's in this crate's `lib.rs`, though.) ## Cheat sheet -* Use `./x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable +* Use `./x.py build --stage 1 src/libstd src/tools/rustdoc` to make a usable rustdoc you can run on other projects. * Add `src/libtest` to be able to use `rustdoc --test`. * If you've used `rustup toolchain link local /path/to/build/$TARGET/stage1` diff --git a/src/traits/canonicalization.md b/src/traits/canonicalization.md index 1ef1da717..2cc35d899 100644 --- a/src/traits/canonicalization.md +++ b/src/traits/canonicalization.md @@ -56,7 +56,7 @@ for { ?0: Foo<'?1, ?2> } This `for<>` gives some information about each of the canonical variables within. In this case, each `T` indicates a type variable, -so `?0` and `?2` are types; the `L` indicates a lifetime varibale, so +so `?0` and `?2` are types; the `L` indicates a lifetime variable, so `?1` is a lifetime. The `canonicalize` method *also* gives back a `CanonicalVarValues` array OV with the "original values" for each canonicalized variable: diff --git a/src/traits/lowering-rules.md b/src/traits/lowering-rules.md index e37999e57..41c9625f6 100644 --- a/src/traits/lowering-rules.md +++ b/src/traits/lowering-rules.md @@ -10,7 +10,7 @@ reference the [domain goals][dg] defined in an earlier section. ## Notation The nonterminal `Pi` is used to mean some generic *parameter*, either a -named lifetime like `'a` or a type paramter like `A`. +named lifetime like `'a` or a type parameter like `A`. The nonterminal `Ai` is used to mean some generic *argument*, which might be a lifetime like `'a` or a type like `Vec`. diff --git a/src/traits/resolution.md b/src/traits/resolution.md index e33e0c8ec..987d9a28d 100644 --- a/src/traits/resolution.md +++ b/src/traits/resolution.md @@ -117,7 +117,7 @@ know whether an impl/where-clause applies or not – this occurs when the obligation contains unbound inference variables. The subroutines that decide whether a particular impl/where-clause/etc -applies to a particular obligation are collectively refered to as the +applies to a particular obligation are collectively referred to as the process of _matching_. At the moment, this amounts to unifying the `Self` types, but in the future we may also recursively consider some of the nested obligations, in the case of an impl. From 5ec3e1a6206fefba98139ca069435ff4b759d045 Mon Sep 17 00:00:00 2001 From: Philipp Hansch Date: Tue, 11 Sep 2018 07:47:13 +0200 Subject: [PATCH 344/648] Fix line lengths --- src/macro-expansion.md | 6 +++--- src/tests/running.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/macro-expansion.md b/src/macro-expansion.md index b03a218c2..c62301247 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -96,9 +96,9 @@ three cases has occurred: from metavariables to the corresponding token trees. - Failure: `tts` does not match `ms`. This results in an error message such as "No rule expected token _blah_". -- Error: some fatal error has occurred _in the parser_. For example, this happens - if there are more than one pattern match, since that indicates the macro is - ambiguous. +- Error: some fatal error has occurred _in the parser_. For example, this + happens if there are more than one pattern match, since that indicates + the macro is ambiguous. The full interface is defined [here][code_parse_int]. diff --git a/src/tests/running.md b/src/tests/running.md index d410b79c3..778fd00ec 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -76,9 +76,9 @@ enable it in the `config.toml`, too: incremental = true ``` -Note that incremental compilation will use more disk space than -usual. If disk space is a concern for you, you might want to check the -size of the `build` directory from time to time. +Note that incremental compilation will use more disk space than usual. +If disk space is a concern for you, you might want to check the size +of the `build` directory from time to time. ## Running tests manually From 6e7e4d2d35e871e538152b2dfde2265dd662d1a9 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 20 Aug 2018 11:15:35 -0500 Subject: [PATCH 345/648] use mdbook 0.2 --- ci/install.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install.sh b/ci/install.sh index 29e3a2f9b..dc413f68c 100755 --- a/ci/install.sh +++ b/ci/install.sh @@ -20,5 +20,5 @@ function cargo_install() { fi } -cargo_install mdbook 0.1.8 +cargo_install mdbook 0.2 cargo_install mdbook-linkcheck 0.1.2 From 5e7d9f6b4226c1b13d6b1bbc1b4585b3cce7e173 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 20 Aug 2018 11:19:02 -0500 Subject: [PATCH 346/648] update readme with mdbook version --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a01758d48..57b3a0ea4 100644 --- a/README.md +++ b/README.md @@ -38,4 +38,4 @@ be found. > cargo install mdbook-linkcheck ``` You will need `mdbook` version `>= 0.1`. `linkcheck` will be run automatically -when you run `mdbook build`. +when you run `mdbook build`. We are currently using `mdbook 0.2`. From 5ca9b3b3260a33b784eab6d299b8a864620ab5e1 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sat, 8 Sep 2018 21:35:59 -0500 Subject: [PATCH 347/648] Fix all the links to be relative for mdbook 2 --- src/appendix/background.md | 2 +- src/appendix/code-index.md | 26 ++++++------- src/appendix/glossary.md | 58 ++++++++++++++-------------- src/borrow_check/region_inference.md | 8 ++-- src/mir/index.md | 6 +-- src/mir/passes.md | 4 +- src/tests/intro.md | 8 ++-- src/traits/associated-types.md | 4 +- src/traits/caching.md | 8 ++-- src/traits/canonical-queries.md | 10 ++--- src/traits/canonicalization.md | 10 ++--- src/traits/chalk-overview.md | 12 +++--- src/traits/goals-and-clauses.md | 14 +++---- src/traits/index.md | 14 +++---- src/traits/lowering-module.md | 4 +- src/traits/lowering-rules.md | 10 ++--- src/traits/lowering-to-logic.md | 4 +- src/traits/resolution.md | 4 +- 18 files changed, 103 insertions(+), 103 deletions(-) diff --git a/src/appendix/background.md b/src/appendix/background.md index efdce4f2a..dbfe6a3db 100644 --- a/src/appendix/background.md +++ b/src/appendix/background.md @@ -94,7 +94,7 @@ and Michael I. Schwartzbach is an incredible resource! Check out the subtyping chapter from the [Rust Nomicon](https://doc.rust-lang.org/nomicon/subtyping.html). -See the [variance](./variance.html) chapter of this guide for more info on how +See the [variance](../variance.html) chapter of this guide for more info on how the type checker handles variance. diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index 75d35c6c5..e1bde6863 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -29,16 +29,16 @@ Item | Kind | Short description | Chapter | `Ty<'tcx>` | struct | This is the internal representation of a type used for type checking | [Type checking] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/type.Ty.html) `TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries | [The `ty` modules] | [src/librustc/ty/context.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html) -[The HIR]: hir.html -[Identifiers in the HIR]: hir.html#hir-id -[The parser]: the-parser.html -[The Rustc Driver]: rustc-driver.html -[Type checking]: type-checking.html -[The `ty` modules]: ty.html -[Rustdoc]: rustdoc.html -[Emitting Diagnostics]: diag.html -[Macro expansion]: macro-expansion.html -[Name resolution]: name-resolution.html -[Parameter Environment]: param_env.html -[Trait Solving: Goals and Clauses]: traits/goals-and-clauses.html#domain-goals -[Trait Solving: Lowering impls]: traits/lowering-rules.html#lowering-impls +[The HIR]: ../hir.html +[Identifiers in the HIR]: ../hir.html#hir-id +[The parser]: ../the-parser.html +[The Rustc Driver]: ../rustc-driver.html +[Type checking]: ../type-checking.html +[The `ty` modules]: ../ty.html +[Rustdoc]: ../rustdoc.html +[Emitting Diagnostics]: ../diag.html +[Macro expansion]: ../macro-expansion.html +[Name resolution]: ../name-resolution.html +[Parameter Environment]: ../param_env.html +[Trait Solving: Goals and Clauses]: ../traits/goals-and-clauses.html#domain-goals +[Trait Solving: Lowering impls]: ../traits/lowering-rules.html#lowering-impls diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 44ac26ca3..5c7d82741 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -7,25 +7,25 @@ them better. Term | Meaning ------------------------|-------- AST | the abstract syntax tree produced by the syntax crate; reflects user syntax very closely. -binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and \|`a`\|` ...` is a binder for the parameter `a`. See [the background chapter for more](./appendix/background.html#free-vs-bound) -bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession \|`a`\|` a * 2`. See [the background chapter for more](./appendix/background.html#free-vs-bound) +binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and \|`a`\|` ...` is a binder for the parameter `a`. See [the background chapter for more](./background.html#free-vs-bound) +bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession \|`a`\|` a * 2`. See [the background chapter for more](./background.html#free-vs-bound) codegen | the code to translate MIR into LLVM IR. codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. completeness | completeness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). -control-flow graph | a representation of the control-flow of a program; see [the background chapter for more](./appendix/background.html#cfg) -CTFE | Compile-Time Function Evaluation. This is the ability of the compiler to evaluate `const fn`s at compile time. This is part of the compiler's constant evaluation system. ([see more](./const-eval.html)) +control-flow graph | a representation of the control-flow of a program; see [the background chapter for more](./background.html#cfg) +CTFE | Compile-Time Function Evaluation. This is the ability of the compiler to evaluate `const fn`s at compile time. This is part of the compiler's constant evaluation system. ([see more](../const-eval.html)) cx | we tend to use "cx" as an abbreviation for context. See also `tcx`, `infcx`, etc. -DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](incremental-compilation.html)) -data-flow analysis | a static analysis that figures out what properties are true at each point in the control-flow of a program; see [the background chapter for more](./appendix/background.html#dataflow) +DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](../incremental-compilation.html)) +data-flow analysis | a static analysis that figures out what properties are true at each point in the control-flow of a program; see [the background chapter for more](./background.html#dataflow) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. Double pointer | a pointer with additional metadata. See "fat pointer" for more. DST | Dynamically-Sized Type. A type for which the compiler cannot statically know the size in memory (e.g. `str` or `[u8]`). Such types don't implement `Sized` and cannot be allocated on the stack. They can only occur as the last field in a struct. They can only be used behind a pointer (e.g. `&str` or `&[u8]`). empty type | see "uninhabited type". Fat pointer | a two word value carrying the address of some value, along with some further information necessary to put the value to use. Rust includes two kinds of "fat pointers": references to slices, and trait objects. A reference to a slice carries the starting address of the slice and its length. A trait object carries a value's address and a pointer to the trait's implementation appropriate to that value. "Fat pointers" are also known as "wide pointers", and "double pointers". -free variable | a "free variable" is one that is not bound within an expression or term; see [the background chapter for more](./appendix/background.html#free-vs-bound) -'gcx | the lifetime of the global arena ([see more](ty.html)) +free variable | a "free variable" is one that is not bound within an expression or term; see [the background chapter for more](./background.html#free-vs-bound) +'gcx | the lifetime of the global arena ([see more](../ty.html)) generics | the set of generic type parameters defined on a type or item -HIR | the High-level IR, created by lowering and desugaring the AST ([see more](hir.html)) +HIR | the High-level IR, created by lowering and desugaring the AST ([see more](../hir.html)) HirId | identifies a particular node in the HIR by combining a def-id with an "intra-definition offset". HIR Map | The HIR map, accessible via tcx.hir, allows you to quickly navigate the HIR and convert between various forms of identifiers. ICE | internal compiler error. When the compiler crashes. @@ -36,39 +36,39 @@ IR | Intermediate Representation. A general term in compil local crate | the crate currently being compiled. LTO | Link-Time Optimizations. A set of optimizations offered by LLVM that occur just before the final binary is linked. These include optimizations like removing functions that are never used in the final program, for example. _ThinLTO_ is a variant of LTO that aims to be a bit more scalable and efficient, but possibly sacrifices some optimizations. You may also read issues in the Rust repo about "FatLTO", which is the loving nickname given to non-Thin LTO. LLVM documentation: [here][lto] and [here][thinlto] [LLVM] | (actually not an acronym :P) an open-source compiler backend. It accepts LLVM IR and outputs native binaries. Various languages (e.g. Rust) can then implement a compiler front-end that output LLVM IR and use LLVM to compile to all the platforms LLVM supports. -MIR | the Mid-level IR that is created after type-checking for use by borrowck and codegen ([see more](./mir/index.html)) -miri | an interpreter for MIR used for constant evaluation ([see more](./miri.html)) -normalize | a general term for converting to a more canonical form, but in the case of rustc typically refers to [associated type normalization](./traits/associated-types.html#normalize) +MIR | the Mid-level IR that is created after type-checking for use by borrowck and codegen ([see more](../mir/index.html)) +miri | an interpreter for MIR used for constant evaluation ([see more](../miri.html)) +normalize | a general term for converting to a more canonical form, but in the case of rustc typically refers to [associated type normalization](../traits/associated-types.html#normalize) newtype | a "newtype" is a wrapper around some other type (e.g., `struct Foo(T)` is a "newtype" for `T`). This is commonly used in Rust to give a stronger type for indices. -NLL | [non-lexical lifetimes](./borrow_check/region_inference.html), an extension to Rust's borrowing system to make it be based on the control-flow graph. +NLL | [non-lexical lifetimes](../borrow_check/region_inference.html), an extension to Rust's borrowing system to make it be based on the control-flow graph. node-id or NodeId | an index identifying a particular node in the AST or HIR; gradually being phased out and replaced with `HirId`. -obligation | something that must be proven by the trait system ([see more](traits/resolution.html)) -projection | a general term for a "relative path", e.g. `x.f` is a "field projection", and `T::Item` is an ["associated type projection"](./traits/goals-and-clauses.html#trait-ref) -promoted constants | constants extracted from a function and lifted to static scope; see [this section](./mir/index.html#promoted) for more details. -provider | the function that executes a query ([see more](query.html)) -quantified | in math or logic, existential and universal quantification are used to ask questions like "is there any type T for which is true?" or "is this true for all types T?"; see [the background chapter for more](./appendix/background.html#quantified) -query | perhaps some sub-computation during compilation ([see more](query.html)) +obligation | something that must be proven by the trait system ([see more](../traits/resolution.html)) +projection | a general term for a "relative path", e.g. `x.f` is a "field projection", and `T::Item` is an ["associated type projection"](../traits/goals-and-clauses.html#trait-ref) +promoted constants | constants extracted from a function and lifted to static scope; see [this section](../mir/index.html#promoted) for more details. +provider | the function that executes a query ([see more](../query.html)) +quantified | in math or logic, existential and universal quantification are used to ask questions like "is there any type T for which is true?" or "is this true for all types T?"; see [the background chapter for more](./background.html#quantified) +query | perhaps some sub-computation during compilation ([see more](../query.html)) region | another term for "lifetime" often used in the literature and in the borrow checker. -rib | a data structure in the name resolver that keeps track of a single scope for names. ([see more](./name-resolution.html)) +rib | a data structure in the name resolver that keeps track of a single scope for names. ([see more](../name-resolution.html)) sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. sigil | like a keyword but composed entirely of non-alphanumeric tokens. For example, `&` is a sigil for references. -skolemization | a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)`) as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on skolemization and universes](./borrow_check/region_inference.html#skol) for more details. +skolemization | a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)`) as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on skolemization and universes](../borrow_check/region_inference.html#skol) for more details. soundness | soundness is a technical term in type theory. Roughly, if a type system is sound, then if a program type-checks, it is type-safe; i.e. I can never (in safe rust) force a value into a variable of the wrong type. (see "completeness"). span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. substs | the substitutions for a given generic type or item (e.g. the `i32`, `u32` in `HashMap`) -tcx | the "typing context", main data structure of the compiler ([see more](ty.html)) -'tcx | the lifetime of the currently active inference context ([see more](ty.html)) -trait reference | the name of a trait along with a suitable set of input type/lifetimes ([see more](./traits/goals-and-clauses.html#trait-ref)) -token | the smallest unit of parsing. Tokens are produced after lexing ([see more](the-parser.html)). +tcx | the "typing context", main data structure of the compiler ([see more](../ty.html)) +'tcx | the lifetime of the currently active inference context ([see more](../ty.html)) +trait reference | the name of a trait along with a suitable set of input type/lifetimes ([see more](../traits/goals-and-clauses.html#trait-ref)) +token | the smallest unit of parsing. Tokens are produced after lexing ([see more](../the-parser.html)). [TLS] | Thread-Local Storage. Variables may be defined so that each thread has its own copy (rather than all threads sharing the variable). This has some interactions with LLVM. Not all platforms support TLS. trans | the code to translate MIR into LLVM IR. Renamed to codegen. -trait reference | a trait and values for its type parameters ([see more](ty.html)). -ty | the internal representation of a type ([see more](ty.html)). -UFCS | Universal Function Call Syntax. An unambiguous syntax for calling a method ([see more](type-checking.html)). +trait reference | a trait and values for its type parameters ([see more](../ty.html)). +ty | the internal representation of a type ([see more](../ty.html)). +UFCS | Universal Function Call Syntax. An unambiguous syntax for calling a method ([see more](../type-checking.html)). uninhabited type | a type which has _no_ values. This is not the same as a ZST, which has exactly 1 value. An example of an uninhabited type is `enum Foo {}`, which has no variants, and so, can never be created. The compiler can treat code that deals with uninhabited types as dead code, since there is no such value to be manipulated. `!` (the never type) is an uninhabited type. Uninhabited types are also called "empty types". upvar | a variable captured by a closure from outside the closure. -variance | variance determines how changes to a generic type/lifetime parameter affect subtyping; for example, if `T` is a subtype of `U`, then `Vec` is a subtype `Vec` because `Vec` is *covariant* in its generic parameter. See [the background chapter](./appendix/background.html#variance) for a more general explanation. See the [variance chapter](./variance.html) for an explanation of how type checking handles variance. +variance | variance determines how changes to a generic type/lifetime parameter affect subtyping; for example, if `T` is a subtype of `U`, then `Vec` is a subtype `Vec` because `Vec` is *covariant* in its generic parameter. See [the background chapter](./background.html#variance) for a more general explanation. See the [variance chapter](../variance.html) for an explanation of how type checking handles variance. Wide pointer | a pointer with additional metadata. See "fat pointer" for more. ZST | Zero-Sized Type. A type whose values have size 0 bytes. Since `2^0 = 1`, such types can have exactly one value. For example, `()` (unit) is a ZST. `struct Foo;` is also a ZST. The compiler can do some nice optimizations around ZSTs. diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index 2cd6066ec..5a09f0bb0 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -35,7 +35,7 @@ The MIR-based region analysis consists of two major functions: - More details to come, though the [NLL RFC] also includes fairly thorough (and hopefully readable) coverage. -[fvb]: appendix/background.html#free-vs-bound +[fvb]: ../appendix/background.html#free-vs-bound [NLL RFC]: http://rust-lang.github.io/rfcs/2094-nll.html ## Universal regions @@ -131,7 +131,7 @@ the type of `foo` the type `bar` expects We handle this sort of subtyping by taking the variables that are bound in the supertype and **skolemizing** them: this means that we replace them with -[universally quantified](appendix/background.html#quantified) +[universally quantified](../appendix/background.html#quantified) representatives, written like `!1`. We call these regions "skolemized regions" – they represent, basically, "some unknown region". @@ -148,7 +148,7 @@ what we wanted. So let's work through what happens next. To check if two functions are subtypes, we check if their arguments have the desired relationship -(fn arguments are [contravariant](./appendix/background.html#variance), so +(fn arguments are [contravariant](../appendix/background.html#variance), so we swap the left and right here): ```text @@ -187,7 +187,7 @@ Here, the root universe would consist of the lifetimes `'static` and the same concept to types, in which case the types `Foo` and `T` would be in the root universe (along with other global types, like `i32`). Basically, the root universe contains all the names that -[appear free](./appendix/background.html#free-vs-bound) in the body of `bar`. +[appear free](../appendix/background.html#free-vs-bound) in the body of `bar`. Now let's extend `bar` a bit by adding a variable `x`: diff --git a/src/mir/index.md b/src/mir/index.md index ea72decdc..d69606d66 100644 --- a/src/mir/index.md +++ b/src/mir/index.md @@ -1,7 +1,7 @@ # The MIR (Mid-level IR) MIR is Rust's _Mid-level Intermediate Representation_. It is -constructed from [HIR](./hir.html). MIR was introduced in +constructed from [HIR](../hir.html). MIR was introduced in [RFC 1211]. It is a radically simplified form of Rust that is used for certain flow-sensitive safety checks – notably the borrow checker! – and also for optimization and code generation. @@ -26,7 +26,7 @@ Some of the key characteristics of MIR are: - It does not have nested expressions. - All types in MIR are fully explicit. -[cfg]: ./appendix/background.html#cfg +[cfg]: ../appendix/background.html#cfg ## Key MIR vocabulary @@ -244,4 +244,4 @@ but [you can read about those below](#promoted)). [mir]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir [mirmanip]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir [mir]: https://github.com/rust-lang/rust/tree/master/src/librustc/mir -[newtype'd]: appendix/glossary.html +[newtype'd]: ../appendix/glossary.html diff --git a/src/mir/passes.md b/src/mir/passes.md index c48ef4576..7dc1249a0 100644 --- a/src/mir/passes.md +++ b/src/mir/passes.md @@ -156,7 +156,7 @@ then `mir_const_qualif(D)` would succeed if it came before `mir_validated(D)`, but fail otherwise. Therefore, `mir_validated(D)` will **force** `mir_const_qualif` before it actually steals, thus ensuring that the reads have already happened (remember that -[queries are memoized](./query.html), so executing a query twice +[queries are memoized](../query.html), so executing a query twice simply loads from a cache the second time): ```text @@ -174,4 +174,4 @@ alternatives in [rust-lang/rust#41710]. [rust-lang/rust#41710]: https://github.com/rust-lang/rust/issues/41710 [mirtransform]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/transform/ [`NoLandingPads`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/transform/no_landing_pads/struct.NoLandingPads.html -[MIR visitor]: mir/visitor.html +[MIR visitor]: ./visitor.html diff --git a/src/tests/intro.md b/src/tests/intro.md index bfe084618..8b2937f9f 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -5,8 +5,8 @@ by the build system (`x.py test`). The main test harness for testing the compiler itself is a tool called compiletest (sources in the [`src/tools/compiletest`]). This section gives a brief overview of how the testing framework is setup, and then gets into some of the details -on [how to run tests](./tests/running.html#ui) as well as -[how to add new tests](./tests/adding.html). +on [how to run tests](./running.html#ui) as well as +[how to add new tests](./adding.html). [`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest @@ -24,7 +24,7 @@ Here is a brief summary of the test suites as of this writing and what they mean. In some cases, the test suites are linked to parts of the manual that give more details. -- [`ui`](./tests/adding.html#ui) – tests that check the exact +- [`ui`](./adding.html#ui) – tests that check the exact stdout/stderr from compilation and/or running the test - `run-pass` – tests that are expected to compile and execute successfully (no panics) @@ -59,7 +59,7 @@ including: - **Tidy** – This is a custom tool used for validating source code style and formatting conventions, such as rejecting long lines. There is more information in the - [section on coding conventions](./conventions.html#formatting). + [section on coding conventions](../conventions.html#formatting). Example: `./x.py test src/tools/tidy` diff --git a/src/traits/associated-types.md b/src/traits/associated-types.md index 1972a70e6..b8ac7f8b9 100644 --- a/src/traits/associated-types.md +++ b/src/traits/associated-types.md @@ -17,7 +17,7 @@ type can be referenced by the user using an **associated type projection** like ` as IntoIterator>::Item`. (Often, though, people will use the shorthand syntax `T::Item` – presently, that syntax is expanded during -["type collection"](./type-checking.html) into the explicit form, +["type collection"](../type-checking.html) into the explicit form, though that is something we may want to change in the future.) [intoiter-item]: https://doc.rust-lang.org/nightly/core/iter/trait.IntoIterator.html#associatedtype.Item @@ -130,7 +130,7 @@ any given associated item. Now we are ready to discuss how associated type equality integrates with unification. As described in the -[type inference](./type-inference.html) section, unification is +[type inference](../type-inference.html) section, unification is basically a procedure with a signature like this: ```text diff --git a/src/traits/caching.md b/src/traits/caching.md index 228ff0917..f84539509 100644 --- a/src/traits/caching.md +++ b/src/traits/caching.md @@ -24,7 +24,7 @@ On the other hand, if there is no hit, we need to go through the [selection process] from scratch. Suppose, we come to the conclusion that the only possible impl is this one, with def-id 22: -[selection process]: ./traits/resolution.html#selection +[selection process]: ./resolution.html#selection ```rust,ignore impl Foo for usize { ... } // Impl #22 @@ -34,7 +34,7 @@ We would then record in the cache `usize : Foo<$0> => ImplCandidate(22)`. Next we would [confirm] `ImplCandidate(22)`, which would (as a side-effect) unify `$t` with `isize`. -[confirm]: ./traits/resolution.html#confirmation +[confirm]: ./resolution.html#confirmation Now, at some later time, we might come along and see a `usize : Foo<$u>`. When skolemized, this would yield `usize : Foo<$0>`, just as @@ -61,7 +61,7 @@ to be pretty clearly safe and also still retains a very high hit rate **TODO**: it looks like `pick_candidate_cache` no longer exists. In general, is this section still accurate at all? -[`ParamEnv`]: ./param_env.html -[`tcx`]: ./ty.html +[`ParamEnv`]: ../param_env.html +[`tcx`]: ../ty.html [#18290]: https://github.com/rust-lang/rust/issues/18290 [#22019]: https://github.com/rust-lang/rust/issues/22019 diff --git a/src/traits/canonical-queries.md b/src/traits/canonical-queries.md index c58fe3696..876c5d8a6 100644 --- a/src/traits/canonical-queries.md +++ b/src/traits/canonical-queries.md @@ -3,11 +3,11 @@ The "start" of the trait system is the **canonical query** (these are both queries in the more general sense of the word – something you would like to know the answer to – and in the -[rustc-specific sense](./query.html)). The idea is that the type +[rustc-specific sense](../query.html)). The idea is that the type checker or other parts of the system, may in the course of doing their thing want to know whether some trait is implemented for some type (e.g., is `u32: Debug` true?). Or they may want to -[normalize some associated type](./traits/associated-types.html). +[normalize some associated type](./associated-types.html). This section covers queries at a fairly high level of abstraction. The subsections look a bit more closely at how these ideas are implemented @@ -106,7 +106,7 @@ value for a type variable, that means that this is the **only possible instantiation** that you could use, given the current set of impls and where-clauses, that would be provable. (Internally within the solver, though, they can potentially enumerate all possible answers. See -[the description of the SLG solver](./traits/slg.html) for details.) +[the description of the SLG solver](./slg.html) for details.) The response to a trait query in rustc is typically a `Result, NoSolution>` (where the `T` will vary a bit @@ -132,7 +132,7 @@ we did find. It consists of four parts: - **Region constraints:** these are relations that must hold between the lifetimes that you supplied as inputs. We'll ignore these here, but see the - [section on handling regions in traits](./traits/regions.html) for + [section on handling regions in traits](./regions.html) for more details. - **Value:** The query result also comes with a value of type `T`. For some specialized queries – like normalizing associated types – @@ -219,7 +219,7 @@ As a result of this assignment, the type of `u` is forced to be `Option>`, where `?V` represents the element type of the vector. This in turn implies that `?U` is [unified] to `Vec`. -[unified]: ./type-checking.html +[unified]: ../type-checking.html Let's suppose that the type checker decides to revisit the "as-yet-unproven" trait obligation we saw before, `Vec: diff --git a/src/traits/canonicalization.md b/src/traits/canonicalization.md index 2cc35d899..df0c4a2b9 100644 --- a/src/traits/canonicalization.md +++ b/src/traits/canonicalization.md @@ -6,7 +6,7 @@ from its context. It is a key part of implementing to get more context. Canonicalization is really based on a very simple concept: every -[inference variable](./type-inference.html#vars) is always in one of +[inference variable](../type-inference.html#vars) is always in one of two states: either it is **unbound**, in which case we don't know yet what type it is, or it is **bound**, in which case we do. So to isolate some data-structure T that contains types/regions from its @@ -16,7 +16,7 @@ starting from zero and numbered in a fixed order (left to right, for the most part, but really it doesn't matter as long as it is consistent). -[cq]: ./traits/canonical-queries.html +[cq]: ./canonical-queries.html So, for example, if we have the type `X = (?T, ?U)`, where `?T` and `?U` are distinct, unbound inference variables, then the canonical @@ -41,7 +41,7 @@ trait query: `?A: Foo<'static, ?B>`, where `?A` and `?B` are unbound. This query contains two unbound variables, but it also contains the lifetime `'static`. The trait system generally ignores all lifetimes and treats them equally, so when canonicalizing, we will *also* -replace any [free lifetime](./appendix/background.html#free-vs-bound) with a +replace any [free lifetime](../appendix/background.html#free-vs-bound) with a canonical variable. Therefore, we get the following result: ```text @@ -98,12 +98,12 @@ Remember that substitution S though! We're going to need it later. OK, now that we have a fresh inference context and an instantiated query, we can go ahead and try to solve it. The trait solver itself is -explained in more detail in [another section](./traits/slg.html), but +explained in more detail in [another section](./slg.html), but suffice to say that it will compute a [certainty value][cqqr] (`Proven` or `Ambiguous`) and have side-effects on the inference variables we've created. For example, if there were only one impl of `Foo`, like so: -[cqqr]: ./traits/canonical-queries.html#query-response +[cqqr]: ./canonical-queries.html#query-response ```rust,ignore impl<'a, X> Foo<'a, X> for Vec diff --git a/src/traits/chalk-overview.md b/src/traits/chalk-overview.md index e737edcd9..818e1dc4d 100644 --- a/src/traits/chalk-overview.md +++ b/src/traits/chalk-overview.md @@ -131,18 +131,18 @@ See [The SLG Solver][slg]. [rustc-issues]: https://github.com/rust-lang-nursery/rustc-guide/issues [chalk]: https://github.com/rust-lang-nursery/chalk -[lowering-to-logic]: traits/lowering-to-logic.html -[lowering-rules]: traits/lowering-rules.html +[lowering-to-logic]: ./lowering-to-logic.html +[lowering-rules]: ./lowering-rules.html [ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree [chalk-ast]: https://github.com/rust-lang-nursery/chalk/blob/master/chalk-parse/src/ast.rs [universal quantification]: https://en.wikipedia.org/wiki/Universal_quantification -[lowering-forall]: ./traits/lowering-to-logic.html#type-checking-generic-functions-beyond-horn-clauses +[lowering-forall]: ./lowering-to-logic.html#type-checking-generic-functions-beyond-horn-clauses [programclause]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L721 [clause]: https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md#clause -[goals-and-clauses]: ./traits/goals-and-clauses.html +[goals-and-clauses]: ./goals-and-clauses.html [well-formedness-checks]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir/lowering.rs#L230-L232 [ir-code]: https://github.com/rust-lang-nursery/chalk/blob/master/src/ir.rs -[HIR]: hir.html +[HIR]: ../hir.html [binders-struct]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L661 [rules-environment]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/rules.rs#L9 -[slg]: ./traits/slg.html +[slg]: ./slg.html diff --git a/src/traits/goals-and-clauses.md b/src/traits/goals-and-clauses.md index 93bafe33f..5aa3755ad 100644 --- a/src/traits/goals-and-clauses.md +++ b/src/traits/goals-and-clauses.md @@ -2,7 +2,7 @@ In logic programming terms, a **goal** is something that you must prove and a **clause** is something that you know is true. As -described in the [lowering to logic](./traits/lowering-to-logic.html) +described in the [lowering to logic](./lowering-to-logic.html) chapter, Rust's trait solver is based on an extension of hereditary harrop (HH) clauses, which extend traditional Prolog Horn clauses with a few new superpowers. @@ -37,7 +37,7 @@ paper ["A Proof Procedure for the Logic of Hereditary Harrop Formulas"][pphhf] gives the details. -[pphhf]: ./traits/bibliography.html#pphhf +[pphhf]: ./bibliography.html#pphhf @@ -94,7 +94,7 @@ e.g. `ProjectionEq::Item = u8` The given associated type `Projection` is equal to `Type`; this can be proved with either normalization or using skolemized types. See [the section -on associated types](./traits/associated-types.html). +on associated types](./associated-types.html). #### Normalize(Projection -> Type) e.g. `ProjectionEq::Item -> u8` @@ -102,12 +102,12 @@ e.g. `ProjectionEq::Item -> u8` The given associated type `Projection` can be [normalized][n] to `Type`. As discussed in [the section on associated -types](./traits/associated-types.html), `Normalize` implies `ProjectionEq`, +types](./associated-types.html), `Normalize` implies `ProjectionEq`, but not vice versa. In general, proving `Normalize(::Item -> U)` also requires proving `Implemented(T: Trait)`. -[n]: ./traits/associated-types.html#normalize -[at]: ./traits/associated-types.html +[n]: ./associated-types.html#normalize +[at]: ./associated-types.html #### FromEnv(TraitRef), FromEnv(Projection = Type) e.g. `FromEnv(Self: Add)` @@ -212,7 +212,7 @@ In addition to auto traits, `WellFormed` predicates are co-inductive. These are used to achieve a similar "enumerate all the cases" pattern, as described in the section on [implied bounds]. -[implied bounds]: ./traits/lowering-rules.html#implied-bounds +[implied bounds]: ./lowering-rules.html#implied-bounds ## Incomplete chapter diff --git a/src/traits/index.md b/src/traits/index.md index c070d1b88..c5afb280b 100644 --- a/src/traits/index.md +++ b/src/traits/index.md @@ -4,7 +4,7 @@ [process of being implemented][wg]; this chapter serves as a kind of in-progress design document. If you would prefer to read about how the current trait solver works, check out -[this other chapter](./traits/resolution.html). (By the way, if you +[this other chapter](./resolution.html). (By the way, if you would like to help in hacking on the new solver, you will find instructions for getting involved in the [Traits Working Group tracking issue][wg].) 🚧 @@ -13,20 +13,20 @@ instructions for getting involved in the Trait solving is based around a few key ideas: -- [Lowering to logic](./traits/lowering-to-logic.html), which expresses +- [Lowering to logic](./lowering-to-logic.html), which expresses Rust traits in terms of standard logical terms. - - The [goals and clauses](./traits/goals-and-clauses.html) chapter + - The [goals and clauses](./goals-and-clauses.html) chapter describes the precise form of rules we use, and - [lowering rules](./traits/lowering-rules.html) gives the complete set of + [lowering rules](./lowering-rules.html) gives the complete set of lowering rules in a more reference-like form. -- [Canonical queries](./traits/canonical-queries.html), which allow us +- [Canonical queries](./canonical-queries.html), which allow us to solve trait problems (like "is `Foo` implemented for the type `Bar`?") once, and then apply that same result independently in many different inference contexts. -- [Lazy normalization](./traits/associated-types.html), which is the +- [Lazy normalization](./associated-types.html), which is the technique we use to accommodate associated types when figuring out whether types are equal. -- [Region constraints](./traits/regions.html), which are accumulated +- [Region constraints](./regions.html), which are accumulated during trait solving but mostly ignored. This means that trait solving effectively ignores the precise regions involved, always – but we still remember the constraints on them so that those diff --git a/src/traits/lowering-module.md b/src/traits/lowering-module.md index 030e19860..c9c4accc1 100644 --- a/src/traits/lowering-module.md +++ b/src/traits/lowering-module.md @@ -1,7 +1,7 @@ # The lowering module in rustc The program clauses described in the -[lowering rules](./traits/lowering-rules.html) section are actually +[lowering rules](./lowering-rules.html) section are actually created in the [`rustc_traits::lowering`][lowering] module. [lowering]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_traits/lowering/ @@ -16,7 +16,7 @@ query is invoked on a `DefId` that identifies something like a trait, an impl, or an associated item definition. It then produces and returns a vector of program clauses. -[query]: ./query.html +[query]: ../query.html diff --git a/src/traits/lowering-rules.md b/src/traits/lowering-rules.md index 41c9625f6..218164c0f 100644 --- a/src/traits/lowering-rules.md +++ b/src/traits/lowering-rules.md @@ -4,8 +4,8 @@ This section gives the complete lowering rules for Rust traits into [program clauses][pc]. It is a kind of reference. These rules reference the [domain goals][dg] defined in an earlier section. -[pc]: ./traits/goals-and-clauses.html -[dg]: ./traits/goals-and-clauses.html#domain-goals +[pc]: ./goals-and-clauses.html +[dg]: ./goals-and-clauses.html#domain-goals ## Notation @@ -16,7 +16,7 @@ The nonterminal `Ai` is used to mean some generic *argument*, which might be a lifetime like `'a` or a type like `Vec`. When defining the lowering rules, we will give goals and clauses in -the [notation given in this section](./traits/goals-and-clauses.html). +the [notation given in this section](./goals-and-clauses.html). We sometimes insert "macros" like `LowerWhereClause!` into these definitions; these macros reference other sections within this chapter. @@ -141,7 +141,7 @@ This `WellFormed` rule states that `T: Trait` is well-formed if (a) `T: Trait` is implemented and (b) all the where-clauses declared on `Trait` are well-formed (and hence they are implemented). Remember that the `WellFormed` predicate is -[coinductive](./traits/goals-and-clauses.html#coinductive); in this +[coinductive](./goals-and-clauses.html#coinductive); in this case, it is serving as a kind of "carrier" that allows us to enumerate all the where clauses that are transitively implied by `T: Trait`. @@ -266,7 +266,7 @@ where WC We will produce a number of program clauses. The first two define the rules by which `ProjectionEq` can succeed; these two clauses are discussed -in detail in the [section on associated types](./traits/associated-types.html), +in detail in the [section on associated types](./associated-types.html), but reproduced here for reference: ```text diff --git a/src/traits/lowering-to-logic.md b/src/traits/lowering-to-logic.md index cf8c1c25a..a0490adf3 100644 --- a/src/traits/lowering-to-logic.md +++ b/src/traits/lowering-to-logic.md @@ -170,8 +170,8 @@ example Gopalan Nadathur's excellent ["A Proof Procedure for the Logic of Hereditary Harrop Formulas"][pphhf] in [the bibliography]. -[the bibliography]: ./traits/bibliography.html -[pphhf]: ./traits/bibliography.html#pphhf +[the bibliography]: ./bibliography.html +[pphhf]: ./bibliography.html#pphhf It turns out that supporting FOHH is not really all that hard. And once we are able to do that, we can easily describe the type-checking diff --git a/src/traits/resolution.md b/src/traits/resolution.md index 987d9a28d..2ba451677 100644 --- a/src/traits/resolution.md +++ b/src/traits/resolution.md @@ -6,7 +6,7 @@ some non-obvious things. **Note:** This chapter (and its subchapters) describe how the trait solver **currently** works. However, we are in the process of designing a new trait solver. If you'd prefer to read about *that*, -see [*this* traits chapter](./traits/index.html). +see [*this* traits chapter](./index.html). ## Major concepts @@ -220,7 +220,7 @@ in that list. If so, it is considered satisfied. More precisely, we want to check whether there is a where-clause obligation that is for the same trait (or some subtrait) and which can match against the obligation. -[parameter environment]: ./param_env.html +[parameter environment]: ../param_env.html Consider this simple example: From ba5c94015df6c8c7d746e0a2121fc303ec0e2b2e Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sat, 8 Sep 2018 21:37:07 -0500 Subject: [PATCH 348/648] use new mdbook-linkcheck version --- ci/install.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install.sh b/ci/install.sh index dc413f68c..eca1b5cdf 100755 --- a/ci/install.sh +++ b/ci/install.sh @@ -21,4 +21,4 @@ function cargo_install() { } cargo_install mdbook 0.2 -cargo_install mdbook-linkcheck 0.1.2 +cargo_install mdbook-linkcheck 0.2.1 From f013b867c5eafab3ebee94da59ffee9cbcade232 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sat, 8 Sep 2018 21:38:07 -0500 Subject: [PATCH 349/648] update readme --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 57b3a0ea4..2a59a40bb 100644 --- a/README.md +++ b/README.md @@ -37,5 +37,6 @@ be found. ```bash > cargo install mdbook-linkcheck ``` -You will need `mdbook` version `>= 0.1`. `linkcheck` will be run automatically -when you run `mdbook build`. We are currently using `mdbook 0.2`. + +You will need `mdbook` version `>= 0.2`. `linkcheck` will be run automatically +when you run `mdbook build`. From 32c6280c71f58f5628d0b0f98aa8b74d9a883f85 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 12 Sep 2018 21:55:06 -0500 Subject: [PATCH 350/648] check web links --- book.toml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/book.toml b/book.toml index ecbe513b7..8c13fa1aa 100644 --- a/book.toml +++ b/book.toml @@ -5,6 +5,7 @@ description = "A guide to developing rustc" [output.html] -[output.linkcheck] - [output.html.search] + +[output.linkcheck] +follow-web-links = true From d36f89182d79055ae268f8bfdcb98cb9995f95c7 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 12 Sep 2018 22:01:44 -0500 Subject: [PATCH 351/648] fix the last link --- src/profiling/with_perf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index 7d8276ced..258899400 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -14,7 +14,7 @@ This is a guide for how to profile rustc with [perf](https://perf.wiki.kernel.or - Make a rustup toolchain pointing to that result - see [the "build and run" section for instructions][b-a-r] -[b-a-r]: ./how-to-build-and-run.html#toolchain +[b-a-r]: ../how-to-build-and-run.html#toolchain ## Gathering a perf profile From ac194d428aa55d4f8289890b08f66b0968b41b9f Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 21 Sep 2018 14:36:11 -0500 Subject: [PATCH 352/648] don't check crates.io links --- book.toml | 1 + 1 file changed, 1 insertion(+) diff --git a/book.toml b/book.toml index 8c13fa1aa..93e40ef50 100644 --- a/book.toml +++ b/book.toml @@ -9,3 +9,4 @@ description = "A guide to developing rustc" [output.linkcheck] follow-web-links = true +exclude = [ "crates\\.io" ] From 9297caaac337c49f2c3903c255171469f37155ed Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 21 Sep 2018 15:09:45 -0500 Subject: [PATCH 353/648] Fix all the links! --- book.toml | 2 +- src/appendix/stupid-stats.md | 17 +++++++++++------ src/const-eval.md | 2 +- src/diag.md | 2 +- src/miri.md | 2 +- src/tests/intro.md | 2 +- src/traits/chalk-overview.md | 2 +- 7 files changed, 17 insertions(+), 12 deletions(-) diff --git a/book.toml b/book.toml index 93e40ef50..b9092a969 100644 --- a/book.toml +++ b/book.toml @@ -9,4 +9,4 @@ description = "A guide to developing rustc" [output.linkcheck] follow-web-links = true -exclude = [ "crates\\.io" ] +exclude = [ "crates\\.io", "gcc\\.godbolt\\.org" ] diff --git a/src/appendix/stupid-stats.md b/src/appendix/stupid-stats.md index 842a2a328..a36cac42b 100644 --- a/src/appendix/stupid-stats.md +++ b/src/appendix/stupid-stats.md @@ -74,7 +74,7 @@ and a bunch of other crates with the 'librustc_' prefix. Next is translation, this translates the AST (and all those side tables) into LLVM IR (intermediate representation). We do this by calling into the LLVM libraries, rather than actually writing IR directly to a file. The code for -this is in [librustc_trans](https://github.com/rust-lang/rust/tree/master/src/librustc_trans). +this is in librustc_trans. The next phase is running the LLVM backend. This runs LLVM's optimisation passes on the generated IR and then generates machine code. The result is object files. @@ -83,17 +83,22 @@ interface between LLVM and rustc is in [librustc_llvm](https://github.com/rust-l Finally, we link the object files into an executable. Again we outsource this to other programs and it's not really part of the rust compiler. The interface is -in [librustc_back](https://github.com/rust-lang/rust/tree/master/src/librustc_back) -(which also contains some things used primarily during translation). +in librustc_back (which also contains some things used primarily during +translation). + +> NOTE: `librustc_trans` and `librustc_back` no longer exist, and we don't +> translate AST or HIR directly to LLVM IR anymore. Instead, see +> [`librustc_codegen_llvm`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_llvm/index.html) +> and [`librustc_codegen_utils`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_utils/index.html). All these phases are coordinated by the driver. To see the exact sequence, look at [the `compile_input` function in `librustc_driver`][compile-input]. -The driver handles all the highest level coordination of compilation - - 1. handling command-line arguments +The driver handles all the highest level coordination of compilation - + 1. handling command-line arguments 2. maintaining compilation state (primarily in the `Session`) 3. calling the appropriate code to run each phase of compilation 4. handles high level coordination of pretty printing and testing. -To create a drop-in compiler replacement or a compiler replacement, +To create a drop-in compiler replacement or a compiler replacement, we leave most of compilation alone and customise the driver using its APIs. [compile-input]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/fn.compile_input.html diff --git a/src/const-eval.md b/src/const-eval.md index 70c946f17..1f801fb22 100644 --- a/src/const-eval.md +++ b/src/const-eval.md @@ -35,4 +35,4 @@ integer or fat pointer, it will directly yield the value (via `Value::ByVal` or memory allocation (via `Value::ByRef`). This means that the `const_eval` function cannot be used to create miri-pointers to the evaluated constant or static. If you need that, you need to directly work with the functions in -[src/librustc_mir/interpret/const_eval.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/interpret/const_eval/). +[src/librustc_mir/const_eval.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/const_eval/index.html). diff --git a/src/diag.md b/src/diag.md index b30ec2eca..936420ab6 100644 --- a/src/diag.md +++ b/src/diag.md @@ -304,5 +304,5 @@ lints we want to emit. Instead, the [`BufferedEarlyLintId`] type is used. If you are defining a new lint, you will want to add an entry to this enum. Then, add an appropriate mapping to the body of [`Lint::from_parser_lint_id`][fplid]. -[`BufferedEarlyLintId`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/early_buffered_lints/struct.BufferedEarlyLintId.html +[`BufferedEarlyLintId`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/early_buffered_lints/enum.BufferedEarlyLintId.html [fplid]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/lint/struct.Lint.html#from_parser_lint_id diff --git a/src/miri.md b/src/miri.md index be9587408..a3c7b3ff4 100644 --- a/src/miri.md +++ b/src/miri.md @@ -112,7 +112,7 @@ to a pointer to `b`. Although the main entry point to constant evaluation is the `tcx.const_eval` query, there are additional functions in -[librustc_mir/interpret/const_eval.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/interpret/const_eval/) +[librustc_mir/const_eval.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/const_eval/index.html) that allow accessing the fields of a `Value` (`ByRef` or otherwise). You should never have to access an `Allocation` directly except for translating it to the compilation target (at the moment just LLVM). diff --git a/src/tests/intro.md b/src/tests/intro.md index 8b2937f9f..4d509f3a8 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -171,7 +171,7 @@ communicate with the server to coordinate running tests (see ## Crater [Crater](https://github.com/rust-lang-nursery/crater) is a tool for compiling -and running tests for _every_ crate on [crates.io](https://crates.io/) (and a +and running tests for _every_ crate on [crates.io](https://crates.io) (and a few on GitHub). It is mainly used for checking for extent of breakage when implementing potentially breaking changes and ensuring lack of breakage by running beta vs stable compiler versions. diff --git a/src/traits/chalk-overview.md b/src/traits/chalk-overview.md index 818e1dc4d..3473a0764 100644 --- a/src/traits/chalk-overview.md +++ b/src/traits/chalk-overview.md @@ -141,7 +141,7 @@ See [The SLG Solver][slg]. [clause]: https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md#clause [goals-and-clauses]: ./goals-and-clauses.html [well-formedness-checks]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir/lowering.rs#L230-L232 -[ir-code]: https://github.com/rust-lang-nursery/chalk/blob/master/src/ir.rs +[ir-code]: https://github.com/rust-lang-nursery/chalk/tree/master/chalk-ir [HIR]: ../hir.html [binders-struct]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L661 [rules-environment]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/rules.rs#L9 From 766a5e354dd6bf5a50fe67dbcadff8814284ea1a Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 21 Sep 2018 15:28:41 -0500 Subject: [PATCH 354/648] use the right version of mdbook linkcheck --- ci/install.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install.sh b/ci/install.sh index eca1b5cdf..80f8de4e8 100755 --- a/ci/install.sh +++ b/ci/install.sh @@ -21,4 +21,4 @@ function cargo_install() { } cargo_install mdbook 0.2 -cargo_install mdbook-linkcheck 0.2.1 +cargo_install mdbook-linkcheck 0.2 From 8871bd77b9169b09e0da7dd598e259b53c79e1a6 Mon Sep 17 00:00:00 2001 From: Tom Tromey Date: Thu, 27 Sep 2018 09:27:30 -0600 Subject: [PATCH 355/648] Document no-system-llvm --- src/tests/adding.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/tests/adding.md b/src/tests/adding.md index b9f05b9dd..1120315b9 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -158,6 +158,7 @@ source. this test; see also `ignore-gdb-version` * `min-lldb-version` specifies the minimum lldb version required for this test +* `no-system-llvm` causes the test to be ignored if the system llvm is used * `min-llvm-version` specifies the minimum llvm version required for this test * `min-system-llvm-version` specifies the minimum system llvm version From 946e11a5787f34cdf26699b31be0badae4a71179 Mon Sep 17 00:00:00 2001 From: Alcaro Date: Sat, 29 Sep 2018 00:03:40 +0200 Subject: [PATCH 356/648] conventions: Fix semicolon/colon typo --- src/conventions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/conventions.md b/src/conventions.md index 93c5c44be..d392ebd57 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -69,7 +69,7 @@ for something that you want to get back to before you land your PR: ```rust,ignore fn do_something() { if something_else { - unimplemented!(): // TODO write this + unimplemented!(); // TODO write this } } ``` From d3b0eaed0114bfc76e72882b46dd0c4035b61977 Mon Sep 17 00:00:00 2001 From: David Sanders Date: Fri, 28 Sep 2018 18:47:52 -0700 Subject: [PATCH 357/648] Typo nits #181 (#205) --- src/hir.md | 2 +- src/lowering.md | 4 ++-- src/traits/canonical-queries.md | 2 +- src/traits/canonicalization.md | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src/hir.md b/src/hir.md index a5e99e8b3..e6bca7f37 100644 --- a/src/hir.md +++ b/src/hir.md @@ -81,7 +81,7 @@ sorts of identifiers in active use: about the crate (such as its version number, as two versions of the same crate can co-exist). - A [`DefId`] really consists of two parts, a `CrateNum` (which - identifies the crate) and a `DefIndex` (which indixes into a list + identifies the crate) and a `DefIndex` (which indexes into a list of items that is maintained per crate). - [`HirId`], which combines the index of a particular item with an offset within that item. diff --git a/src/lowering.md b/src/lowering.md index 2504ed491..c3a1a96cc 100644 --- a/src/lowering.md +++ b/src/lowering.md @@ -25,7 +25,7 @@ sanity checks in `src/librustc/hir/map/hir_id_validator.rs`: any `NodeId`s in the `HIR` are checked for existing `HirId`s) 2. Lowering a `HirId` must be done in the scope of the *owning* item. This means you need to use `with_hir_id_owner` if you are creating parts - of another item than the one being currently lowered. This happens for + of an item other than the one being currently lowered. This happens for example during the lowering of existential `impl Trait` 3. A `NodeId` that will be placed into a HIR structure must be lowered, even if its `HirId` is unused. Calling @@ -45,4 +45,4 @@ generate a new `NodeId` in all those places because you'd also get a new Having the `NodeId` also allows the `DefCollector` to generate the `DefId`s instead of lowering having to do it on the fly. Centralizing the `DefId` -generation in one place makes it easier to refactor and reason about. \ No newline at end of file +generation in one place makes it easier to refactor and reason about. diff --git a/src/traits/canonical-queries.md b/src/traits/canonical-queries.md index 876c5d8a6..cbf7d880d 100644 --- a/src/traits/canonical-queries.md +++ b/src/traits/canonical-queries.md @@ -143,7 +143,7 @@ we did find. It consists of four parts: Let's work through an example query to see what all the parts mean. Consider [the `Borrow` trait][borrow]. This trait has a number of -impls; among them, there are these two (for clarify, I've written the +impls; among them, there are these two (for clarity, I've written the `Sized` bounds explicitly): [borrow]: https://doc.rust-lang.org/std/borrow/trait.Borrow.html diff --git a/src/traits/canonicalization.md b/src/traits/canonicalization.md index df0c4a2b9..62c4d9f74 100644 --- a/src/traits/canonicalization.md +++ b/src/traits/canonicalization.md @@ -29,7 +29,7 @@ they are repeated. We use this to improve caching as well as to detect cycles and other things during trait resolution. Roughly speaking, the idea is that if -two trait queries have the same canonicalize form, then they will get +two trait queries have the same canonical form, then they will get the same answer. That answer will be expressed in terms of the canonical variables (`?0`, `?1`), which we can then map back to the original variables (`?T`, `?U`). From 0f65024f26eab7607ef7f4f51da8354fc15ac5e7 Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Thu, 6 Sep 2018 20:13:36 -0400 Subject: [PATCH 358/648] issue-130 copy contents related x.py from rust-forge --- src/build-install-distribution-artifacts.md | 27 +++++++++++ src/compiler-benchmarking.md | 11 +++++ src/compiler-documenting.md | 49 ++++++++++++++++++++ src/how-to-build-and-run.md | 51 ++++++++++++++++++++- src/tests/running.md | 23 ++++++++++ src/what-is-x-py.md | 42 +++++++++++++++++ 6 files changed, 201 insertions(+), 2 deletions(-) create mode 100644 src/build-install-distribution-artifacts.md create mode 100644 src/compiler-benchmarking.md create mode 100644 src/compiler-documenting.md create mode 100644 src/what-is-x-py.md diff --git a/src/build-install-distribution-artifacts.md b/src/build-install-distribution-artifacts.md new file mode 100644 index 000000000..472c074a3 --- /dev/null +++ b/src/build-install-distribution-artifacts.md @@ -0,0 +1,27 @@ +# Build distribution artifacts + +You might want to build and package up the compiler for distribution. +You’ll want to run this command to do it: + + `./x.py dist` + +Other Flags + +The same flags from build are available here. +You might want to consider adding on the -j flag for faster builds +when building a distribution artifact. + +``` +-j, --jobs JOBS number of jobs to run in parallel +``` + + +# Install distribution artifacts + +If you’ve built a distribution artifact you might want to install it and +test that it works on your target system. You’ll want to run this command: + + `./x.py install` + +Other Flags +The same flags from build are available \ No newline at end of file diff --git a/src/compiler-benchmarking.md b/src/compiler-benchmarking.md new file mode 100644 index 000000000..1ce1e05ac --- /dev/null +++ b/src/compiler-benchmarking.md @@ -0,0 +1,11 @@ +# Benchmarking rustc + +This one is a easier compared to the others. +All you’re doing is running benchmarks of the compiler itself +so it’ll build it and run the one set of benchmarks available to it. +The command is: + + `./x.py bench` + +Benchmarking lacks `--no-fail-fast` flag that `test` command has. + \ No newline at end of file diff --git a/src/compiler-documenting.md b/src/compiler-documenting.md new file mode 100644 index 000000000..919914e98 --- /dev/null +++ b/src/compiler-documenting.md @@ -0,0 +1,49 @@ +# Documenting rustc + +You might want to build documentation of the various components +available like the standard library. There’s two ways to go about this. + You can run rustdoc directly on the file to make sure the HTML is + correct which is fast or you can build the documentation as part of the + build process through x.py. Both are viable methods since documentation + is more about the content. + +## Document everything + + `./x.py doc` + +## If you want to avoid the whole Stage 2 build + + `./x.py doc --stage 1` + +First the compiler and rustdoc get built to make sure everything is okay +and then it documents the files. + +## Document specific components + +```bash + ./x.py doc src/doc/book + ./x.py doc src/doc/nomicon + ./x.py doc src/doc/book src/libstd +``` + +Much like individual tests or building certain components you can build only + the documentation you want. + +## Document internal rustc items +By default rustc does not build the compiler docs for its internal items. +Mostly because this is useless for the average user. However, you might need +to have it available so you can understand the types. Here’s how you can +compile it yourself. From the top level directory where x.py is located run: + + cp config.toml.example config.toml + +Next open up config.toml and make sure these two lines are set to true: + +docs = true +compiler-docs = true +When you want to build the compiler docs as well run this command: + + `./x.py doc` + +This will see that the docs and compiler-docs options are set to true +and build the normally hidden compiler docs! \ No newline at end of file diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 66ba8efda..07a5eeb4d 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -134,6 +134,44 @@ build`) has quite a few more steps: +### Build different stages + + `./x.py build --stage 0` + + # Stage 1 is typically enough to test out all of your changes + # to the compiler + + `./x.py build --stage 1` + + # Equivalent to ./x.py build + + `./x.py build --stage 2` + +You can pass the --stage flag with what stage you want to build to. +It is recommended that you build to Stage 1 as this is enough to know +your changes can successfully compile and should let you run tests +with your changes. + +### Build specific components + + Build only the libcore library + + `./x.py build src/libcore` + + Build the libcore and libproc_macro library only + + `./x.py build src/libcore src/libproc_macro` + + Build only libcore up to Stage 1 + + `./x.py build src/libcore --stage 1` + +Sometimes you might just want to test if the part you’re working on can +compile. Using these commands you can test that it compiles before doing +a bigger build to make sure it works with the compiler. As shown before +you can also pass flags at the end such as --stage. + + ### Creating a rustup toolchain Once you have successfully built rustc, you will have created a bunch @@ -145,8 +183,8 @@ you will likely need to build at some point; for example, if you want to run the entire test suite). ```bash -> rustup toolchain link stage1 build//stage1 -> rustup toolchain link stage2 build//stage2 + rustup toolchain link stage1 build//stage1 + rustup toolchain link stage2 build//stage2 ``` The `` would typically be one of the following: @@ -263,3 +301,12 @@ This allows you to do "jump-to-def" with whatever functions were around when you last built, which is ridiculously useful. [etags]: https://github.com/nikomatsakis/rust-etags + +### Cleaning out build directories + +Sometimes you need to start fresh, but this is normally not the case. +If you need to run this then rustbuild is most likely not acting right and +you should file a bug as to what is going wrong. If you do need to clean +everything up then you only need to run one command! + + `./x.py clean` \ No newline at end of file diff --git a/src/tests/running.md b/src/tests/running.md index 778fd00ec..970829cce 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -93,3 +93,26 @@ This is much faster, but doesn't always work. For example, some tests include directives that specify specific compiler flags, or which rely on other crates, and they may not run the same without those options. +### Run specific tests + +# Run only the tidy script +```bash +> ./x.py test src/tools/tidy +``` +# Run tests on the standard library +```bash +> ./x.py test src/libstd +``` + +# Run tests on the standard library and run the tidy script +```bash +> ./x.py test src/libstd src/tools/tidy +``` + +# Run tests on the standard library using a stage 1 compiler +```bash +> ./x.py test src/libstd --stage 1 +``` + +By listing which test suites you want to run you avoid having to run tests for +components you did not change at all. diff --git a/src/what-is-x-py.md b/src/what-is-x-py.md new file mode 100644 index 000000000..315a3f821 --- /dev/null +++ b/src/what-is-x-py.md @@ -0,0 +1,42 @@ +# what is x.py? + +x.py is the script used to orchestrate the tooling in the rustc repository. +It is the script that can build docs, run tests, and compile rustc. +It is the now preferred way to build rustc and it replaces the old makefiles +from before. Below are the different ways to utilize x.py in order to +effectively deal with the repo for various common tasks. + +### Build Flags + +There are other flags you can pass to the build portion of x.py that can be +beneficial to cutting down compile times or fitting other things you might +need to change. They are: + +``` +Options: + -v, --verbose use verbose output (-vv for very verbose) + -i, --incremental use incremental compilation + --config FILE TOML configuration file for build + --build BUILD build target of the stage0 compiler + --host HOST host targets to build + --target TARGET target targets to build + --on-fail CMD command to run on failure + --stage N stage to build + --keep-stage N stage to keep without recompiling + --src DIR path to the root of the rust checkout + -j, --jobs JOBS number of jobs to run in parallel + -h, --help print this help message +``` + +Note that the options --incremental, --keep-stage 0 and --jobs JOBS can be +used in tandem with --stage to help reduce build times significantly by +reusing already built components, reusing the first bootstrapped stage, and +running compilation in parallel. To test changes you could run something like: + +```bash + ./x.py build --stage 1 --keep-stage 0 -j 4 -i +``` + +Please follow the links to build, document, test, benchmark and install +distribution + artifacts for rustc respectively. From 062e158a676fa751d4c42b27b2485e68c169df60 Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Tue, 18 Sep 2018 17:29:59 -0400 Subject: [PATCH 359/648] issue-130 incorporate the review comments --- src/SUMMARY.md | 2 + src/build-install-distribution-artifacts.md | 15 ++-- src/compiler-documenting.md | 25 ++++-- src/how-to-build-and-run.md | 92 +++++++++++++++------ src/tests/running.md | 10 +-- src/what-is-x-py.md | 42 ---------- 6 files changed, 100 insertions(+), 86 deletions(-) delete mode 100644 src/what-is-x-py.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 338cb7fe1..213a645ab 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -3,6 +3,8 @@ - [About this guide](./about-this-guide.md) - [About the compiler team](./compiler-team.md) - [How to build the compiler and run what you built](./how-to-build-and-run.md) + - [Build and Install distribution artifacts](./build-install-distribution-artifacts.md) + - [Documenting Compiler](./compiler-documenting.md) - [Coding conventions](./conventions.md) - [Walkthrough: a typical contribution](./walkthrough.md) - [The compiler testing framework](./tests/intro.md) diff --git a/src/build-install-distribution-artifacts.md b/src/build-install-distribution-artifacts.md index 472c074a3..89b1422a0 100644 --- a/src/build-install-distribution-artifacts.md +++ b/src/build-install-distribution-artifacts.md @@ -3,15 +3,17 @@ You might want to build and package up the compiler for distribution. You’ll want to run this command to do it: - `./x.py dist` + ```bash + ./x.py dist + ``` -Other Flags +## Other Flags The same flags from build are available here. You might want to consider adding on the -j flag for faster builds when building a distribution artifact. -``` +```bash -j, --jobs JOBS number of jobs to run in parallel ``` @@ -21,7 +23,6 @@ when building a distribution artifact. If you’ve built a distribution artifact you might want to install it and test that it works on your target system. You’ll want to run this command: - `./x.py install` - -Other Flags -The same flags from build are available \ No newline at end of file + ```bash + ./x.py install + ``` \ No newline at end of file diff --git a/src/compiler-documenting.md b/src/compiler-documenting.md index 919914e98..61d65f376 100644 --- a/src/compiler-documenting.md +++ b/src/compiler-documenting.md @@ -3,17 +3,21 @@ You might want to build documentation of the various components available like the standard library. There’s two ways to go about this. You can run rustdoc directly on the file to make sure the HTML is - correct which is fast or you can build the documentation as part of the - build process through x.py. Both are viable methods since documentation - is more about the content. + correct, which is fast. Alternatively, you can build the documentation + as part of the build process through x.py. Both are viable methods + since documentation is more about the content. ## Document everything - `./x.py doc` + ```bash + ./x.py doc + ``` ## If you want to avoid the whole Stage 2 build - `./x.py doc --stage 1` +```bash +./x.py doc --stage 1 +``` First the compiler and rustdoc get built to make sure everything is okay and then it documents the files. @@ -35,15 +39,22 @@ Mostly because this is useless for the average user. However, you might need to have it available so you can understand the types. Here’s how you can compile it yourself. From the top level directory where x.py is located run: - cp config.toml.example config.toml +```bash +cp config.toml.example config.toml +``` Next open up config.toml and make sure these two lines are set to true: +```bash docs = true compiler-docs = true +``` + When you want to build the compiler docs as well run this command: - `./x.py doc` +```bash +./x.py doc +``` This will see that the docs and compiler-docs options are set to true and build the normally hidden compiler docs! \ No newline at end of file diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 07a5eeb4d..c11d269bd 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -47,6 +47,14 @@ debuginfo-lines = true use-jemalloc = false ``` +### what is x.py? + +x.py is the script used to orchestrate the tooling in the rustc repository. +It is the script that can build docs, run tests, and compile rustc. +It is the now preferred way to build rustc and it replaces the old makefiles +from before. Below are the different ways to utilize x.py in order to +effectively deal with the repo for various common tasks. + ### Running x.py and building a stage1 compiler One thing to keep in mind is that `rustc` is a _bootstrapping_ @@ -80,6 +88,52 @@ compiling `rustc` is done in stages: can build the libraries with the stage2 compiler. The result ought to be identical to before, unless something has broken. + +#### Build Flags + +There are other flags you can pass to the build portion of x.py that can be +beneficial to cutting down compile times or fitting other things you might +need to change. They are: + +```bash +Options: + -v, --verbose use verbose output (-vv for very verbose) + -i, --incremental use incremental compilation + --config FILE TOML configuration file for build + --build BUILD build target of the stage0 compiler + --host HOST host targets to build + --target TARGET target targets to build + --on-fail CMD command to run on failure + --stage N stage to build + --keep-stage N stage to keep without recompiling + --src DIR path to the root of the rust checkout + -j, --jobs JOBS number of jobs to run in parallel + -h, --help print this help message +``` + +One thing to keep in mind is that `rustc` is a _bootstrapping_ compiler. That +is, since `rustc` is written in Rust, we need to use an older version of the +compiler to compile the newer version. In particular, the newer version of the +compiler, `libstd`, and other tooling may use some unstable features +internally. The result is the compiling `rustc` is done in stages. + +- **Stage 0:** the stage0 compiler can be your existing + (perhaps older version of) + Rust compiler, the current _beta_ compiler or you may download the binary + from the internet. +- **Stage 1:** the code in your clone (for new version) + is then compiled with the stage0 + compiler to produce the stage1 compiler. + However, it was built with an older compiler (stage0), + so to optimize the stage1 compiler we go to next stage. +- **Stage 2:** we rebuild our stage1 compiler with itself + to produce the stage2 compiler (i.e. it builds + itself) to have all the _latest optimizations_. +- _(Optional)_ **Stage 3**: to sanity check of our new compiler, + we can build it again + with stage2 compiler which must be identical to itself, + unless something has broken. + For hacking, often building the stage 1 compiler is enough, but for final testing and release, the stage 2 compiler is used. @@ -134,37 +188,25 @@ build`) has quite a few more steps: -### Build different stages - - `./x.py build --stage 0` - - # Stage 1 is typically enough to test out all of your changes - # to the compiler - - `./x.py build --stage 1` - - # Equivalent to ./x.py build - - `./x.py build --stage 2` - -You can pass the --stage flag with what stage you want to build to. -It is recommended that you build to Stage 1 as this is enough to know -your changes can successfully compile and should let you run tests -with your changes. - ### Build specific components Build only the libcore library - `./x.py build src/libcore` +```bash +> ./x.py build src/libcore +``` Build the libcore and libproc_macro library only - `./x.py build src/libcore src/libproc_macro` +```bash +> ./x.py build src/libcore src/libproc_macro +``` Build only libcore up to Stage 1 - `./x.py build src/libcore --stage 1` +```bash +> ./x.py build src/libcore --stage 1 +``` Sometimes you might just want to test if the part you’re working on can compile. Using these commands you can test that it compiles before doing @@ -183,8 +225,8 @@ you will likely need to build at some point; for example, if you want to run the entire test suite). ```bash - rustup toolchain link stage1 build//stage1 - rustup toolchain link stage2 build//stage2 +> rustup toolchain link stage1 build//stage1 +> rustup toolchain link stage2 build//stage2 ``` The `` would typically be one of the following: @@ -309,4 +351,6 @@ If you need to run this then rustbuild is most likely not acting right and you should file a bug as to what is going wrong. If you do need to clean everything up then you only need to run one command! - `./x.py clean` \ No newline at end of file + ```bash + > ./x.py clean + ``` diff --git a/src/tests/running.md b/src/tests/running.md index 970829cce..07030c799 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -93,23 +93,21 @@ This is much faster, but doesn't always work. For example, some tests include directives that specify specific compiler flags, or which rely on other crates, and they may not run the same without those options. -### Run specific tests - -# Run only the tidy script +### Run only the tidy script ```bash > ./x.py test src/tools/tidy ``` -# Run tests on the standard library +### Run tests on the standard library ```bash > ./x.py test src/libstd ``` -# Run tests on the standard library and run the tidy script +### Run tests on the standard library and run the tidy script ```bash > ./x.py test src/libstd src/tools/tidy ``` -# Run tests on the standard library using a stage 1 compiler +### Run tests on the standard library using a stage 1 compiler ```bash > ./x.py test src/libstd --stage 1 ``` diff --git a/src/what-is-x-py.md b/src/what-is-x-py.md deleted file mode 100644 index 315a3f821..000000000 --- a/src/what-is-x-py.md +++ /dev/null @@ -1,42 +0,0 @@ -# what is x.py? - -x.py is the script used to orchestrate the tooling in the rustc repository. -It is the script that can build docs, run tests, and compile rustc. -It is the now preferred way to build rustc and it replaces the old makefiles -from before. Below are the different ways to utilize x.py in order to -effectively deal with the repo for various common tasks. - -### Build Flags - -There are other flags you can pass to the build portion of x.py that can be -beneficial to cutting down compile times or fitting other things you might -need to change. They are: - -``` -Options: - -v, --verbose use verbose output (-vv for very verbose) - -i, --incremental use incremental compilation - --config FILE TOML configuration file for build - --build BUILD build target of the stage0 compiler - --host HOST host targets to build - --target TARGET target targets to build - --on-fail CMD command to run on failure - --stage N stage to build - --keep-stage N stage to keep without recompiling - --src DIR path to the root of the rust checkout - -j, --jobs JOBS number of jobs to run in parallel - -h, --help print this help message -``` - -Note that the options --incremental, --keep-stage 0 and --jobs JOBS can be -used in tandem with --stage to help reduce build times significantly by -reusing already built components, reusing the first bootstrapped stage, and -running compilation in parallel. To test changes you could run something like: - -```bash - ./x.py build --stage 1 --keep-stage 0 -j 4 -i -``` - -Please follow the links to build, document, test, benchmark and install -distribution - artifacts for rustc respectively. From 61cadd2097a40a6ade9b75e1b09460d9819ee24b Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Wed, 26 Sep 2018 10:32:50 -0400 Subject: [PATCH 360/648] issue-130 updated the review comments --- src/build-install-distribution-artifacts.md | 20 +++------ src/compiler-benchmarking.md | 11 ----- src/compiler-documenting.md | 9 +--- src/how-to-build-and-run.md | 26 +---------- src/tests/running.md | 49 ++++++++++++--------- 5 files changed, 37 insertions(+), 78 deletions(-) delete mode 100644 src/compiler-benchmarking.md diff --git a/src/build-install-distribution-artifacts.md b/src/build-install-distribution-artifacts.md index 89b1422a0..521c441a2 100644 --- a/src/build-install-distribution-artifacts.md +++ b/src/build-install-distribution-artifacts.md @@ -7,22 +7,16 @@ You’ll want to run this command to do it: ./x.py dist ``` -## Other Flags - -The same flags from build are available here. -You might want to consider adding on the -j flag for faster builds -when building a distribution artifact. - -```bash --j, --jobs JOBS number of jobs to run in parallel -``` - - # Install distribution artifacts -If you’ve built a distribution artifact you might want to install it and +If you’ve built a distribution artifact you might want to install it and test that it works on your target system. You’ll want to run this command: ```bash ./x.py install - ``` \ No newline at end of file + ``` + + Note: If you are testing out a modification to a compiler, you might want to use it to compile some project. + Usually, you do not want to use ./x.py install for testing. + Rather, you should create a toolchain as discussed in how-to-build-and-run.html#creating-a-rustup-toolchain. + For example, if the toolchain you created is called foo, you would then invoke it with rustc +foo ... (where ... represents the rest of the arguments). \ No newline at end of file diff --git a/src/compiler-benchmarking.md b/src/compiler-benchmarking.md deleted file mode 100644 index 1ce1e05ac..000000000 --- a/src/compiler-benchmarking.md +++ /dev/null @@ -1,11 +0,0 @@ -# Benchmarking rustc - -This one is a easier compared to the others. -All you’re doing is running benchmarks of the compiler itself -so it’ll build it and run the one set of benchmarks available to it. -The command is: - - `./x.py bench` - -Benchmarking lacks `--no-fail-fast` flag that `test` command has. - \ No newline at end of file diff --git a/src/compiler-documenting.md b/src/compiler-documenting.md index 61d65f376..fceb73c91 100644 --- a/src/compiler-documenting.md +++ b/src/compiler-documenting.md @@ -34,14 +34,9 @@ Much like individual tests or building certain components you can build only the documentation you want. ## Document internal rustc items -By default rustc does not build the compiler docs for its internal items. -Mostly because this is useless for the average user. However, you might need -to have it available so you can understand the types. Here’s how you can -compile it yourself. From the top level directory where x.py is located run: -```bash -cp config.toml.example config.toml -``` +Compiler documentation is not built by default - there's a flag in config.toml for achieving the same. +But, when enabled, compiler documentation does include internal items. Next open up config.toml and make sure these two lines are set to true: diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index c11d269bd..eb2f8a663 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -47,7 +47,7 @@ debuginfo-lines = true use-jemalloc = false ``` -### what is x.py? +### What is x.py? x.py is the script used to orchestrate the tooling in the rustc repository. It is the script that can build docs, run tests, and compile rustc. @@ -88,7 +88,6 @@ compiling `rustc` is done in stages: can build the libraries with the stage2 compiler. The result ought to be identical to before, unless something has broken. - #### Build Flags There are other flags you can pass to the build portion of x.py that can be @@ -111,29 +110,6 @@ Options: -h, --help print this help message ``` -One thing to keep in mind is that `rustc` is a _bootstrapping_ compiler. That -is, since `rustc` is written in Rust, we need to use an older version of the -compiler to compile the newer version. In particular, the newer version of the -compiler, `libstd`, and other tooling may use some unstable features -internally. The result is the compiling `rustc` is done in stages. - -- **Stage 0:** the stage0 compiler can be your existing - (perhaps older version of) - Rust compiler, the current _beta_ compiler or you may download the binary - from the internet. -- **Stage 1:** the code in your clone (for new version) - is then compiled with the stage0 - compiler to produce the stage1 compiler. - However, it was built with an older compiler (stage0), - so to optimize the stage1 compiler we go to next stage. -- **Stage 2:** we rebuild our stage1 compiler with itself - to produce the stage2 compiler (i.e. it builds - itself) to have all the _latest optimizations_. -- _(Optional)_ **Stage 3**: to sanity check of our new compiler, - we can build it again - with stage2 compiler which must be identical to itself, - unless something has broken. - For hacking, often building the stage 1 compiler is enough, but for final testing and release, the stage 2 compiler is used. diff --git a/src/tests/running.md b/src/tests/running.md index 07030c799..f8889c8a4 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -40,6 +40,33 @@ the debuginfo test suite: > ./x.py test --stage 1 src/test/debuginfo ``` +### Run only the tidy script + +```bash +> ./x.py test src/tools/tidy +``` + +### Run tests on the standard library + +```bash +> ./x.py test src/libstd +``` + +### Run tests on the standard library and run the tidy script + +```bash +> ./x.py test src/libstd src/tools/tidy +``` + +### Run tests on the standard library using a stage 1 compiler + +```bash +> ./x.py test src/libstd --stage 1 +``` + +By listing which test suites you want to run you avoid having to run +tests for components you did not change at all. + **Warning:** Note that bors only runs the tests with the full stage 2 build; therefore, while the tests **usually** work fine with stage 1, there are some limitations. In particular, the stage1 compiler doesn't @@ -92,25 +119,3 @@ just `rs` files, so you can do something like This is much faster, but doesn't always work. For example, some tests include directives that specify specific compiler flags, or which rely on other crates, and they may not run the same without those options. - -### Run only the tidy script -```bash -> ./x.py test src/tools/tidy -``` -### Run tests on the standard library -```bash -> ./x.py test src/libstd -``` - -### Run tests on the standard library and run the tidy script -```bash -> ./x.py test src/libstd src/tools/tidy -``` - -### Run tests on the standard library using a stage 1 compiler -```bash -> ./x.py test src/libstd --stage 1 -``` - -By listing which test suites you want to run you avoid having to run tests for -components you did not change at all. From ceff08f6b3f9666489be7ecd417b066e117aa474 Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Fri, 28 Sep 2018 23:01:47 -0400 Subject: [PATCH 361/648] updated with review comments --- src/build-install-distribution-artifacts.md | 13 ++++++++++--- src/compiler-documenting.md | 3 ++- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/src/build-install-distribution-artifacts.md b/src/build-install-distribution-artifacts.md index 521c441a2..7430ffb9b 100644 --- a/src/build-install-distribution-artifacts.md +++ b/src/build-install-distribution-artifacts.md @@ -16,7 +16,14 @@ test that it works on your target system. You’ll want to run this command: ./x.py install ``` - Note: If you are testing out a modification to a compiler, you might want to use it to compile some project. + Note: If you are testing out a modification to a compiler, you + might want to use it to compile some project. Usually, you do not want to use ./x.py install for testing. - Rather, you should create a toolchain as discussed in how-to-build-and-run.html#creating-a-rustup-toolchain. - For example, if the toolchain you created is called foo, you would then invoke it with rustc +foo ... (where ... represents the rest of the arguments). \ No newline at end of file + Rather, you should create a toolchain as discussed in + [here][create-rustup-toolchain]. + + For example, if the toolchain you created is called foo, you + would then invoke it with `rustc +foo ...` (where ... represents + the rest of the arguments). + +[create-rustup-toolchain]: ./how-to-build-and-run.md#creating-a-rustup-toolchain \ No newline at end of file diff --git a/src/compiler-documenting.md b/src/compiler-documenting.md index fceb73c91..bf63c0120 100644 --- a/src/compiler-documenting.md +++ b/src/compiler-documenting.md @@ -35,7 +35,8 @@ Much like individual tests or building certain components you can build only ## Document internal rustc items -Compiler documentation is not built by default - there's a flag in config.toml for achieving the same. +Compiler documentation is not built by default. There's a flag in +config.toml for achieving the same. But, when enabled, compiler documentation does include internal items. Next open up config.toml and make sure these two lines are set to true: From 7cf032f9ec4506cdb821d9b4023490feac6c22bb Mon Sep 17 00:00:00 2001 From: mark Date: Sat, 13 Oct 2018 11:35:53 -0500 Subject: [PATCH 362/648] Update a few parts of the readme --- README.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 2a59a40bb..45c7503fe 100644 --- a/README.md +++ b/README.md @@ -10,18 +10,15 @@ You may also find the rustdocs [for the compiler itself][rustdocs] useful. [rustdocs]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ The guide can be useful today, but it has a lot of work still go. -Once it gets more complete, the plan is probably to move it into the -[main Rust repository](https://github.com/rust-lang/rust/). ### Contributing to the guide -If you'd like to help finish the guide, we'd love to have you! The -main tracking issue for the guide -[can be found here](https://github.com/rust-lang-nursery/rustc-guide/issues/6). From -there, you can find a list of all the planned chapters and subsections. -If you think something is missing, please open an issue about it! -Otherwise, find a chapter that sounds interesting to you and then go -to its associated issue. There should be a list of things to do. +If you'd like to help improve the guide, we'd love to have you! You can find +plenty of issues on the [issue +tracker](https://github.com/rust-lang-nursery/rustc-guide/issue). Just post a +comment on the issue you would like to work on to make sure that we don't +accidentally duplicate work. If you think something is missing, please open an +issue about it! **In general, if you don't know how the compiler works, that is not a problem!** In that case, what we will do is to schedule a bit of time @@ -29,6 +26,10 @@ for you to talk with someone who **does** know the code, or who wants to pair with you and figure it out. Then you can work on writing up what you learned. +In general, when writing about a particular part of the compiler's code, we +recommend that you link to the relevant parts of the [rustc +rustdocs][rustdocs]. + To help prevent accidentally introducing broken links, we use the `mdbook-linkcheck`. If installed on your machine `mdbook` will automatically invoke this link checker, otherwise it will emit a warning saying it couldn't From a01aa11b7be1df9b4ce7a4c98ef41d60ea40628b Mon Sep 17 00:00:00 2001 From: Collins Abitekaniza Date: Sat, 20 Oct 2018 01:19:52 +0300 Subject: [PATCH 363/648] fix typo in build instructions --- src/how-to-build-and-run.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index eb2f8a663..4315cd708 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -230,7 +230,7 @@ LLVM version: 4.0 ### Suggested workflows for faster builds of the compiler -There are two workflows that are useful for faster builders of the +There are two workflows that are useful for faster builds of the compiler. **Check, check, and check again.** The first workflow, which is useful From 4aeae78bdb078e10aee219938788ec9e0ed7936a Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Fri, 19 Oct 2018 16:19:27 -0500 Subject: [PATCH 364/648] Fix typo --- src/borrow_check/moves_and_initialization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/moves_and_initialization.md b/src/borrow_check/moves_and_initialization.md index d1cd41e0f..d1530d6c0 100644 --- a/src/borrow_check/moves_and_initialization.md +++ b/src/borrow_check/moves_and_initialization.md @@ -44,7 +44,7 @@ moves `a` into the call, and hence it becomes uninitialized again. To make it easier to peruse, this section is broken into a number of subsections: -- [Move paths](./moves_and_initialization/move_paths.html the +- [Move paths](./moves_and_initialization/move_paths.html) the *move path* concept that we use to track which local variables (or parts of local variables, in some cases) are initialized. - TODO *Rest not yet written* =) From 3db38fe7f30c1b8bde710495d38fe74dc1b870fa Mon Sep 17 00:00:00 2001 From: scalexm Date: Tue, 16 Oct 2018 14:55:03 +0200 Subject: [PATCH 365/648] Polish lowering chapters and update rules --- src/traits/associated-types.md | 21 ++++++--- src/traits/goals-and-clauses.md | 84 ++++++++++++++++++++++++++------- src/traits/index.md | 14 ++++++ src/traits/lowering-rules.md | 36 ++++++++------ 4 files changed, 116 insertions(+), 39 deletions(-) diff --git a/src/traits/associated-types.md b/src/traits/associated-types.md index b8ac7f8b9..3330ce809 100644 --- a/src/traits/associated-types.md +++ b/src/traits/associated-types.md @@ -67,7 +67,7 @@ type.) We could apply that rule to normalize either of the examples that we've seen so far. -## Skolemized associated types +## Placeholder associated types Sometimes however we want to work with associated types that cannot be normalized. For example, consider this function: @@ -78,20 +78,29 @@ fn foo(...) { ... } In this context, how would we normalize the type `T::Item`? Without knowing what `T` is, we can't really do so. To represent this case, we -introduce a type called a **skolemized associated type +introduce a type called a **placeholder associated type projection**. This is written like so `(IntoIterator::Item)`. You may note that it looks a lot like a regular type (e.g., `Option`), except that the "name" of the type is `(IntoIterator::Item)`. This is -not an accident: skolemized associated type projections work just like +not an accident: placeholder associated type projections work just like ordinary types like `Vec` when it comes to unification. That is, they are only considered equal if (a) they are both references to the same associated type, like `IntoIterator::Item` and (b) their type arguments are equal. -Skolemized associated types are never written directly by the user. +Placeholder associated types are never written directly by the user. They are used internally by the trait system only, as we will see shortly. +In rustc, they correspond to the `TyKind::UnnormalizedProjectionTy` enum +variant, declared in [`librustc/ty/sty.rs`][sty]. In chalk, we use an +`ApplicationTy` with a name living in a special namespace dedicated to +placeholder associated types (see the `TypeName` enum declared in +[`chalk-ir/src/lib.rs`][chalk_type_name]). + +[sty]: https://github.com/rust-lang/rust/blob/master/src/librustc/ty/sty.rs +[chalk_type_name]: https://github.com/rust-lang-nursery/chalk/blob/master/chalk-ir/src/lib.rs + ## Projection equality So far we have seen two ways to answer the question of "When can we @@ -99,7 +108,7 @@ consider an associated type projection equal to another type?": - the `Normalize` predicate could be used to transform associated type projections when we knew which impl was applicable; -- **skolemized** associated types can be used when we don't. +- **placeholder** associated types can be used when we don't. We now introduce the `ProjectionEq` predicate to bring those two cases together. The `ProjectionEq` predicate looks like so: @@ -109,7 +118,7 @@ ProjectionEq(::Item = U) ``` and we will see that it can be proven *either* via normalization or -skolemization. As part of lowering an associated type declaration from +via the placeholder type. As part of lowering an associated type declaration from some trait, we create two program clauses for `ProjectionEq`: ```text diff --git a/src/traits/goals-and-clauses.md b/src/traits/goals-and-clauses.md index 5aa3755ad..8cc1bc7fd 100644 --- a/src/traits/goals-and-clauses.md +++ b/src/traits/goals-and-clauses.md @@ -37,15 +37,33 @@ paper ["A Proof Procedure for the Logic of Hereditary Harrop Formulas"][pphhf] gives the details. +In terms of code, these types are defined in +[`librustc/traits/mod.rs`][traits_mod] in rustc, and in +[`chalk-ir/src/lib.rs`][chalk_ir] in chalk. + [pphhf]: ./bibliography.html#pphhf +[traits_mod]: https://github.com/rust-lang/rust/blob/master/src/librustc/traits/mod.rs +[chalk_ir]: https://github.com/rust-lang-nursery/chalk/blob/master/chalk-ir/src/lib.rs ## Domain goals +*Domain goals* are the atoms of the trait logic. As can be seen in the +definitions given above, general goals basically consist in a combination of +domain goals. + +Moreover, flattenning a bit the definition of clauses given previously, one can +see that clauses are always of the form: +```text +forall { DomainGoal :- Goal } +``` +hence domain goals are in fact clauses LHS. That is, at the most granular level, +domain goals are what the trait solver will end up trying to prove. + -To define the set of *domain goals* in our system, we need to first +To define the set of domain goals in our system, we need to first introduce a few simple formulations. A **trait reference** consists of the name of a trait along with a suitable set of inputs P0..Pn: @@ -70,18 +88,24 @@ Projection = >::AssocItem Given these, we can define a `DomainGoal` as follows: ```text -DomainGoal = Implemented(TraitRef) - | ProjectionEq(Projection = Type) - | Normalize(Projection -> Type) +DomainGoal = Holds(WhereClause) | FromEnv(TraitRef) - | FromEnv(Projection = Type) - | WellFormed(Type) + | FromEnv(Type) | WellFormed(TraitRef) - | WellFormed(Projection = Type) + | WellFormed(Type) + | Normalize(Projection -> Type) + +WhereClause = Implemented(TraitRef) + | ProjectionEq(Projection = Type) | Outlives(Type: Region) | Outlives(Region: Region) ``` +`WhereClause` refers to a `where` clause that a Rust user would actually be able +to write in a Rust program. This abstraction exists only as a convenience as we +sometimes want to only coope with domain goals that are effectively writable in +Rust. + Let's break down each one of these, one-by-one. #### Implemented(TraitRef) @@ -109,12 +133,10 @@ also requires proving `Implemented(T: Trait)`. [n]: ./associated-types.html#normalize [at]: ./associated-types.html -#### FromEnv(TraitRef), FromEnv(Projection = Type) +#### FromEnv(TraitRef) e.g. `FromEnv(Self: Add)` -e.g. `FromEnv(::Item<'a> = &'a [u8])` - -True if the inner `TraitRef` or projection equality is *assumed* to be true; +True if the inner `TraitRef` is *assumed* to be true, that is, if it can be derived from the in-scope where clauses. For example, given the following function: @@ -131,24 +153,50 @@ where clauses nest, so a function body inside an impl body inherits the impl body's where clauses, too. This and the next rule are used to implement [implied bounds]. As we'll see -in the section on lowering, `FromEnv(X)` implies `Implemented(X)`, but not -vice versa. This distinction is crucial to implied bounds. +in the section on lowering, `FromEnv(TraitRef)` implies `Implemented(TraitRef)`, +but not vice versa. This distinction is crucial to implied bounds. + +#### FromEnv(Type) +e.g. `FromEnv(HashSet)` + +True if the inner `Type` is *assumed* to be well-formed, that is, if it is an +input type of a function or an impl. + +For example, given the following code: + +```rust,ignore +struct HashSet where K: Hash { ... } + +fn loud_insert(set: &mut HashSet, item: K) { + println!("inserting!"); + set.insert(item); +} +``` + +`HashSet` is an input type of the `loud_insert` function. Hence, we assume it +to be well-formed, so we would have `FromEnv(HashSet)` inside the body or our +function. As we'll see in the section on lowering, `FromEnv(HashSet)` implies +`Implemented(K: Hash)` because the +`HashSet` declaration was written with a `K: Hash` where clause. Hence, we don't +need to repeat that bound on the `loud_insert` function: we rather automatically +assume that it is true. #### WellFormed(Item) These goals imply that the given item is *well-formed*. We can talk about different types of items being well-formed: -**Types**, like `WellFormed(Vec)`, which is true in Rust, or +* *Types*, like `WellFormed(Vec)`, which is true in Rust, or `WellFormed(Vec)`, which is not (because `str` is not `Sized`.) -**TraitRefs**, like `WellFormed(Vec: Clone)`. - -**Projections**, like `WellFormed(T: Iterator)`. +* *TraitRefs*, like `WellFormed(Vec: Clone)`. Well-formedness is important to [implied bounds]. In particular, the reason -it is okay to assume `FromEnv(T: Clone)` in the example above is that we +it is okay to assume `FromEnv(T: Clone)` in the `loud_clone` example is that we _also_ verify `WellFormed(T: Clone)` for each call site of `loud_clone`. +Similarly, it is okay to assume `FromEnv(HashSet)` in the `loud_insert` +example because we will verify `WellFormed(HashSet)` for each call site of +`loud_insert`. #### Outlives(Type: Region), Outlives(Region: Region) e.g. `Outlives(&'a str: 'b)`, `Outlives('a: 'static)` diff --git a/src/traits/index.md b/src/traits/index.md index c5afb280b..053a26bab 100644 --- a/src/traits/index.md +++ b/src/traits/index.md @@ -33,3 +33,17 @@ Trait solving is based around a few key ideas: constraints can be checked by thet type checker. Note: this is not a complete list of topics. See the sidebar for more. + +The design of the new-style trait solving currently happens in two places: +* The [chalk][chalk] repository is where we experiment with new ideas and + designs for the trait system. It basically consists of a unit testing framework + for the correctness and feasibility of the logical rules defining the new-style + trait system. It also provides the [`chalk_engine`][chalk_engine] crate, which + defines the new-style trait solver used both in the unit testing framework and + in rustc. +* Once we are happy with the logical rules, we proceed to implementing them in + rustc. This mainly happens in [`librustc_traits`][librustc_traits]. + +[chalk]: https://github.com/rust-lang-nursery/chalk +[chalk_engine]: https://github.com/rust-lang-nursery/chalk/tree/master/chalk-engine +[librustc_traits]: https://github.com/rust-lang/rust/tree/master/src/librustc_traits diff --git a/src/traits/lowering-rules.md b/src/traits/lowering-rules.md index 218164c0f..88a61ac4f 100644 --- a/src/traits/lowering-rules.md +++ b/src/traits/lowering-rules.md @@ -27,19 +27,24 @@ comment like so: // Rule Foo-Bar-Baz -you can also search through the `librustc_traits` crate in rustc -to find the corresponding rules from the implementation. +The reference implementation of these rules is to be found in +[`chalk/src/rules.rs`][chalk_rules]. They are also ported in rustc in the +[`librustc_traits`][librustc_traits] crate. + +[chalk_rules]: https://github.com/rust-lang-nursery/chalk/blob/master/src/rules.rs +[librustc_traits]: https://github.com/rust-lang/rust/tree/master/src/librustc_traits ## Lowering where clauses When used in a goal position, where clauses can be mapped directly to -[domain goals][dg], as follows: +the `Holds` variant of [domain goals][dg], as follows: -- `A0: Foo` maps to `Implemented(A0: Foo)`. -- `A0: Foo` maps to - `ProjectionEq(>::Item = T)` +- `A0: Foo` maps to `Implemented(A0: Foo)` - `T: 'r` maps to `Outlives(T, 'r)` - `'a: 'b` maps to `Outlives('a, 'b)` +- `A0: Foo` is a bit special and expands to two distinct + goals, namely `Implemented(A0: Foo)` and + `ProjectionEq(>::Item = T)` In the rules below, we will use `WC` to indicate where clauses that appear in Rust syntax; we will then use the same `WC` to indicate @@ -54,11 +59,10 @@ on the lowered where clauses, as defined here: - `FromEnv(WC)` – this indicates that: - `Implemented(TraitRef)` becomes `FromEnv(TraitRef)` - - `ProjectionEq(Projection = Ty)` becomes `FromEnv(Projection = Ty)` - other where-clauses are left intact - `WellFormed(WC)` – this indicates that: - `Implemented(TraitRef)` becomes `WellFormed(TraitRef)` - - `ProjectionEq(Projection = Ty)` becomes `WellFormed(Projection = Ty)` + - other where-clauses are left intact *TODO*: I suspect that we want to alter the outlives relations too, but Chalk isn't modeling those right now. @@ -99,9 +103,11 @@ forall { #### Implied bounds The next few clauses have to do with implied bounds (see also -[RFC 2089]). For each trait, we produce two clauses: +[RFC 2089] and the [implied bounds][implied_bounds] chapter for a more in depth +cover). For each trait, we produce two clauses: [RFC 2089]: https://rust-lang.github.io/rfcs/2089-implied-bounds.html +[implied_bounds]: ./implied-bounds.md ```text // Rule Implied-Bound-From-Trait @@ -210,7 +216,7 @@ well-formed, we can also assume that its where clauses hold. That is, we produce the following family of rules: ```text -// Rule FromEnv-Type +// Rule Implied-Bound-From-Type // // For each where clause `WC` forall { @@ -280,10 +286,10 @@ forall { ``` ```text -// Rule ProjectionEq-Skolemize +// Rule ProjectionEq-Placeholder // -// ProjectionEq can succeed by skolemizing, see "associated type" -// chapter for more: +// ProjectionEq can succeed through the placeholder associated type, +// see "associated type" chapter for more: forall { ProjectionEq( >::AssocType = @@ -303,7 +309,7 @@ elsewhere. // For each `Bound` in `Bounds`: forall { FromEnv(>::AssocType>: Bound) :- - FromEnv(Self: Trait) + FromEnv(Self: Trait) && WC1 } ``` @@ -314,7 +320,7 @@ type to be well-formed... // Rule WellFormed-AssocTy forall { WellFormed((Trait::AssocType)) :- - WC1, Implemented(Self: Trait) + Implemented(Self: Trait) && WC1 } ``` From 07bf5e332ba6b39e7c94e694a46a89f9e104e869 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Sun, 21 Oct 2018 18:05:32 +0200 Subject: [PATCH 366/648] Add an apostrophe Co-Authored-By: scalexm --- src/traits/goals-and-clauses.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/traits/goals-and-clauses.md b/src/traits/goals-and-clauses.md index 8cc1bc7fd..494ec5b85 100644 --- a/src/traits/goals-and-clauses.md +++ b/src/traits/goals-and-clauses.md @@ -58,7 +58,7 @@ see that clauses are always of the form: ```text forall { DomainGoal :- Goal } ``` -hence domain goals are in fact clauses LHS. That is, at the most granular level, +hence domain goals are in fact clauses' LHS. That is, at the most granular level, domain goals are what the trait solver will end up trying to prove. From ab89ea2446c4fb4f7fc7affdc5c900cb4667ec62 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Sun, 21 Oct 2018 18:05:51 +0200 Subject: [PATCH 367/648] Fix typo Co-Authored-By: scalexm --- src/traits/goals-and-clauses.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/traits/goals-and-clauses.md b/src/traits/goals-and-clauses.md index 494ec5b85..fb12c5a41 100644 --- a/src/traits/goals-and-clauses.md +++ b/src/traits/goals-and-clauses.md @@ -174,7 +174,7 @@ fn loud_insert(set: &mut HashSet, item: K) { ``` `HashSet` is an input type of the `loud_insert` function. Hence, we assume it -to be well-formed, so we would have `FromEnv(HashSet)` inside the body or our +to be well-formed, so we would have `FromEnv(HashSet)` inside the body of our function. As we'll see in the section on lowering, `FromEnv(HashSet)` implies `Implemented(K: Hash)` because the `HashSet` declaration was written with a `K: Hash` where clause. Hence, we don't From dfa38e973e5a45ec907bce1c9ca8a5a3a6971c4c Mon Sep 17 00:00:00 2001 From: scalexm Date: Sun, 21 Oct 2018 18:07:25 +0200 Subject: [PATCH 368/648] Change wording --- src/traits/goals-and-clauses.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/traits/goals-and-clauses.md b/src/traits/goals-and-clauses.md index fb12c5a41..8f1ffa488 100644 --- a/src/traits/goals-and-clauses.md +++ b/src/traits/goals-and-clauses.md @@ -103,7 +103,7 @@ WhereClause = Implemented(TraitRef) `WhereClause` refers to a `where` clause that a Rust user would actually be able to write in a Rust program. This abstraction exists only as a convenience as we -sometimes want to only coope with domain goals that are effectively writable in +sometimes want to only deal with domain goals that are effectively writable in Rust. Let's break down each one of these, one-by-one. @@ -117,8 +117,8 @@ True if the given trait is implemented for the given input types and lifetimes. e.g. `ProjectionEq::Item = u8` The given associated type `Projection` is equal to `Type`; this can be proved -with either normalization or using skolemized types. See [the section -on associated types](./associated-types.html). +with either normalization or using placeholder associated types. See +[the section on associated types](./associated-types.html). #### Normalize(Projection -> Type) e.g. `ProjectionEq::Item -> u8` From 2df0fcfdfb1fdeb8cf124fa317b7aeaaf453bd0c Mon Sep 17 00:00:00 2001 From: scalexm Date: Fri, 19 Oct 2018 15:20:41 +0200 Subject: [PATCH 369/648] Write implied bounds chapter --- src/traits/implied-bounds.md | 501 ++++++++++++++++++++++++++++++++++- 1 file changed, 496 insertions(+), 5 deletions(-) diff --git a/src/traits/implied-bounds.md b/src/traits/implied-bounds.md index 26a63bf32..1feb84e3b 100644 --- a/src/traits/implied-bounds.md +++ b/src/traits/implied-bounds.md @@ -1,9 +1,500 @@ # Implied Bounds -*to be written* +Implied bounds remove the need to repeat where clauses written on +a type declaration or a trait declaration. For example, say we have the +following type declaration: +```rust,ignore +struct HashSet { + ... +} +``` -Cover: +then everywhere we use `HashSet` as an "input" type, that is appearing in +the receiver type of an `impl` or in the arguments of a function, we don't +want to have to repeat the `where K: Hash` bound, as in: -- Why the `FromEnv` setup etc is the way it is -- Perhaps move some of the material from 'lowering rules' in to here -- Show various examples where you could go wrong +```rust,ignore +// I don't want to have to repeat `where K: Hash` here. +impl HashSet { + ... +} + +// Same here. +fn loud_insert(set: &mut HashSet, item: K) { + println!("inserting!"); + set.insert(item); +} +``` + +Note that in the `loud_insert` example, `HashSet` is not the type of an +argument of the `loud_insert` function, it only *appears* in the argument type +`&mut HashSet`. + +The rationale for applying implied bounds to input types is that, for example, +in order to call the `loud_insert` function above, the programmer must have +*produced* the type `HashSet` already, hence the compiler already verified +that `HashSet` was well-formed, i.e. that `K` effectively implemented +`Hash`, as in the following example: + +```rust,ignore +fn main() { + // I am producing a value of type `HashSet`. + // If `i32` was not `Hash`, the compiler would report an error here. + let set: HashSet = HashSet::new(); + loud_insert(&mut set, 5); +} +``` + +hence we don't want to repeat where clauses for input types because that would +sort of duplicate the work of the programmer, having to verify that their types +are well-formed both when calling the function and when using them in the +arguments of their function. The same reasoning applies when using an `impl`. + +Similarly, given the following trait declaration: +```rust,ignore +trait Copy where Self: Clone { + ... +} +``` + +then everywhere we bound over `SomeType: Copy`, we would like to be able to +use the fact that `SomeType: Clone` without having to write it explicitly, +as in: +```rust,ignore +fn loud_clone(x: T) { + println!("cloning!"); + x.clone(); +} + +fn fun_with_copy(x: T) { + println!("will clone a `Copy` type soon..."); + + // I'm using `loud_clone` with `T: Copy`, I know this + // implies `T: Clone` so I don't want to have to write it explicitly. + loud_clone(x); +} +``` + +The rationale for implied bounds for traits is that if a type implement `Copy`, +that is if there exists an `impl Copy` for that type, there *ought* to exist +an `impl Clone` for that type, otherwise the compiler would have reported an +error in the first place. So again, if we were forced to repeat the additionnal +`where SomeType: Clone` everywhere whereas we already know that +`SomeType: Copy` hold, we would kind of duplicate the verification work. + +Implied bounds are not yet completely enforced in rustc, at the moment it only +works for outlive requirements, super trait bounds and bounds on associated +types. The full RFC can be found [here][RFC]. We'll give here a brief view +of how implied bounds work and why we chose to implement it that way. The +complete set of lowering rules can be found in the corresponding +[chapter](./lowering-rules.md). + +[RFC]: https://github.com/rust-lang/rfcs/blob/master/text/2089-implied-bounds.md + +## Implied bounds and lowering rules + +Now we need to express implied bounds in terms of logical rules. We will start +with exposing a naive way to do it. Suppose that we have the following traits: +```rust,ignore +trait Foo { + ... +} + +trait Bar where Self: Foo { } { + ... +} +``` + +So we would like to say that if a type implements `Bar`, then necessarily +it must also implement `Foo`. We might think that a clause like this would +work: +```text +forall { + Implemented(Type: Foo) :- Implemented(Type: Bar). +} +``` + +Now suppose that we just write this impl: +```rust,ignore +struct X; + +impl Bar for X { } +``` + +Clearly this should not be allowed: indeed, we wrote a `Bar` impl for `X`, but +the `Bar` trait requires that we also implement `Foo` for `X`, which we never +did. In terms of what the compiler does, this would look like this: +```rust,ignore +struct X; + +impl Bar for X { + // We are in a `Bar` impl for the type `X`. + // There is a `where Self: Foo` bound on the `Bar` trait declaration. + // Hence I need to prove that `X` also implements `Foo` for that impl + // to be legal. +} +``` +So the compiler would try to prove `Implemented(X: Foo)`. Of course it will +not find any `impl Foo for X` since we did not write any. However, it +will see our implied bound clause: +```text +forall { + Implemented(Type: Foo) :- Implemented(Type: Bar). +} +``` + +so that it may be able to prove `Implemented(X: Foo)` if `Implemented(X: Bar)` +holds. And it turns out that `Implemented(X: Bar)` does hold since we wrote +a `Bar` impl for `X`! Hence the compiler will accept the `Bar` impl while it +should not. + +## Implied bounds coming from the environment + +So the naive approach does not work. What we need to do is to somehow decouple +implied bounds from impls. Suppose we know that a type `SomeType<...>` +implements `Bar` and we want to deduce that `SomeType<...>` must also implement +`Foo`. + +There are two possibilities: first one, we have enough information about +`SomeType<...>` to see that there exists a `Bar` impl in the program which +covers `SomeType<...>`, for example a plain `impl<...> Bar for SomeType<...>`. +Then if the compiler has done its job correctly, there *must* exist a `Foo` +impl which covers `SomeType<...>`, e.g. another plain +`impl<...> Foo for SomeType<...>`. In that case then, we can just use this +impl and we do not need implied bounds at all. + +Second possibility: we do not know enough about `SomeType<...>` in order to +find a `Bar` impl which covers it, for example if `SomeType<...>` is just +a type parameter in a function: +```rust,ignore +fn foo() { + // We'd like to deduce `Implemented(T: Foo)`. +} +``` + +that is, the information that `T` implements `Bar` here comes from the +*environment*. The environment is the set of things that we assume to be true +when we type check some Rust declaration. In that case, what we assume is that +`T: Bar`. Then at that point, we might authorize ourselves to have some kind +of "local" implied bound reasoning which would say +`Implemented(T: Foo) :- Implemented(T: Bar)`. This reasoning would +only be done within our `foo` function in order to avoid the earlier +problem where we had a global clause. + +We can apply these local reasonings everywhere we can have an environment +-- i.e. when we can write where clauses -- that is inside impls, +trait declarations and type declarations. + +## Computing implied bounds with `FromEnv` + +The previous subsection showed that it was only useful to compute implied +bounds for facts coming from the environment. +We talked about "local" rules, but there are multiple possible strategies to +indeed implement the locality of implied bounds. + +In rustc, the current strategy is to *elaborate* bounds: that is, each time +we have a fact in the environment, we recursively derive all the other things +that are implied by this fact until we reach a fixed point. For example, if +we have the following declarations: +```rust,ignore +trait A { } +trait B where Self: A { } +trait C where Self: B { } + +fn foo() { + ... +} +``` +then inside the `foo` function, we start with an environment containing only +`Implemented(T: C)`. Then because of implied bounds for the `C` trait, we +elaborate `Implemented(T: B)` and add it to our environment. Because of +implied bounds for the `B` trait, we elaborate `Implemented(T: A)`and add it +to our environment as well. We cannot elaborate anything else, so we conclude +that our final environment consists of `Implemented(T: A + B + C)`. + +In the new-style trait system, we like to encode as many things as possible +with logical rules. So rather than "elaborating", we have a set of *global* +program clauses defined like so: +```text +forall { Implemented(T: A) :- FromEnv(T: A). } + +forall { Implemented(T: B) :- FromEnv(T: B). } +forall { FromEnv(T: A) :- FromEnv(T: B). } + +forall { Implemented(T: C) :- FromEnv(T: C). } +forall { FromEnv(T: C) :- FromEnv(T: C). } +``` +So these clauses are defined globally (that is they are available from +everywhere in the program) but they cannot be used because the hypothesis +is always of the form `FromEnv(...)` which is a bit special. Indeed, as +indicated by the name, `FromEnv(...)` facts can **only** come from the +environment. +How it works is that in the `foo` function, instead of having an environment +containing `Implemented(T: C)`, we replace this environment with +`FromEnv(T: C)`. From here and thanks to the above clauses, we see that we +are able to reach any of `Implemented(T: A)`, `Implemented(T: B)` or +`Implemented(T: C)`, which is what we wanted. + +## Implied bounds and well-formedness checking + +Implied bounds are tightly related with well-formedness checking. +Well-formedness checking is the process of checking that the impls the +programmer wrote are legal, what we referred to earlier as "the compiler doing +its job correctly". + +We already saw examples of illegal and legal impls: +```rust,ignore +trait Foo { } +trait Bar where Self: Foo { } + +struct X; +struct Y; + +impl Bar for X { + // This impl is not legal: the `Bar` trait requires that we also + // implement `Foo`, and we didn't. +} + +impl Foo for Y { + // This impl is legal: there is nothing to check as there are no where + // clauses on the `Foo` trait. +} + +impl Bar for Y { + // This impl is legal: we have a `Foo` impl for `Y`. +} +``` +We must define what "legal" and "illegal" mean. For this, we introduce another +predicate: `WellFormed(Type: Trait)`. We say that the trait reference +`Type: Trait` is well-formed is `Type` meets the bounds written on the +`Trait` declaration. For each impl we write, assuming that the where clauses +declared on the impl hold, the compiler tries to prove that the corresponding +trait reference is well-formed. The impl is legal if the compiler manages to do +so. + +Coming to the definition of `WellFormed(Type: Trait)`, it would be tempting +to define it as: +```rust,ignore +trait Trait where WC1, WC2, ..., WCn { + ... +} +``` + +```text +forall { + WellFormed(Type: Trait) :- WC1 && WC2 && .. && WCn. +} +``` +and indeed this was basically what was done in rustc until it was noticed that +this mixed badly with implied bounds. The key thing is that implied bounds +allows someone to derive all bounds implied by a fact in the environment, and +this *transitively* as we've seen with the `A + B + C` traits example. +However, the `WellFormed` predicate as defined above only checks that the +*direct* superbounds hold. That is, if we come back to our `A + B + C` +example: +```rust,ignore +trait A { } +// No where clauses, always well-formed. +// forall { WellFormed(Type: A). } + +trait B where Self: A { } +// We only check the direct superbound `Self: A`. +// forall { WellFormed(Type: B) :- Implemented(Type: A). } + +trait C where Self: B { } +// We only check the direct superbound `Self: B`. We do not check +// the `Self: A` implied bound coming from the `Self: B` superbound. +// forall { WellFormed(Type: C) :- Implemented(Type: B). } +``` +There is an asymmetry between the recursive power of implied bounds and +the shallow checking of `WellFormed`. It turns out that this asymmetry +can be [exploited][bug]. Indeed, suppose that we define the following +traits: +```rust,ignore +trait Partial where Self: Copy { } +// WellFormed(Self: Partial) :- Implemented(Self: Copy). + +trait Complete where Self: Partial { } +// WellFormed(Self: Complete) :- Implemented(Self: Partial). + +impl Partial for T where T: Complete { } + +impl Complete for T { } +``` + +For the `Partial` impl, what the compiler must prove is: +```text +forall { + if (T: Complete) { // assume that the where clauses hold + WellFormed(T: Partial) // show that the trait reference is well-formed + } +} +``` +Proving `WellFormed(T: Partial)` amounts to proving `Implemented(T: Copy)`. +However, we have `Implemented(T: Complete)` in our environment: thanks to +implied bounds, we can deduce `Implemented(T: Partial)`. Using implied bounds +one level deeper, we can deduce `Implemented(T: Copy)`. Finally, the `Partial` +impl is legal. + +For the `Complete` impl, what the compiler must prove is: +```text +forall { + WellFormed(T: Complete) // show that the trait reference is well-formed +} +``` +Proving `WellFormed(T: Complete)` amounts to proving `Implemented(T: Partial)`. +We see that the `impl Partial for T` applies if we can prove +`Implemented(T: Complete)`, and it turns out we can prove this fact since our +`impl Complete for T` is a blanket impl without any where clauses. + +So both impls are legal and the compiler accepts the program. Moreover, thanks +to the `Complete` blanket impl, all types implement `Complete`. So we could +now use this impl like so: +```rust,ignore +fn eat(x: T) { } + +fn copy_everything(x: T) { + eat(x); + eat(x); +} + +fn main() { + let not_copiable = vec![1, 2, 3, 4]; + copy_everything(not_copiable); +} +``` +In this program, we use the fact that `Vec` implements `Complete`, as any +other type. Hence we can call `copy_everything` with an argument of type +`Vec`. Inside the `copy_everything` function, we have the +`Implemented(T: Complete)` bound in our environment. Thanks to implied bounds, +we can deduce `Implemented(T: Partial)`. Using implied bounds again, we deduce +`Implemented(T: Copy)` and we can indeed call the `eat` function which moves +the argument twice since its argument is `Copy`. Problem: the `T` type was +in fact `Vec` which is not copy at all, hence we will double-free the +underlying vec storage so we have a memory unsoundness in safe Rust. + +Of course, disregarding the asymmetry between `WellFormed` and implied bounds, +this bug was possible only because we had some kind of self-referencing impls. +But self-referencing impls are very useful in practice and are not the real +culprits in this affair. + +[bug]: https://github.com/rust-lang/rust/pull/43786 + +## Co-inductiveness of `WellFormed` + +So the solution is to fix this asymmetry between `WellFormed` and implied +bounds. For that, we need for the `WellFormed` predicate to not only require +that the direct superbounds hold, but also all the bounds transitively implied +by the superbounds. What we can do is to have the following rules for the +`WellFormed` predicate: +```rust,ignore +trait A { } +// WellFormed(Self: A) :- Implemented(Self: A). + +trait B where Self: A { } +// WellFormed(Self: B) :- Implemented(Self: B) && WellFormed(Self: A). + +trait C where Self: B { } +// WellFormed(Self: C) :- Implemented(Self: C) && WellFormed(Self: B). +``` + +Notice that we are now also requiring `Implemented(Self: Trait)` for +`WellFormed(Self: Trait)` to be true: this is to simplify the process of +traversing all the implied bounds transitively. This does not change anything +when checking whether impls are legal, because since we assume +that the where clauses hold inside the impl, we know that the corresponding +trait reference do hold. Thanks to this setup, you can see that we indeed +require to prove the set of all bounds transitively implied by the where +clauses. + +However there is still a catch. Suppose that we have the following trait +definition: +```rust,ignore +trait Foo where ::Item: Foo { + type Item; +} +``` + +so this definition is a bit more involved than the ones we've seen already +because it defines an associated item. However, the well-formedness rule +would not be more complicated: +```text +WellFormed(Self: Foo) :- + Implemented(Self: Foo) && + WellFormed(::Item: Foo). +``` + +Now we would like to write the following impl: +```rust,ignore +impl Foo for i32 { + type Item = i32; +} +``` +The `Foo` trait definition and the `impl Foo for i32` are perfectly valid +Rust: we're kind of recursively using our `Foo` impl in order to show that +the associated value indeed implements `Foo`, but that's ok. But if we +translates this to our well-formedness setting, the compiler proof process +inside the `Foo` impl is the following: it starts with proving that the +well-formedness goal `WellFormed(i32: Foo)` is true. In order to do that, +it must prove the following goals: `Implemented(i32: Foo)` and +`WellFormed(::Item: Foo)`. `Implemented(i32: Foo)` holds because +there is our impl and there are no where clauses on it so it's always true. +However, because of the associated type value we used, +`WellFormed(::Item: Foo)` simplifies to just +`WellFormed(i32: Foo)`. So in order to prove its original goal +`WellFormed(i32: Foo)`, the compiler needs to prove `WellFormed(i32: Foo)`: +this clearly is a cycle and cycles are usually rejected by the trait solver, +unless... if the `WellFormed` predicate was made to be co-inductive. + +A co-inductive predicate, as discussed in the chapter on +[goals and clauses](./goals-and-clauses.md#coinductive-goals), are predicates +for which the +trait solver accepts cycles. In our setting, this would be a valid thing to do: +indeed, the `WellFormed` predicate just serves as a way of enumerating all +the implied bounds. Hence, it's like a fixed point algorithm: it tries to grow +the set of implied bounds until there is nothing more to add. Here, a cycle +in the chain of `WellFormed` predicates just means that there is no more bounds +to add in that direction, so we can just accept this cycle and focus on other +directions. It's easy to prove that under these co-inductive semantics, we +are effectively visiting all the transitive implied bounds, and only these. + +## Implied bounds on types + +We mainly talked about implied bounds for traits because this was the most +subtle regarding implementation. Implied bounds on types are simpler, +especially because if we assume that a type is well-formed, we don't use that +fact to deduce that other types are well-formed, we only use it to deduce +that e.g. some trait bounds hold. + +For types, we just use rules like these ones: +```rust,ignore +struct Type<...> where WC1, ..., WCn { + ... +} +``` + +```text +forall<...> { + WellFormed(Type<...>) :- WC1, ..., WCn. +} + +forall<...> { + FromEnv(WC1) :- FromEnv(Type<...>). + ... + FromEnv(WCn) :- FromEnv(Type<...>). +} +``` +We can see that we have this asymmetry between well-formedness check, +which only verifies that the direct superbounds hold, and implied bounds which +gives access to all bounds transitively implied by the where clauses. In that +case this is ok because as we said, we don't use `FromEnv(Type<...>)` to deduce +other `FromEnv(OtherType<...>)` things, nor do we use `FromEnv(Type: Trait)` to +deduce `FromEnv(OtherType<...>)` things. So in that sense type definitions are +"less recursive" than traits, and we saw in a previous subsection that +it was the combination of asymmetry and recursive trait / impls that led to +unsoundness. As such, the `WellFormed(Type<...>)` predicate does not need +to be co-inductive. + +This asymmetry optimization is useful because in a real Rust program, we have +to check the well-formedness of types very often (e.g. for each type which +appears in the body of a function). From 2b8f3d40e1fe4923c3bfe19ce0cb5be00a7824d2 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Sun, 21 Oct 2018 18:09:00 +0200 Subject: [PATCH 370/648] Fix typos and punctuation Co-Authored-By: scalexm --- src/traits/implied-bounds.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/src/traits/implied-bounds.md b/src/traits/implied-bounds.md index 1feb84e3b..6f033e1b1 100644 --- a/src/traits/implied-bounds.md +++ b/src/traits/implied-bounds.md @@ -45,7 +45,7 @@ fn main() { } ``` -hence we don't want to repeat where clauses for input types because that would +Hence, we don't want to repeat where clauses for input types because that would sort of duplicate the work of the programmer, having to verify that their types are well-formed both when calling the function and when using them in the arguments of their function. The same reasoning applies when using an `impl`. @@ -75,15 +75,15 @@ fn fun_with_copy(x: T) { } ``` -The rationale for implied bounds for traits is that if a type implement `Copy`, -that is if there exists an `impl Copy` for that type, there *ought* to exist +The rationale for implied bounds for traits is that if a type implements `Copy`, +that is, if there exists an `impl Copy` for that type, there *ought* to exist an `impl Clone` for that type, otherwise the compiler would have reported an error in the first place. So again, if we were forced to repeat the additionnal `where SomeType: Clone` everywhere whereas we already know that `SomeType: Copy` hold, we would kind of duplicate the verification work. Implied bounds are not yet completely enforced in rustc, at the moment it only -works for outlive requirements, super trait bounds and bounds on associated +works for outlive requirements, super trait bounds, and bounds on associated types. The full RFC can be found [here][RFC]. We'll give here a brief view of how implied bounds work and why we chose to implement it that way. The complete set of lowering rules can be found in the corresponding @@ -155,7 +155,7 @@ implied bounds from impls. Suppose we know that a type `SomeType<...>` implements `Bar` and we want to deduce that `SomeType<...>` must also implement `Foo`. -There are two possibilities: first one, we have enough information about +There are two possibilities: first, we have enough information about `SomeType<...>` to see that there exists a `Bar` impl in the program which covers `SomeType<...>`, for example a plain `impl<...> Bar for SomeType<...>`. Then if the compiler has done its job correctly, there *must* exist a `Foo` @@ -172,7 +172,7 @@ fn foo() { } ``` -that is, the information that `T` implements `Bar` here comes from the +That is, the information that `T` implements `Bar` here comes from the *environment*. The environment is the set of things that we assume to be true when we type check some Rust declaration. In that case, what we assume is that `T: Bar`. Then at that point, we might authorize ourselves to have some kind @@ -182,8 +182,8 @@ only be done within our `foo` function in order to avoid the earlier problem where we had a global clause. We can apply these local reasonings everywhere we can have an environment --- i.e. when we can write where clauses -- that is inside impls, -trait declarations and type declarations. +-- i.e. when we can write where clauses -- that is, inside impls, +trait declarations, and type declarations. ## Computing implied bounds with `FromEnv` @@ -224,7 +224,7 @@ forall { FromEnv(T: A) :- FromEnv(T: B). } forall { Implemented(T: C) :- FromEnv(T: C). } forall { FromEnv(T: C) :- FromEnv(T: C). } ``` -So these clauses are defined globally (that is they are available from +So these clauses are defined globally (that is, they are available from everywhere in the program) but they cannot be used because the hypothesis is always of the form `FromEnv(...)` which is a bit special. Indeed, as indicated by the name, `FromEnv(...)` facts can **only** come from the @@ -266,7 +266,7 @@ impl Bar for Y { ``` We must define what "legal" and "illegal" mean. For this, we introduce another predicate: `WellFormed(Type: Trait)`. We say that the trait reference -`Type: Trait` is well-formed is `Type` meets the bounds written on the +`Type: Trait` is well-formed if `Type` meets the bounds written on the `Trait` declaration. For each impl we write, assuming that the where clauses declared on the impl hold, the compiler tries to prove that the corresponding trait reference is well-formed. The impl is legal if the compiler manages to do @@ -433,7 +433,7 @@ impl Foo for i32 { The `Foo` trait definition and the `impl Foo for i32` are perfectly valid Rust: we're kind of recursively using our `Foo` impl in order to show that the associated value indeed implements `Foo`, but that's ok. But if we -translates this to our well-formedness setting, the compiler proof process +translate this to our well-formedness setting, the compiler proof process inside the `Foo` impl is the following: it starts with proving that the well-formedness goal `WellFormed(i32: Foo)` is true. In order to do that, it must prove the following goals: `Implemented(i32: Foo)` and From 39cdd4f9206349c21b019ba4246d4920b4c94204 Mon Sep 17 00:00:00 2001 From: scalexm Date: Sun, 21 Oct 2018 18:30:07 +0200 Subject: [PATCH 371/648] Clarify wording --- src/traits/implied-bounds.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/src/traits/implied-bounds.md b/src/traits/implied-bounds.md index 6f033e1b1..f32c9d0cb 100644 --- a/src/traits/implied-bounds.md +++ b/src/traits/implied-bounds.md @@ -26,9 +26,11 @@ fn loud_insert(set: &mut HashSet, item: K) { } ``` -Note that in the `loud_insert` example, `HashSet` is not the type of an -argument of the `loud_insert` function, it only *appears* in the argument type -`&mut HashSet`. +Note that in the `loud_insert` example, `HashSet` is not the type +of the `set` argument of `loud_insert`, it only *appears* in the +argument type `&mut HashSet`: we care about every type appearing +in the function's header (the header is the signature without the return type), +not only types of the function's arguments. The rationale for applying implied bounds to input types is that, for example, in order to call the `loud_insert` function above, the programmer must have @@ -52,7 +54,7 @@ arguments of their function. The same reasoning applies when using an `impl`. Similarly, given the following trait declaration: ```rust,ignore -trait Copy where Self: Clone { +trait Copy where Self: Clone { // desugared version of `Copy: Clone` ... } ``` @@ -75,11 +77,11 @@ fn fun_with_copy(x: T) { } ``` -The rationale for implied bounds for traits is that if a type implements `Copy`, -that is, if there exists an `impl Copy` for that type, there *ought* to exist -an `impl Clone` for that type, otherwise the compiler would have reported an -error in the first place. So again, if we were forced to repeat the additionnal -`where SomeType: Clone` everywhere whereas we already know that +The rationale for implied bounds for traits is that if a type implements +`Copy`, that is, if there exists an `impl Copy` for that type, there *ought* +to exist an `impl Clone` for that type, otherwise the compiler would have +reported an error in the first place. So again, if we were forced to repeat the +additionnal `where SomeType: Clone` everywhere whereas we already know that `SomeType: Copy` hold, we would kind of duplicate the verification work. Implied bounds are not yet completely enforced in rustc, at the moment it only From 2fd124856dc3a17184a57837c8710b184fbb879d Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Mon, 22 Oct 2018 12:48:58 -0500 Subject: [PATCH 372/648] Add a note on bisecting --- src/compiler-debugging.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index d1171e0e0..aca7f7424 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -384,3 +384,7 @@ create a minimal working example with Godbolt. Go to [env-logger]: https://docs.rs/env_logger/0.4.3/env_logger/ + +## Narrowing (Bisecting) Regressions + +The [cargo-bisect-rustc](https://github.com/rust-lang-nursery/cargo-bisect-rustc) tool can be used as a quick and easy way to find exactly which PR caused a change in `rustc` behavior. It automatically downloads `rustc` PR artifacts and tests them against a project you provide until it finds the regression. You can then look at the PR to get more context on *why* it was changed. See [this tutorial](https://github.com/rust-lang-nursery/cargo-bisect-rustc/blob/master/TUTORIAL.md) on how to use it. From 607a58f8d2f56068ac1cf60c550d54f3ee6d90c6 Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Wed, 24 Oct 2018 15:16:12 -0400 Subject: [PATCH 373/648] issue_100_4 Updated the rustc documentation location --- src/compiler-documenting.md | 12 +++++++++--- src/how-to-build-and-run.md | 6 ++++++ 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/src/compiler-documenting.md b/src/compiler-documenting.md index bf63c0120..97636fad5 100644 --- a/src/compiler-documenting.md +++ b/src/compiler-documenting.md @@ -19,7 +19,7 @@ available like the standard library. There’s two ways to go about this. ./x.py doc --stage 1 ``` -First the compiler and rustdoc get built to make sure everything is okay +First the compiler and rustdoc get built to make sure everything is okay and then it documents the files. ## Document specific components @@ -52,5 +52,11 @@ When you want to build the compiler docs as well run this command: ./x.py doc ``` -This will see that the docs and compiler-docs options are set to true -and build the normally hidden compiler docs! \ No newline at end of file +This will see that the docs and compiler-docs options are set to true +and build the normally hidden compiler docs! + +### Compiler Documentation + +The documentation for the rust components are found at [rustc doc]. + +[rustc doc]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ \ No newline at end of file diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 4315cd708..62461d3ef 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -330,3 +330,9 @@ everything up then you only need to run one command! ```bash > ./x.py clean ``` + +### Compiler Documentation + +The documentation for the rust components are found at [rustc doc]. + +[rustc doc]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ \ No newline at end of file From 875905587687c5582399cd9fbeacc5f8e1cb56c9 Mon Sep 17 00:00:00 2001 From: csmoe <35686186+csmoe@users.noreply.github.com> Date: Wed, 24 Oct 2018 14:37:33 +0800 Subject: [PATCH 374/648] clean up skolemiza in glossary --- src/appendix/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 5c7d82741..04d35777f 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -53,7 +53,7 @@ rib | a data structure in the name resolver that keeps trac sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. sigil | like a keyword but composed entirely of non-alphanumeric tokens. For example, `&` is a sigil for references. -skolemization | a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)`) as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on skolemization and universes](../borrow_check/region_inference.html#skol) for more details. +placeholder | a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)`) as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on placeholder and universes](../borrow_check/region_inference.html#placeholder) for more details. soundness | soundness is a technical term in type theory. Roughly, if a type system is sound, then if a program type-checks, it is type-safe; i.e. I can never (in safe rust) force a value into a variable of the wrong type. (see "completeness"). span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. substs | the substitutions for a given generic type or item (e.g. the `i32`, `u32` in `HashMap`) From a07a6de51d2eb14f068ad0a5792385627a1cd505 Mon Sep 17 00:00:00 2001 From: csmoe <35686186+csmoe@users.noreply.github.com> Date: Wed, 24 Oct 2018 14:43:38 +0800 Subject: [PATCH 375/648] clean up skolemiza in borrow_ck --- src/borrow_check/region_inference.md | 82 ++++++++++++++-------------- 1 file changed, 41 insertions(+), 41 deletions(-) diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index 5a09f0bb0..617133000 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -78,19 +78,19 @@ The kinds of region elements are as follows: etc) control-flow graph. - Similarly, there is an element denoted `end('static)` corresponding to the remainder of program execution after this function returns. -- There is an element `!1` for each skolemized region `!1`. This +- There is an element `!1` for each placeholder region `!1`. This corresponds (intuitively) to some unknown set of other elements – - for details on skolemization, see the section - [skolemization and universes](#skol). + for details on placeholder, see the section + [placeholder and universes](#placeholder). ## Causal tracking *to be written* – describe how we can extend the values of a variable with causal tracking etc - + -## Skolemization and universes +## Placeholders and universes (This section describes ongoing work that hasn't landed yet.) @@ -117,7 +117,7 @@ for its argument, and `bar` wants to be given a function that that accepts **any** reference (so it can call it with something on its stack, for example). But *how* do we reject it and *why*? -### Subtyping and skolemization +### Subtyping and Placeholder When we type-check `main`, and in particular the call `bar(foo)`, we are going to wind up with a subtyping relationship like this one: @@ -129,10 +129,10 @@ the type of `foo` the type `bar` expects ``` We handle this sort of subtyping by taking the variables that are -bound in the supertype and **skolemizing** them: this means that we +bound in the supertype and **placeholder** them: this means that we replace them with [universally quantified](../appendix/background.html#quantified) -representatives, written like `!1`. We call these regions "skolemized +representatives, written like `!1`. We call these regions "placeholder regions" – they represent, basically, "some unknown region". Once we've done that replacement, we have the following relation: @@ -163,7 +163,7 @@ should yield up an error (eventually). ### What is a universe -In the previous section, we introduced the idea of a skolemized +In the previous section, we introduced the idea of a placeholder region, and we denoted it `!1`. We call this number `1` the **universe index**. The idea of a "universe" is that it is a set of names that are in scope within some type or at some point. Universes are formed @@ -198,7 +198,7 @@ fn bar<'a, T>(t: &'a T) { ``` Here, the name `'b` is not part of the root universe. Instead, when we -"enter" into this `for<'b>` (e.g., by skolemizing it), we will create +"enter" into this `for<'b>` (e.g., by placeholder it), we will create a child universe of the root, let's call it U1: ```text @@ -274,25 +274,25 @@ Here, the only way for the two foralls to interact would be through X, but neither Y nor Z are in scope when X is declared, so its value cannot reference either of them. -### Universes and skolemized region elements +### Universes and placeholder region elements But where does that error come from? The way it happens is like this. When we are constructing the region inference context, we can tell -from the type inference context how many skolemized variables exist +from the type inference context how many placeholder variables exist (the `InferCtxt` has an internal counter). For each of those, we create a corresponding universal region variable `!n` and a "region -element" `skol(n)`. This corresponds to "some unknown set of other -elements". The value of `!n` is `{skol(n)}`. +element" `placeholder(n)`. This corresponds to "some unknown set of other +elements". The value of `!n` is `{placeholder(n)}`. At the same time, we also give each existential variable a **universe** (also taken from the `InferCtxt`). This universe -determines which skolemized elements may appear in its value: For -example, a variable in universe U3 may name `skol(1)`, `skol(2)`, and -`skol(3)`, but not `skol(4)`. Note that the universe of an inference +determines which placeholder elements may appear in its value: For +example, a variable in universe U3 may name `placeholder(1)`, `placeholder(2)`, and +`placeholder(3)`, but not `placeholder(4)`. Note that the universe of an inference variable controls what region elements **can** appear in its value; it does not say region elements **will** appear. -### Skolemization and outlives constraints +### Placeholders and outlives constraints In the region inference engine, outlives constraints have the form: @@ -313,23 +313,23 @@ are present in `value(V2)` and we add those nodes to `value(V1)`. If we reach a return point, we add in any `end(X)` elements. That part remains unchanged. -But then *after that* we want to iterate over the skolemized `skol(x)` +But then *after that* we want to iterate over the placeholder `placeholder(x)` elements in V2 (each of those must be visible to `U(V2)`, but we should be able to just assume that is true, we don't have to check it). We have to ensure that `value(V1)` outlives each of those -skolemized elements. +placeholder elements. Now there are two ways that could happen. First, if `U(V1)` can see -the universe `x` (i.e., `x <= U(V1)`), then we can just add `skol(x)` +the universe `x` (i.e., `x <= U(V1)`), then we can just add `placeholder(x)` to `value(V1)` and be done. But if not, then we have to approximate: -we may not know what set of elements `skol(x)` represents, but we +we may not know what set of elements `placeholder(x)` represents, but we should be able to compute some sort of **upper bound** B for it – -some region B that outlives `skol(x)`. For now, we'll just use +some region B that outlives `placeholder(x)`. For now, we'll just use `'static` for that (since it outlives everything) – in the future, we can sometimes be smarter here (and in fact we have code for doing this already in other contexts). Moreover, since `'static` is in the root universe U0, we know that all variables can see it – so basically if -we find that `value(V2)` contains `skol(x)` for some universe `x` +we find that `value(V2)` contains `placeholder(x)` for some universe `x` that `V1` can't see, then we force `V1` to `'static`. ### Extending the "universal regions" check @@ -337,20 +337,20 @@ that `V1` can't see, then we force `V1` to `'static`. After all constraints have been propagated, the NLL region inference has one final check, where it goes over the values that wound up being computed for each universal region and checks that they did not get -'too large'. In our case, we will go through each skolemized region -and check that it contains *only* the `skol(u)` element it is known to +'too large'. In our case, we will go through each placeholder region +and check that it contains *only* the `placeholder(u)` element it is known to outlive. (Later, we might be able to know that there are relationships -between two skolemized regions and take those into account, as we do +between two placeholder regions and take those into account, as we do for universal regions from the fn signature.) Put another way, the "universal regions" check can be considered to be checking constraints like: ```text -{skol(1)}: V1 +{placeholder(1)}: V1 ``` -where `{skol(1)}` is like a constant set, and V1 is the variable we +where `{placeholder(1)}` is like a constant set, and V1 is the variable we made to represent the `!1` region. ## Back to our example @@ -365,7 +365,7 @@ fn(&'static u32) <: fn(&'!1 u32) @ P // this point P is not imp't here The region inference engine will create a region element domain like this: ```text -{ CFG; end('static); skol(1) } +{ CFG; end('static); placeholder(1) } --- ------------ ------- from the universe `!1` | 'static is always in scope all points in the CFG; not especially relevant here @@ -377,7 +377,7 @@ will have initial values like so: ```text Vs = { CFG; end('static) } // it is in U0, so can't name anything else -V1 = { skol(1) } +V1 = { placeholder(1) } ``` From the subtyping constraint above, we would have an outlives constraint like @@ -390,7 +390,7 @@ To process this, we would grow the value of V1 to include all of Vs: ```text Vs = { CFG; end('static) } -V1 = { CFG; end('static), skol(1) } +V1 = { CFG; end('static), placeholder(1) } ``` At that point, constraint propagation is complete, because all the @@ -411,7 +411,7 @@ for<'a> fn(&'a u32, &'a u32) for<'b, 'c> fn(&'b u32, &'c u32) ``` -Here we would skolemize the supertype, as before, yielding: +Here we would placeholer the supertype, as before, yielding: ```text for<'a> fn(&'a u32, &'a u32) @@ -476,7 +476,7 @@ an error. That's good: the problem is that we've gone from a fn that promises to return one of its two arguments, to a fn that is promising to return the first one. That is unsound. Let's see how it plays out. -First, we skolemize the supertype: +First, we placeholder the supertype: ```text for<'a> fn(&'a u32, &'a u32) -> &'a u32 @@ -512,26 +512,26 @@ V3: V1 Those variables will have these initial values: ```text -V1 in U1 = {skol(1)} -V2 in U2 = {skol(2)} +V1 in U1 = {placeholder(1)} +V2 in U2 = {placeholder(2)} V3 in U2 = {} ``` -Now because of the `V3: V1` constraint, we have to add `skol(1)` into `V3` (and +Now because of the `V3: V1` constraint, we have to add `placeholder(1)` into `V3` (and indeed it is visible from `V3`), so we get: ```text -V3 in U2 = {skol(1)} +V3 in U2 = {placeholder(1)} ``` then we have this constraint `V2: V3`, so we wind up having to enlarge -`V2` to include `skol(1)` (which it can also see): +`V2` to include `placeholder(1)` (which it can also see): ```text -V2 in U2 = {skol(1), skol(2)} +V2 in U2 = {placeholder(1), placeholder(2)} ``` Now constraint propagation is done, but when we check the outlives -relationships, we find that `V2` includes this new element `skol(1)`, +relationships, we find that `V2` includes this new element `placeholder(1)`, so we report an error. From 294fcd31acab74f30172cff042745138f014811c Mon Sep 17 00:00:00 2001 From: csmoe <35686186+csmoe@users.noreply.github.com> Date: Wed, 24 Oct 2018 14:46:43 +0800 Subject: [PATCH 376/648] clean up skolemiza in traits --- src/traits/associated-types.md | 2 +- src/traits/caching.md | 6 +++--- src/traits/hrtb.md | 28 ++++++++++++++-------------- 3 files changed, 18 insertions(+), 18 deletions(-) diff --git a/src/traits/associated-types.md b/src/traits/associated-types.md index 3330ce809..1fffa3ff8 100644 --- a/src/traits/associated-types.md +++ b/src/traits/associated-types.md @@ -154,7 +154,7 @@ The key point is that, on success, unification can also give back to us a set of subgoals that still remain to be proven (it can also give back region constraints, but those are not relevant here). -Whenever unification encounters an (unskolemized!) associated type +Whenever unification encounters an (un-placeholder!) associated type projection P being equated with some other type T, it always succeeds, but it produces a subgoal `ProjectionEq(P = T)` that is propagated back up. Thus it falls to the ordinary workings of the trait system diff --git a/src/traits/caching.md b/src/traits/caching.md index f84539509..7978306d1 100644 --- a/src/traits/caching.md +++ b/src/traits/caching.md @@ -11,10 +11,10 @@ but *replay* its effects on the type variables. ## An example The high-level idea of how the cache works is that we first replace -all unbound inference variables with skolemized versions. Therefore, +all unbound inference variables with placeholder versions. Therefore, if we had a trait reference `usize : Foo<$t>`, where `$t` is an unbound inference variable, we might replace it with `usize : Foo<$0>`, where -`$0` is a skolemized type. We would then look this up in the cache. +`$0` is a placeholder type. We would then look this up in the cache. If we found a hit, the hit would tell us the immediate next step to take in the selection process (e.g. apply impl #22, or apply where @@ -37,7 +37,7 @@ we would [confirm] `ImplCandidate(22)`, which would (as a side-effect) unify [confirm]: ./resolution.html#confirmation Now, at some later time, we might come along and see a `usize : -Foo<$u>`. When skolemized, this would yield `usize : Foo<$0>`, just as +Foo<$u>`. When placeholder, this would yield `usize : Foo<$0>`, just as before, and hence the cache lookup would succeed, yielding `ImplCandidate(22)`. We would confirm `ImplCandidate(22)` which would (as a side-effect) unify `$u` with `isize`. diff --git a/src/traits/hrtb.md b/src/traits/hrtb.md index 7f77f274c..9986bbb16 100644 --- a/src/traits/hrtb.md +++ b/src/traits/hrtb.md @@ -5,7 +5,7 @@ bounds*. An example of such a bound is `for<'a> MyTrait<&'a isize>`. Let's walk through how selection on higher-ranked trait references works. -## Basic matching and skolemization leaks +## Basic matching and placeholder leaks Suppose we have a trait `Foo`: @@ -36,11 +36,11 @@ to the subtyping for higher-ranked types (which is described [here][hrsubtype] and also in a [paper by SPJ]. If you wish to understand higher-ranked subtyping, we recommend you read the paper). There are a few parts: -**TODO**: We should define _skolemize_. +**TODO**: We should define _placeholder_. 1. _Skolemize_ the obligation. -2. Match the impl against the skolemized obligation. -3. Check for _skolemization leaks_. +2. Match the impl against the placeholder obligation. +3. Check for _placeholder leaks_. [hrsubtype]: https://github.com/rust-lang/rust/tree/master/src/librustc/infer/higher_ranked/README.md [paper by SPJ]: http://research.microsoft.com/en-us/um/people/simonpj/papers/higher-rank/ @@ -48,8 +48,8 @@ subtyping, we recommend you read the paper). There are a few parts: So let's work through our example. 1. The first thing we would do is to -skolemize the obligation, yielding `AnyInt : Foo<&'0 isize>` (here `'0` -represents skolemized region #0). Note that we now have no quantifiers; +placeholder the obligation, yielding `AnyInt : Foo<&'0 isize>` (here `'0` +represents placeholder region #0). Note that we now have no quantifiers; in terms of the compiler type, this changes from a `ty::PolyTraitRef` to a `TraitRef`. We would then create the `TraitRef` from the impl, using fresh variables for it's bound regions (and thus getting @@ -59,10 +59,10 @@ using fresh variables for it's bound regions (and thus getting we relate the two trait refs, yielding a graph with the constraint that `'0 == '$a`. -3. Finally, we check for skolemization "leaks" – a -leak is basically any attempt to relate a skolemized region to another -skolemized region, or to any region that pre-existed the impl match. -The leak check is done by searching from the skolemized region to find +3. Finally, we check for placeholder "leaks" – a +leak is basically any attempt to relate a placeholder region to another +placeholder region, or to any region that pre-existed the impl match. +The leak check is done by searching from the placeholder region to find the set of regions that it is related to in any way. This is called the "taint" set. To pass the check, that set must consist *solely* of itself and region variables from the impl. If the taint set includes @@ -78,7 +78,7 @@ impl Foo<&'static isize> for StaticInt; We want the obligation `StaticInt : for<'a> Foo<&'a isize>` to be considered unsatisfied. The check begins just as before. `'a` is -skolemized to `'0` and the impl trait reference is instantiated to +placeholder to `'0` and the impl trait reference is instantiated to `Foo<&'static isize>`. When we relate those two, we get a constraint like `'static == '0`. This means that the taint set for `'0` is `{'0, 'static}`, which fails the leak check. @@ -111,16 +111,16 @@ Now let's say we have a obligation `Baz: for<'a> Foo<&'a isize>` and we match this impl. What obligation is generated as a result? We want to get `Baz: for<'a> Bar<&'a isize>`, but how does that happen? -After the matching, we are in a position where we have a skolemized +After the matching, we are in a position where we have a placeholder substitution like `X => &'0 isize`. If we apply this substitution to the impl obligations, we get `F : Bar<&'0 isize>`. Obviously this is not -directly usable because the skolemized region `'0` cannot leak out of +directly usable because the placeholder region `'0` cannot leak out of our computation. What we do is to create an inverse mapping from the taint set of `'0` back to the original bound region (`'a`, here) that `'0` resulted from. (This is done in `higher_ranked::plug_leaks`). We know that the -leak check passed, so this taint set consists solely of the skolemized +leak check passed, so this taint set consists solely of the placeholder region itself plus various intermediate region variables. We then walk the trait-reference and convert every region in that taint set back to a late-bound region, so in this case we'd wind up with From 695728741e87065800dd03eff3b398040cfc8aa9 Mon Sep 17 00:00:00 2001 From: csmoe Date: Thu, 25 Oct 2018 11:01:08 +0800 Subject: [PATCH 377/648] replace bound region with placeholder --- src/appendix/glossary.md | 2 +- src/borrow_check/region_inference.md | 15 +++++++-------- src/traits/hrtb.md | 14 +++++++------- 3 files changed, 15 insertions(+), 16 deletions(-) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 04d35777f..82d6f9a21 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -53,7 +53,7 @@ rib | a data structure in the name resolver that keeps trac sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. sigil | like a keyword but composed entirely of non-alphanumeric tokens. For example, `&` is a sigil for references. -placeholder | a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)`) as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on placeholder and universes](../borrow_check/region_inference.html#placeholder) for more details. +placeholder | **NOTE: skolemization is deprecated by placeholder** a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)`) as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on placeholder and universes](../borrow_check/region_inference.html#placeholder) for more details. soundness | soundness is a technical term in type theory. Roughly, if a type system is sound, then if a program type-checks, it is type-safe; i.e. I can never (in safe rust) force a value into a variable of the wrong type. (see "completeness"). span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. substs | the substitutions for a given generic type or item (e.g. the `i32`, `u32` in `HashMap`) diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index 617133000..c754a5000 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -80,8 +80,8 @@ The kinds of region elements are as follows: to the remainder of program execution after this function returns. - There is an element `!1` for each placeholder region `!1`. This corresponds (intuitively) to some unknown set of other elements – - for details on placeholder, see the section - [placeholder and universes](#placeholder). + for details on placeholders, see the section + [placeholders and universes](#placeholder). ## Causal tracking @@ -117,7 +117,7 @@ for its argument, and `bar` wants to be given a function that that accepts **any** reference (so it can call it with something on its stack, for example). But *how* do we reject it and *why*? -### Subtyping and Placeholder +### Subtyping and Placeholders When we type-check `main`, and in particular the call `bar(foo)`, we are going to wind up with a subtyping relationship like this one: @@ -129,8 +129,7 @@ the type of `foo` the type `bar` expects ``` We handle this sort of subtyping by taking the variables that are -bound in the supertype and **placeholder** them: this means that we -replace them with +bound in the supertype and replace them with [universally quantified](../appendix/background.html#quantified) representatives, written like `!1`. We call these regions "placeholder regions" – they represent, basically, "some unknown region". @@ -198,7 +197,7 @@ fn bar<'a, T>(t: &'a T) { ``` Here, the name `'b` is not part of the root universe. Instead, when we -"enter" into this `for<'b>` (e.g., by placeholder it), we will create +"enter" into this `for<'b>` (e.g., by replace it with a placeholder), we will create a child universe of the root, let's call it U1: ```text @@ -411,7 +410,7 @@ for<'a> fn(&'a u32, &'a u32) for<'b, 'c> fn(&'b u32, &'c u32) ``` -Here we would placeholer the supertype, as before, yielding: +Here we would replace the bound region in the supertype with a placeholder, as before, yielding: ```text for<'a> fn(&'a u32, &'a u32) @@ -476,7 +475,7 @@ an error. That's good: the problem is that we've gone from a fn that promises to return one of its two arguments, to a fn that is promising to return the first one. That is unsound. Let's see how it plays out. -First, we placeholder the supertype: +First, we replace the bound region in the supertype with a placeholder: ```text for<'a> fn(&'a u32, &'a u32) -> &'a u32 diff --git a/src/traits/hrtb.md b/src/traits/hrtb.md index 9986bbb16..b56275881 100644 --- a/src/traits/hrtb.md +++ b/src/traits/hrtb.md @@ -36,20 +36,20 @@ to the subtyping for higher-ranked types (which is described [here][hrsubtype] and also in a [paper by SPJ]. If you wish to understand higher-ranked subtyping, we recommend you read the paper). There are a few parts: -**TODO**: We should define _placeholder_. - -1. _Skolemize_ the obligation. -2. Match the impl against the placeholder obligation. +1. replace bound regions in the obligation with placeholders. +2. Match the impl against the [placeholder] obligation. 3. Check for _placeholder leaks_. +[placeholder]: ../appendix/glossary.html#appendix-c-glossary [hrsubtype]: https://github.com/rust-lang/rust/tree/master/src/librustc/infer/higher_ranked/README.md [paper by SPJ]: http://research.microsoft.com/en-us/um/people/simonpj/papers/higher-rank/ So let's work through our example. 1. The first thing we would do is to -placeholder the obligation, yielding `AnyInt : Foo<&'0 isize>` (here `'0` -represents placeholder region #0). Note that we now have no quantifiers; +replace the bound region in the obligation with a placeholder, yielding +`AnyInt : Foo<&'0 isize>` (here `'0` represents placeholder region #0). +Note that we now have no quantifiers; in terms of the compiler type, this changes from a `ty::PolyTraitRef` to a `TraitRef`. We would then create the `TraitRef` from the impl, using fresh variables for it's bound regions (and thus getting @@ -78,7 +78,7 @@ impl Foo<&'static isize> for StaticInt; We want the obligation `StaticInt : for<'a> Foo<&'a isize>` to be considered unsatisfied. The check begins just as before. `'a` is -placeholder to `'0` and the impl trait reference is instantiated to +replaced with a placeholder `'0` and the impl trait reference is instantiated to `Foo<&'static isize>`. When we relate those two, we get a constraint like `'static == '0`. This means that the taint set for `'0` is `{'0, 'static}`, which fails the leak check. From 8cbf135744e84a84d7e4c8f0e6d20fce7cce85e0 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 25 Oct 2018 11:33:30 +0800 Subject: [PATCH 378/648] Update src/borrow_check/region_inference.md Co-Authored-By: csmoe --- src/borrow_check/region_inference.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index c754a5000..0484b14d6 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -129,7 +129,7 @@ the type of `foo` the type `bar` expects ``` We handle this sort of subtyping by taking the variables that are -bound in the supertype and replace them with +bound in the supertype and replacing them with [universally quantified](../appendix/background.html#quantified) representatives, written like `!1`. We call these regions "placeholder regions" – they represent, basically, "some unknown region". From bd8d3f5f30676af7b3513416b1b5cd9bfb7fc1a3 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 25 Oct 2018 11:33:33 +0800 Subject: [PATCH 379/648] Update src/borrow_check/region_inference.md Co-Authored-By: csmoe --- src/borrow_check/region_inference.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index 0484b14d6..95c2bc804 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -197,7 +197,7 @@ fn bar<'a, T>(t: &'a T) { ``` Here, the name `'b` is not part of the root universe. Instead, when we -"enter" into this `for<'b>` (e.g., by replace it with a placeholder), we will create +"enter" into this `for<'b>` (e.g., by replacing it with a placeholder), we will create a child universe of the root, let's call it U1: ```text From e35e081b521209148ac833af02b543fb5c7da344 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 25 Oct 2018 11:33:38 +0800 Subject: [PATCH 380/648] Update src/traits/hrtb.md Co-Authored-By: csmoe --- src/traits/hrtb.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/traits/hrtb.md b/src/traits/hrtb.md index b56275881..8b3a9f649 100644 --- a/src/traits/hrtb.md +++ b/src/traits/hrtb.md @@ -36,7 +36,7 @@ to the subtyping for higher-ranked types (which is described [here][hrsubtype] and also in a [paper by SPJ]. If you wish to understand higher-ranked subtyping, we recommend you read the paper). There are a few parts: -1. replace bound regions in the obligation with placeholders. +1. Replace bound regions in the obligation with placeholders. 2. Match the impl against the [placeholder] obligation. 3. Check for _placeholder leaks_. From 4196710286c120e1ba9f3e3fcbabdf96307b11b0 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 25 Oct 2018 11:33:44 +0800 Subject: [PATCH 381/648] Update src/traits/caching.md Co-Authored-By: csmoe --- src/traits/caching.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/traits/caching.md b/src/traits/caching.md index 7978306d1..c963aafc4 100644 --- a/src/traits/caching.md +++ b/src/traits/caching.md @@ -37,7 +37,7 @@ we would [confirm] `ImplCandidate(22)`, which would (as a side-effect) unify [confirm]: ./resolution.html#confirmation Now, at some later time, we might come along and see a `usize : -Foo<$u>`. When placeholder, this would yield `usize : Foo<$0>`, just as +Foo<$u>`. When replaced with a placeholder, this would yield `usize : Foo<$0>`, just as before, and hence the cache lookup would succeed, yielding `ImplCandidate(22)`. We would confirm `ImplCandidate(22)` which would (as a side-effect) unify `$u` with `isize`. From 34c9d3fc773c47ac90191ba66c692fb9991c9bec Mon Sep 17 00:00:00 2001 From: Nikita Popov Date: Fri, 26 Oct 2018 14:34:09 +0200 Subject: [PATCH 382/648] Update TypeVariants to TyKind This has been renamed in https://github.com/rust-lang/rust/pull/53581. --- src/ty.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ty.md b/src/ty.md index 44017dd5b..fea9afbeb 100644 --- a/src/ty.md +++ b/src/ty.md @@ -99,7 +99,7 @@ fn test_type<'tcx>(ty: Ty<'tcx>) { ``` The `sty` field (the origin of this name is unclear to me; perhaps -structural type?) is of type `TypeVariants<'tcx>`, which is an enum +structural type?) is of type `TyKind<'tcx>`, which is an enum defining all of the different kinds of types in the compiler. > N.B. inspecting the `sty` field on types during type inference can be From 5c67241e1ab4383035cec2a0faa17c97ae51c007 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Wed, 24 Oct 2018 21:15:11 -0500 Subject: [PATCH 383/648] Group logic-related things under one section I found the layout to be a bit inconsistent before. This groups anything that touches logic rules under "Lowering to logic". This might be crowding too many things under that section, but it makes more sense to me overall. --- src/SUMMARY.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 213a645ab..9308b7d51 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -38,12 +38,12 @@ - [Goals and clauses](./traits/goals-and-clauses.md) - [Equality and associated types](./traits/associated-types.md) - [Implied bounds](./traits/implied-bounds.md) + - [Well-formedness checking](./traits/wf.md) - [Region constraints](./traits/regions.md) + - [The lowering module in rustc](./traits/lowering-module.md) + - [Lowering rules](./traits/lowering-rules.md) - [Canonical queries](./traits/canonical-queries.md) - [Canonicalization](./traits/canonicalization.md) - - [Lowering rules](./traits/lowering-rules.md) - - [The lowering module in rustc](./traits/lowering-module.md) - - [Well-formedness checking](./traits/wf.md) - [The SLG solver](./traits/slg.md) - [An Overview of Chalk](./traits/chalk-overview.md) - [Bibliography](./traits/bibliography.md) From 9ae8422607397558bd3d397ac3c01a915db24c56 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Wed, 24 Oct 2018 21:22:32 -0500 Subject: [PATCH 384/648] Reflect traits chapter structure in index key ideas --- src/traits/index.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/traits/index.md b/src/traits/index.md index 053a26bab..6a0e560a2 100644 --- a/src/traits/index.md +++ b/src/traits/index.md @@ -19,18 +19,18 @@ Trait solving is based around a few key ideas: describes the precise form of rules we use, and [lowering rules](./lowering-rules.html) gives the complete set of lowering rules in a more reference-like form. + - [Lazy normalization](./associated-types.html), which is the + technique we use to accommodate associated types when figuring out + whether types are equal. + - [Region constraints](./regions.html), which are accumulated + during trait solving but mostly ignored. This means that trait + solving effectively ignores the precise regions involved, always – + but we still remember the constraints on them so that those + constraints can be checked by the type checker. - [Canonical queries](./canonical-queries.html), which allow us to solve trait problems (like "is `Foo` implemented for the type `Bar`?") once, and then apply that same result independently in many different inference contexts. -- [Lazy normalization](./associated-types.html), which is the - technique we use to accommodate associated types when figuring out - whether types are equal. -- [Region constraints](./regions.html), which are accumulated - during trait solving but mostly ignored. This means that trait - solving effectively ignores the precise regions involved, always – - but we still remember the constraints on them so that those - constraints can be checked by thet type checker. Note: this is not a complete list of topics. See the sidebar for more. From c4708f8039f325b2f22cb4cc94744615e4529418 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Wed, 24 Oct 2018 21:22:57 -0500 Subject: [PATCH 385/648] Traits: Improve index layout, add chalk blurb --- src/traits/index.md | 51 +++++++++++++++++++++++++++++---------------- 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/src/traits/index.md b/src/traits/index.md index 6a0e560a2..2e3e6715c 100644 --- a/src/traits/index.md +++ b/src/traits/index.md @@ -1,17 +1,26 @@ # Trait solving (new-style) -🚧 This chapter describes "new-style" trait solving. This is still in the -[process of being implemented][wg]; this chapter serves as a kind of -in-progress design document. If you would prefer to read about how the -current trait solver works, check out -[this other chapter](./resolution.html). (By the way, if you -would like to help in hacking on the new solver, you will find -instructions for getting involved in the -[Traits Working Group tracking issue][wg].) 🚧 +> 🚧 This chapter describes "new-style" trait solving. This is still in the +> [process of being implemented][wg]; this chapter serves as a kind of +> in-progress design document. If you would prefer to read about how the +> current trait solver works, check out +> [this other chapter](./resolution.html). 🚧 +> +> By the way, if you would like to help in hacking on the new solver, you will +> find instructions for getting involved in the +> [Traits Working Group tracking issue][wg]. [wg]: https://github.com/rust-lang/rust/issues/48416 -Trait solving is based around a few key ideas: +The new-style trait solver is based on the work done in [chalk][chalk]. Chalk +recasts Rust's trait system explicitly in terms of logic programming. It does +this by "lowering" Rust code into a kind of logic program we can then execute +queries against. + +You can read more about chalk itself in the +[Overview of Chalk](./chalk-overview.md) section. + +Trait solving in rustc is based around a few key ideas: - [Lowering to logic](./lowering-to-logic.html), which expresses Rust traits in terms of standard logical terms. @@ -32,17 +41,23 @@ Trait solving is based around a few key ideas: `Bar`?") once, and then apply that same result independently in many different inference contexts. -Note: this is not a complete list of topics. See the sidebar for more. +> This is not a complete list of topics. See the sidebar for more. +## Ongoing work The design of the new-style trait solving currently happens in two places: -* The [chalk][chalk] repository is where we experiment with new ideas and - designs for the trait system. It basically consists of a unit testing framework - for the correctness and feasibility of the logical rules defining the new-style - trait system. It also provides the [`chalk_engine`][chalk_engine] crate, which - defines the new-style trait solver used both in the unit testing framework and - in rustc. -* Once we are happy with the logical rules, we proceed to implementing them in - rustc. This mainly happens in [`librustc_traits`][librustc_traits]. + +**chalk**. The [chalk][chalk] repository is where we experiment with new ideas +and designs for the trait system. It primarily consists of two parts: +* a unit testing framework + for the correctness and feasibility of the logical rules defining the + new-style trait system. +* the [`chalk_engine`][chalk_engine] crate, which + defines the new-style trait solver used both in the unit testing framework + and in rustc. + +**rustc**. Once we are happy with the logical rules, we proceed to +implementing them in rustc. This mainly happens in +[`librustc_traits`][librustc_traits]. [chalk]: https://github.com/rust-lang-nursery/chalk [chalk_engine]: https://github.com/rust-lang-nursery/chalk/tree/master/chalk-engine From 99a23f84a7b1276ecc280da4ae4aba11f73f5edf Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Wed, 24 Oct 2018 21:31:53 -0500 Subject: [PATCH 386/648] Associated types: Mention "lazy normalization" somewhere --- src/traits/associated-types.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/src/traits/associated-types.md b/src/traits/associated-types.md index 1fffa3ff8..23da15228 100644 --- a/src/traits/associated-types.md +++ b/src/traits/associated-types.md @@ -5,7 +5,7 @@ associated types. The full system consists of several moving parts, which we will introduce one by one: - Projection and the `Normalize` predicate -- Skolemization +- Placeholder associated type projections - The `ProjectionEq` predicate - Integration with unification @@ -106,9 +106,10 @@ placeholder associated types (see the `TypeName` enum declared in So far we have seen two ways to answer the question of "When can we consider an associated type projection equal to another type?": -- the `Normalize` predicate could be used to transform associated type - projections when we knew which impl was applicable; -- **placeholder** associated types can be used when we don't. +- the `Normalize` predicate could be used to transform projections when we + knew which impl applied; +- **placeholder** associated types can be used when we don't. This is also + known as **lazy normalization**. We now introduce the `ProjectionEq` predicate to bring those two cases together. The `ProjectionEq` predicate looks like so: From fbb3ec6744e3a010f37e88c260949de6485575b8 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Wed, 24 Oct 2018 21:32:04 -0500 Subject: [PATCH 387/648] Associated types: Break up text for readability --- src/traits/associated-types.md | 59 +++++++++++++++++----------------- 1 file changed, 30 insertions(+), 29 deletions(-) diff --git a/src/traits/associated-types.md b/src/traits/associated-types.md index 23da15228..d35fb71e1 100644 --- a/src/traits/associated-types.md +++ b/src/traits/associated-types.md @@ -14,11 +14,11 @@ which we will introduce one by one: When a trait defines an associated type (e.g., [the `Item` type in the `IntoIterator` trait][intoiter-item]), that type can be referenced by the user using an **associated type -projection** like ` as IntoIterator>::Item`. (Often, -though, people will use the shorthand syntax `T::Item` – presently, -that syntax is expanded during -["type collection"](../type-checking.html) into the explicit form, -though that is something we may want to change in the future.) +projection** like ` as IntoIterator>::Item`. + +> Often, people will use the shorthand syntax `T::Item`. Presently, that +> syntax is expanded during ["type collection"](../type-checking.html) into the +> explicit form, though that is something we may want to change in the future. [intoiter-item]: https://doc.rust-lang.org/nightly/core/iter/trait.IntoIterator.html#associatedtype.Item @@ -41,10 +41,11 @@ IntoIterator>::Item` to just `u32`. In this case, the projection was a "monomorphic" one – that is, it did not have any type parameters. Monomorphic projections are special -because they can **always** be fully normalized – but often we can -normalize other associated type projections as well. For example, -` as IntoIterator>::Item` (where `?T` is an inference -variable) can be normalized to just `?T`. +because they can **always** be fully normalized. + +Often, we can normalize other associated type projections as well. For +example, ` as IntoIterator>::Item`, where `?T` is an inference +variable, can be normalized to just `?T`. In our logic, normalization is defined by a predicate `Normalize`. The `Normalize` clauses arise only from @@ -60,9 +61,8 @@ forall { where in this case, the one `Implemented` condition is always true. -(An aside: since we do not permit quantification over traits, this is -really more like a family of program clauses, one for each associated -type.) +> Since we do not permit quantification over traits, this is really more like +> a family of program clauses, one for each associated type. We could apply that rule to normalize either of the examples that we've seen so far. @@ -76,17 +76,18 @@ normalized. For example, consider this function: fn foo(...) { ... } ``` -In this context, how would we normalize the type `T::Item`? Without -knowing what `T` is, we can't really do so. To represent this case, we -introduce a type called a **placeholder associated type -projection**. This is written like so `(IntoIterator::Item)`. You -may note that it looks a lot like a regular type (e.g., `Option`), -except that the "name" of the type is `(IntoIterator::Item)`. This is -not an accident: placeholder associated type projections work just like -ordinary types like `Vec` when it comes to unification. That is, -they are only considered equal if (a) they are both references to the -same associated type, like `IntoIterator::Item` and (b) their type -arguments are equal. +In this context, how would we normalize the type `T::Item`? + +Without knowing what `T` is, we can't really do so. To represent this case, +we introduce a type called a **placeholder associated type projection**. This +is written like so: `(IntoIterator::Item)`. + +You may note that it looks a lot like a regular type (e.g., `Option`), +except that the "name" of the type is `(IntoIterator::Item)`. This is not an +accident: placeholder associated type projections work just like ordinary +types like `Vec` when it comes to unification. That is, they are only +considered equal if (a) they are both references to the same associated type, +like `IntoIterator::Item` and (b) their type arguments are equal. Placeholder associated types are never written directly by the user. They are used internally by the trait system only, as we will see @@ -152,16 +153,16 @@ might just fail, in which case we get back `Err(NoSolution)`. This would happen, for example, if we tried to unify `u32` and `i32`. The key point is that, on success, unification can also give back to -us a set of subgoals that still remain to be proven (it can also give +us a set of subgoals that still remain to be proven. (It can also give back region constraints, but those are not relevant here). -Whenever unification encounters an (un-placeholder!) associated type +Whenever unification encounters a non-placeholder associated type projection P being equated with some other type T, it always succeeds, but it produces a subgoal `ProjectionEq(P = T)` that is propagated back up. Thus it falls to the ordinary workings of the trait system to process that constraint. -(If we unify two projections P1 and P2, then unification produces a -variable X and asks us to prove that `ProjectionEq(P1 = X)` and -`ProjectionEq(P2 = X)`. That used to be needed in an older system to -prevent cycles; I rather doubt it still is. -nmatsakis) +> If we unify two projections P1 and P2, then unification produces a +> variable X and asks us to prove that `ProjectionEq(P1 = X)` and +> `ProjectionEq(P2 = X)`. (That used to be needed in an older system to +> prevent cycles; I rather doubt it still is. -nmatsakis) From c41019addab167642d743eebca4ce4a4a0865451 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Wed, 24 Oct 2018 21:24:53 -0500 Subject: [PATCH 388/648] Add status of regions --- src/traits/regions.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/traits/regions.md b/src/traits/regions.md index baa3582b6..4657529dc 100644 --- a/src/traits/regions.md +++ b/src/traits/regions.md @@ -1,3 +1,9 @@ # Region constraints -*to be written* +*To be written.* + +Chalk does not have the concept of region constraints, and as of this +writing, work on rustc was not far enough to worry about them. + +In the meantime, you can read about region constraints in the +[type inference](../type-inference.html#region-constraints) section. From 2de4dc7a12c2945454d958672c908b68d3495a75 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Wed, 24 Oct 2018 21:30:33 -0500 Subject: [PATCH 389/648] Get excited --- src/traits/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/traits/index.md b/src/traits/index.md index 2e3e6715c..84f812394 100644 --- a/src/traits/index.md +++ b/src/traits/index.md @@ -8,7 +8,7 @@ > > By the way, if you would like to help in hacking on the new solver, you will > find instructions for getting involved in the -> [Traits Working Group tracking issue][wg]. +> [Traits Working Group tracking issue][wg]! [wg]: https://github.com/rust-lang/rust/issues/48416 From 74e2af231be0f1ff3350271b3624568c4244ef2d Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Fri, 26 Oct 2018 13:42:14 -0500 Subject: [PATCH 390/648] Put "well-formedness checking" under "lowering rules" This was also intended to be a reference chapter, according to @scalexm. --- src/SUMMARY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 9308b7d51..efe963c3f 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -38,10 +38,10 @@ - [Goals and clauses](./traits/goals-and-clauses.md) - [Equality and associated types](./traits/associated-types.md) - [Implied bounds](./traits/implied-bounds.md) - - [Well-formedness checking](./traits/wf.md) - [Region constraints](./traits/regions.md) - [The lowering module in rustc](./traits/lowering-module.md) - [Lowering rules](./traits/lowering-rules.md) + - [Well-formedness checking](./traits/wf.md) - [Canonical queries](./traits/canonical-queries.md) - [Canonicalization](./traits/canonicalization.md) - [The SLG solver](./traits/slg.md) From 34af41dddb22bcdb594de0ccb9296ac459398b78 Mon Sep 17 00:00:00 2001 From: Tom Tromey Date: Fri, 26 Oct 2018 12:16:25 -0600 Subject: [PATCH 391/648] Document the rust-lldb directive Commit ac33b2e578de58016271012452e4f1be6af9d516 added a `rust-lldb` directive to restrict the lldb part of a debuginfo test to only versions of lldb that include the Rust plugin. --- src/tests/adding.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/tests/adding.md b/src/tests/adding.md index 1120315b9..580741ccf 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -158,6 +158,8 @@ source. this test; see also `ignore-gdb-version` * `min-lldb-version` specifies the minimum lldb version required for this test +* `rust-lldb` causes the lldb part of the test to only be run if the + lldb in use contains the Rust plugin * `no-system-llvm` causes the test to be ignored if the system llvm is used * `min-llvm-version` specifies the minimum llvm version required for this test From 462b7c3558c3ba85dd001e8c1239cfdcd4165b42 Mon Sep 17 00:00:00 2001 From: scalexm Date: Wed, 17 Oct 2018 20:15:19 +0200 Subject: [PATCH 392/648] Write well-formedness checking chapter --- src/traits/wf.md | 268 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 263 insertions(+), 5 deletions(-) diff --git a/src/traits/wf.md b/src/traits/wf.md index c002b8cc1..cd0fb5564 100644 --- a/src/traits/wf.md +++ b/src/traits/wf.md @@ -1,11 +1,269 @@ # Well-formedness checking -This chapter is mostly *to be written*. WF checking, in short, has the -job of checking that the various declarations in a Rust program are -well-formed. This is the basis for implied bounds, and partly for that -reason, this checking can be surprisingly subtle! (For example, we +WF checking has the job of checking that the various declarations in a Rust +program are well-formed. This is the basis for implied bounds, and partly for +that reason, this checking can be surprisingly subtle! For example, we have to be sure that each impl proves the WF conditions declared on -the trait.) +the trait. +For each declaration in a Rust program, we will generate a logical goal and try +to prove it using the lowered rules we described in the +[lowering rules](./lowering-rules.md) chapter. If we are able to prove it, we +say that the construct is well-formed. If not, we report an error to the user. +Well-formedness checking happens in the [`src/rules/wf.rs`][wf] module in +chalk. After you have read this chapter, you may find useful to see an +extended set of examples in the [`src/rules/wf/test.rs`][wf_test] submodule. +The new-style WF checking has not been implemented in rustc yet. + +[wf]: https://github.com/rust-lang-nursery/chalk/blob/master/src/rules/wf.rs +[wf_test]: https://github.com/rust-lang-nursery/chalk/blob/master/src/rules/wf/test.rs + +We give here a complete reference of the generated goals for each Rust +declaration. + +In addition with the notations introduced in the chapter about +lowering rules, we'll introduce another notation: when WF checking a +declaration, we'll often have to prove that all types that appear are +well-formed, except type parameters that we always assume to be WF. Hence, +we'll use the following notation: for a type `SomeType<...>`, we denote +`InputTypes(SomeType<...>)` the set of all non-parameter types appearing in +`SomeType<...>`, including `SomeType<...>` itself. + +Examples: +* `InputTypes((u32, f32)) = [u32, f32, (u32, f32)]` +* `InputTypes(Box) = [Box]` +* `InputTypes(Box>) = [Box, Box>]` + +We may naturally extend the `InputTypes` notation to where clauses, for example +`InputTypes(A0: Trait)` is the union of `InputTypes(A0)`, +`InputTypes(A1)`, ..., `InputTypes(An)`. + +# Type definitions + +Given a general type definition: +```rust,ignore +struct Type where WC_type { + field1: A1, + ... + fieldn: An, +} +``` + +we generate the following goal: +``` +forall { + if (FromEnv(WC_type)) { + WellFormed(InputTypes(WC_type)) && + WellFormed(InputTypes(A1)) && + ... + WellFormed(InputTypes(An)) + } +} +``` + +which in English gives: assuming that the where clauses defined on the type +hold, prove that every type appearing in the type definition is well-formed. + +Some examples: +```rust,ignore +struct OnlyClone where T: Clone { + clonable: T, +} +// The only types appearing are type parameters: we have nothing to check, +// the type definition is well-formed. + +struct Foo where T: Clone { + foo: OnlyClone, +} +// The only non-parameter type which appears in this definition is +// `OnlyClone`. The generated goal is the following: +// ``` +// forall { +// if (FromEnv(T: Clone)) { +// WellFormed(OnlyClone) +// } +// } +// ``` +// which is provable. + +struct Bar where OnlyClone: Debug { + bar: i32, +} +// The only non-parameter type which appears in this definition is +// `OnlyClone`. The generated goal is the following: +// ``` +// forall { +// if (FromEnv(OnlyClone: Debug)) { +// WellFormed(OnlyClone) +// } +// } +// ``` +// which is not provable since `WellFormed(OnlyClone)` requires proving +// `Implemented(T: Clone)`, and we are unable to prove that for an unknown `T`. +// Hence, this type definition is considered illegal. An additional +// `where T: Clone` would make it legal. +``` + +# Trait definitions + +Given a general trait definition: +```rust,ignore +trait Trait where WC_trait { + type Assoc: Bounds_assoc where WC_assoc; +} +``` + +we generate the following goal: +```text +forall { + if (FromEnv(WC_trait)) { + WellFormed(InputTypes(WC_trait)) && + + forall { + if (FromEnv(WC_assoc)) { + WellFormed(InputTypes(Bounds_assoc)) && + WellFormed(InputTypes(WC_assoc)) + } + } + } +} +``` + +There is not much to verify in a trait definition. We just want +to prove that the types appearing in the trait definition are well-formed, +under the assumption that the different where clauses hold. + +Some examples: +```rust,ignore +struct OnlyClone { ... } + +trait Foo where T: Clone, OnlyClone: Debug { + ... +} +// The only non-parameter type which appears in this definition is +// `OnlyClone`. The generated goal is the following: +// ``` +// forall { +// if (FromEnv(T: Clone), FromEnv(OnlyClone: Debug)) { +// WellFormed(OnlyClone) +// } +// } +// ``` +// which is provable thanks to the `FromEnv(T: Clone)` assumption. + +trait Bar { + type Assoc: From>; +} +// The only non-parameter type which appears in this definition is +// `OnlyClone`. The generated goal is the following: +// forall { +// WellFormed(OnlyClone) +// } +// which is not provable, hence the trait definition is considered illegal. + +trait Baz { + type Assoc: From> where T: Clone; +} +// The generated goal is now: +// forall { +// if (FromEnv(T: Clone)) { +// WellFormed(OnlyClone) +// } +// } +// which is now provable. +``` + +# Impls + +Now we give ourselves a general impl for the trait defined above: +```rust,ignore +impl Trait for SomeType where WC_impl { + type Assoc = SomeValue where WC_assoc; +} +``` + +Note that here, `WC_assoc` are the same where clauses as those defined on the +associated type definition in the trait declaration, *except* that type +parameters from the trait are substituted with values provided by the impl +(see example below). You cannot add new where clauses. You may omit to write +the where clauses if you want to emphasize the fact that you are actually not +relying on them. + +Some examples to illustrate that: +```rust,ignore +trait Foo { + type Assoc where T: Clone; +} + +struct OnlyClone { ... } + +impl Foo> for () { + // We substitute type parameters from the trait by the ones provided + // by the impl, that is instead of having a `T: Clone` where clause, + // we have an `Option: Clone` one. + type Assoc = OnlyClone> where Option: Clone; +} + +impl Foo for i32 { + // I'm not using the `T: Clone` where clause from the trait, so I can + // omit it. + type Assoc = u32; +} + +impl Foo for f32 { + type Assoc = OnlyClone> where Option: Clone; + // ^^^^^^^^^^^^^^^^^^^^^^ + // this where clause does not exist + // on the original trait decl: illegal +} +``` + +So where clauses on associated types work *exactly* like where clauses on +trait methods: in an impl, we must substitute the parameters from the traits +with values provided by the impl, we may omit them if we don't need them, and +we cannot add new where clauses. + +Now let's see the generated goal for this general impl: +``` +forall { + if (FromEnv(WC_impl), FromEnv(InputTypes(SomeType))) { + WellFormed(SomeType: Trait) && + WellFormed(InputTypes(WC_impl)) && + + forall { + if (FromEnv(WC_assoc)) { + WellFormed(SomeValue: Bounds_assoc) && + WellFormed(InputTypes(SomeValue)) + } + } + } +} +``` + +Here is the most complex goal. As always, a first thing is that assuming that +the various where clauses hold, we prove that every type appearing in the impl +is well-formed, ***except*** types appearing in the receiver type +`SomeType`. Instead, we *assume* that those types are well-formed +(hence the `if (FromEnv(InputTypes(SomeType)))` condition). This is +part of the implied bounds proposal, so that we can rely on the bounds +written on the definition of the `SomeType` type (and that we don't +need to repeat those bounds). + +Next, assuming that the where clauses on the impl `WC_impl` hold and that the +input types of `SomeType` are well-formed, we prove that +`WellFormed(SomeType: Trait)` hold. That is, we want to prove +that `SomeType` verify all the where clauses that might transitively +come from the `Trait` definition (see +[this subsection](./implied-bounds#co-inductiveness-of-wellformed)). + +Lastly, assuming that the where clauses on the associated type `WC_assoc` hold, +we prove that `WellFormed(SomeValue: Bounds_assoc)` hold. Again, we are +not only proving `Implemented(SomeValue: Bounds_assoc)`, but also +all the facts that might transitively come from `Bounds_assoc`. This is because +we allow the use of implied bounds on associated types: if we have +`FromEnv(SomeType: Trait)` in our environment, the lowering rules +chapter indicates that we are able to deduce +`FromEnv(::Assoc: Bounds_assoc)` without knowing what the +precise value of `::Assoc` is. From b5dff2ed5fc7ba2d7966ffd3c9275fd690a50236 Mon Sep 17 00:00:00 2001 From: scalexm Date: Mon, 29 Oct 2018 15:54:37 +0100 Subject: [PATCH 393/648] Add some examples for impls --- src/traits/wf.md | 213 ++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 192 insertions(+), 21 deletions(-) diff --git a/src/traits/wf.md b/src/traits/wf.md index cd0fb5564..bd1188d87 100644 --- a/src/traits/wf.md +++ b/src/traits/wf.md @@ -73,7 +73,9 @@ struct OnlyClone where T: Clone { } // The only types appearing are type parameters: we have nothing to check, // the type definition is well-formed. +``` +```rust,ignore struct Foo where T: Clone { foo: OnlyClone, } @@ -87,23 +89,28 @@ struct Foo where T: Clone { // } // ``` // which is provable. +``` -struct Bar where OnlyClone: Debug { +```rust,ignore +struct Bar where ::Item: Debug { bar: i32, } -// The only non-parameter type which appears in this definition is -// `OnlyClone`. The generated goal is the following: +// The only non-parameter types which appear in this definition are +// `::Item` and `i32`. The generated goal is the following: // ``` // forall { -// if (FromEnv(OnlyClone: Debug)) { -// WellFormed(OnlyClone) +// if (FromEnv(::Item: Debug)) { +// WellFormed(::Item) && +// WellFormed(i32) // } // } // ``` -// which is not provable since `WellFormed(OnlyClone)` requires proving -// `Implemented(T: Clone)`, and we are unable to prove that for an unknown `T`. +// which is not provable since `WellFormed(::Item)` requires +// proving `Implemented(T: Iterator)`, and we are unable to prove that for an +// unknown `T`. +// // Hence, this type definition is considered illegal. An additional -// `where T: Clone` would make it legal. +// `where T: Iterator` would make it legal. ``` # Trait definitions @@ -137,41 +144,47 @@ under the assumption that the different where clauses hold. Some examples: ```rust,ignore -struct OnlyClone { ... } - -trait Foo where T: Clone, OnlyClone: Debug { +trait Foo where T: Iterator, ::Item: Debug { ... } // The only non-parameter type which appears in this definition is -// `OnlyClone`. The generated goal is the following: +// `::Item`. The generated goal is the following: // ``` // forall { -// if (FromEnv(T: Clone), FromEnv(OnlyClone: Debug)) { -// WellFormed(OnlyClone) +// if (FromEnv(T: Iterator), FromEnv(::Item: Debug)) { +// WellFormed(::Item) // } // } // ``` -// which is provable thanks to the `FromEnv(T: Clone)` assumption. +// which is provable thanks to the `FromEnv(T: Iterator)` assumption. +``` +```rust,ignore trait Bar { - type Assoc: From>; + type Assoc: From<::Item>; } // The only non-parameter type which appears in this definition is -// `OnlyClone`. The generated goal is the following: +// `::Item`. The generated goal is the following: +// ``` // forall { -// WellFormed(OnlyClone) +// WellFormed(::Item) // } +// ``` // which is not provable, hence the trait definition is considered illegal. +``` +```rust,ignore trait Baz { - type Assoc: From> where T: Clone; + type Assoc: From<::Item> where T: Iterator; } // The generated goal is now: +// ``` // forall { -// if (FromEnv(T: Clone)) { -// WellFormed(OnlyClone) +// if (FromEnv(T: Iterator)) { +// WellFormed(::Item) // } // } +// ``` // which is now provable. ``` @@ -267,3 +280,161 @@ we allow the use of implied bounds on associated types: if we have chapter indicates that we are able to deduce `FromEnv(::Assoc: Bounds_assoc)` without knowing what the precise value of `::Assoc` is. + +Some examples for the generated goal: +```rust,ignore +trait Copy { } +// `WellFormed(Self: Copy) :- Implemented(Self: Copy).` + +trait Partial where Self: Copy { } +// ``` +// WellFormed(Self: Partial) :- +// Implemented(Self: Partial) && +// WellFormed(Self: Copy). +// ``` + +trait Complete where Self: Partial { } +// ``` +// WellFormed(Self: Complete) :- +// Implemented(Self: Complete) && +// WellFormed(Self: Partial). +// ``` + +impl Partial for T where T: Complete { } +// The generated goal is: +// ``` +// forall { +// if (FromEnv(T: Complete)) { +// WellFormed(T: Partial) +// } +// } +// ``` +// Then proving `WellFormed(T: Partial)` amounts to proving +// `Implemented(T: Partial)` and `Implemented(T: Copy)`. +// Both those facts can be deduced from the `FromEnv(T: Complete)` in our +// environment: this impl is legal. + +impl Complete for T { } +// The generated goal is: +// ``` +// forall { +// WellFormed(T: Complete) +// } +// ``` +// Then proving `WellFormed(T: Complete)` amounts to proving +// `Implemented(T: Complete)`, `Implemented(T: Partial)` and +// `Implemented(T: Copy)`. +// +// `Implemented(T: Complete)` can be proved thanks to the +// `impl Complete for T` blanket impl. +// +// `Implemented(T: Partial)` can be proved thanks to the +// `impl Partial for T where T: Complete` impl and because we know +// `T: Complete` holds. + +// However, `Implemented(T: Copy)` cannot be proved: the impl is illegal. +// An additional `where T: Copy` bound would be sufficient to make that impl +// legal. +``` + +```rust,ignore +trait Bar { } + +impl Bar for T where ::Item: Bar { } +// We have a non-parameter type appearing in the where clauses: +// `::Item`. The generated goal is: +// ``` +// forall { +// if (FromEnv(::Item: Bar)) { +// WellFormed(T: Bar) && +// WellFormed(::Item: Bar) +// } +// } +// ``` +// And `WellFormed(::Item: Bar)` is not provable: we'd need +// an additional `where T: Iterator` for example. +``` + +```rust,ignore +trait Foo { } + +trait Bar { + type Item: Foo; +} + +struct Stuff { } + +impl Bar for Stuff where T: Foo { + type Item = T; +} +// The generated goal is: +// ``` +// forall { +// if (FromEnv(T: Foo)) { +// WellFormed(T: Foo). +// } +// } +// ``` +// which is provable. +``` + +```rust,ignore +trait Debug { ... } +// `WellFormed(Self: Debug) :- Implemented(Self: Debug).` + +struct Box { ... } +impl Debug for Box where T: Debug { ... } + +trait PointerFamily { + type Pointer: Debug where T: Debug; +} +// `WellFormed(Self: PointerFamily) :- Implemented(Self: PointerFamily).` + +struct BoxFamily; + +impl PointerFamily for CowFamily { + type Pointer = Box where T: Debug; +} +// The generated goal is: +// ``` +// forall { +// WellFormed(CowFamily: PointerFamily) && +// +// if (FromEnv(T: Debug)) { +// WellFormed(Box: Debug) && +// WellFormed(Box) +// } +// } +// ``` +// `WellFormed(CowFamily: PointerFamily)` amounts to proving +// `Implemented(CowFamily: PointerFamily)`, which is ok thanks to our impl. +// +// `WellFormed(Box)` is always true (there are no where clauses on the +// `Box` type definition). +// +// Moreover, we have an `impl Debug for Box`, hence +// we can prove `WellFormed(Box: Debug)` and the impl is indeed legal. +``` + +```rust,ignore +trait Foo { + type Assoc; +} + +struct OnlyClone { ... } + +impl Foo for i32 { + type Assoc = OnlyClone; +} +// The generated goal is: +// ``` +// forall { +// WellFormed(i32: Foo) && +// WellFormed(OnlyClone) +// } +// ``` +// however `WellFormed(OnlyClone)` is not provable because it requires +// `Implemented(T: Clone)`. It would be tempting to just add a `where T: Clone` +// bound inside the `impl Foo for i32` block, however we saw that it was +// illegal to add where clauses that didn't come from the trait definition. +``` From 83652097f3f39c007318939355bf45b3fc6671de Mon Sep 17 00:00:00 2001 From: scalexm Date: Mon, 29 Oct 2018 18:58:13 +0100 Subject: [PATCH 394/648] Fix code blocks --- src/traits/wf.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/traits/wf.md b/src/traits/wf.md index bd1188d87..4cef9ec84 100644 --- a/src/traits/wf.md +++ b/src/traits/wf.md @@ -52,7 +52,7 @@ struct Type where WC_type { ``` we generate the following goal: -``` +```text forall { if (FromEnv(WC_type)) { WellFormed(InputTypes(WC_type)) && @@ -239,7 +239,7 @@ with values provided by the impl, we may omit them if we don't need them, and we cannot add new where clauses. Now let's see the generated goal for this general impl: -``` +```text forall { if (FromEnv(WC_impl), FromEnv(InputTypes(SomeType))) { WellFormed(SomeType: Trait) && From e20f283a1e44d3616f80dea2fa5a67282664ed72 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Mon, 29 Oct 2018 22:20:26 +0100 Subject: [PATCH 395/648] Fix a few things Co-Authored-By: scalexm --- src/traits/wf.md | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/src/traits/wf.md b/src/traits/wf.md index 4cef9ec84..ec391924a 100644 --- a/src/traits/wf.md +++ b/src/traits/wf.md @@ -23,8 +23,8 @@ The new-style WF checking has not been implemented in rustc yet. We give here a complete reference of the generated goals for each Rust declaration. -In addition with the notations introduced in the chapter about -lowering rules, we'll introduce another notation: when WF checking a +In addition to the notations introduced in the chapter about +lowering rules, we'll introduce another notation: when checking WF of a declaration, we'll often have to prove that all types that appear are well-formed, except type parameters that we always assume to be WF. Hence, we'll use the following notation: for a type `SomeType<...>`, we denote @@ -212,11 +212,11 @@ trait Foo { struct OnlyClone { ... } -impl Foo> for () { +impl Foo> for () { // We substitute type parameters from the trait by the ones provided // by the impl, that is instead of having a `T: Clone` where clause, - // we have an `Option: Clone` one. - type Assoc = OnlyClone> where Option: Clone; + // we have an `Option: Clone` one. + type Assoc = OnlyClone> where Option: Clone; } impl Foo for i32 { @@ -235,7 +235,7 @@ impl Foo for f32 { So where clauses on associated types work *exactly* like where clauses on trait methods: in an impl, we must substitute the parameters from the traits -with values provided by the impl, we may omit them if we don't need them, and +with values provided by the impl, we may omit them if we don't need them, but we cannot add new where clauses. Now let's see the generated goal for this general impl: @@ -255,7 +255,7 @@ forall { } ``` -Here is the most complex goal. As always, a first thing is that assuming that +Here is the most complex goal. As always, first, assuming that the various where clauses hold, we prove that every type appearing in the impl is well-formed, ***except*** types appearing in the receiver type `SomeType`. Instead, we *assume* that those types are well-formed @@ -269,7 +269,7 @@ input types of `SomeType` are well-formed, we prove that `WellFormed(SomeType: Trait)` hold. That is, we want to prove that `SomeType` verify all the where clauses that might transitively come from the `Trait` definition (see -[this subsection](./implied-bounds#co-inductiveness-of-wellformed)). +[this subsection](./implied-bounds.md#co-inductiveness-of-wellformed)). Lastly, assuming that the where clauses on the associated type `WC_assoc` hold, we prove that `WellFormed(SomeValue: Bounds_assoc)` hold. Again, we are @@ -284,6 +284,10 @@ precise value of `::Assoc` is. Some examples for the generated goal: ```rust,ignore trait Copy { } +// This is a program clause that comes from the trait definition above +// and that the trait solver can use for its reasonings. I'm just restating +// it here (and also the few other ones coming just after) so that we have +// them in mind. // `WellFormed(Self: Copy) :- Implemented(Self: Copy).` trait Partial where Self: Copy { } @@ -392,13 +396,13 @@ trait PointerFamily { struct BoxFamily; -impl PointerFamily for CowFamily { +impl PointerFamily for BoxFamily { type Pointer = Box where T: Debug; } // The generated goal is: // ``` // forall { -// WellFormed(CowFamily: PointerFamily) && +// WellFormed(BoxFamily: PointerFamily) && // // if (FromEnv(T: Debug)) { // WellFormed(Box: Debug) && @@ -406,8 +410,8 @@ impl PointerFamily for CowFamily { // } // } // ``` -// `WellFormed(CowFamily: PointerFamily)` amounts to proving -// `Implemented(CowFamily: PointerFamily)`, which is ok thanks to our impl. +// `WellFormed(BoxFamily: PointerFamily)` amounts to proving +// `Implemented(BoxFamily: PointerFamily)`, which is ok thanks to our impl. // // `WellFormed(Box)` is always true (there are no where clauses on the // `Box` type definition). From 90716dc9da1a4d72507d3dea24caab286657df32 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Mon, 29 Oct 2018 21:59:17 -0500 Subject: [PATCH 396/648] Add section on chalk structure --- src/traits/chalk-overview.md | 65 ++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/src/traits/chalk-overview.md b/src/traits/chalk-overview.md index 3473a0764..029b1460e 100644 --- a/src/traits/chalk-overview.md +++ b/src/traits/chalk-overview.md @@ -16,6 +16,71 @@ existing, somewhat ad-hoc implementation into something far more principled and expressive, which should behave better in corner cases, and be much easier to extend. +## Chalk Structure +Chalk has two main "products". The first of these is the +[`chalk_engine`][doc-chalk-engine] crate, which defines the core [SLG +solver][slg]. This is the part rustc uses. + +The rest of chalk can be considered an elaborate testing harness. Chalk is +capable of parsing Rust-like "programs", lowering them to logic, and +performing queries on them. + +Here's a sample session in the chalk repl, chalki. After feeding it our +program, we perform some queries on it. + +```rust,ignore +?- program +Enter a program; press Ctrl-D when finished +| struct Foo { } +| struct Bar { } +| struct Vec { } +| trait Clone { } +| impl Clone for Vec where T: Clone { } +| impl Clone for Foo { } + +?- Vec: Clone +Unique; substitution [], lifetime constraints [] + +?- Vec: Clone +No possible solution. + +?- exists { Vec: Clone } +Ambiguous; no inference guidance +``` + +You can see more examples of programs and queries in the [unit tests][chalk-tests]. + +[chalk-tests]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L115 + +### Crates +- [**chalk_engine**][doc-chalk-engine]: Defines the core [SLG solver][slg]. +- [**chalk_ir**][doc-chalk-ir]: Defines chalk's internal representation of + types, lifetimes, and goals. +- [**chalk_solve**][doc-chalk-solve]: Combines `chalk_ir` and `chalk_engine`, + effectively. + - [`chalk_engine::context`][engine-context] provides the necessary hooks. +- [**chalk_parse**][doc-chalk-parse]: Defines the raw AST and a parser. +- [**chalk**][doc-chalk]: Brings everything together. Defines the following + modules: + - [`rust_ir`][doc-chalk-rust-ir], containing the "HIR-like" form of the AST + - `rust_ir::lowering`, which converts AST to `rust_ir` + - `rules`, which implements logic rules + converting `rust_ir` to `chalk_ir` + - `coherence`, which implements coherence rules + - Also includes [chalki][doc-chalki], chalk's REPL. + +[Browse source on GitHub](https://github.com/rust-lang-nursery/chalk) + +[engine-context]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/context/index.html + +[doc-chalk-engine]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/index.html +[doc-chalk-ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/index.html +[doc-chalk-solve]: https://rust-lang-nursery.github.io/chalk/doc/chalk_solve/index.html +[doc-chalk-parse]: https://rust-lang-nursery.github.io/chalk/doc/chalk_parse/index.html +[doc-chalk]: https://rust-lang-nursery.github.io/chalk/doc/chalk/index.html +[doc-chalk-rust-ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk/rules/index.html +[doc-chalki]: https://rust-lang-nursery.github.io/chalk/doc/chalki/index.html + ## Resources * [Chalk Source Code](https://github.com/rust-lang-nursery/chalk) From b62fe76a354ec60835d6e83d6a6ff70b17515fc3 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Mon, 29 Oct 2018 22:19:58 -0500 Subject: [PATCH 397/648] Move Resources to bottom --- src/traits/chalk-overview.md | 34 +++++++++++++++------------------- 1 file changed, 15 insertions(+), 19 deletions(-) diff --git a/src/traits/chalk-overview.md b/src/traits/chalk-overview.md index 029b1460e..20a71ef87 100644 --- a/src/traits/chalk-overview.md +++ b/src/traits/chalk-overview.md @@ -50,7 +50,7 @@ Ambiguous; no inference guidance You can see more examples of programs and queries in the [unit tests][chalk-tests]. -[chalk-tests]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L115 +Next we'll go through each stage required to produce the output above. ### Crates - [**chalk_engine**][doc-chalk-engine]: Defines the core [SLG solver][slg]. @@ -80,22 +80,7 @@ You can see more examples of programs and queries in the [unit tests][chalk-test [doc-chalk]: https://rust-lang-nursery.github.io/chalk/doc/chalk/index.html [doc-chalk-rust-ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk/rules/index.html [doc-chalki]: https://rust-lang-nursery.github.io/chalk/doc/chalki/index.html - -## Resources - -* [Chalk Source Code](https://github.com/rust-lang-nursery/chalk) -* [Chalk Glossary](https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md) -* The traits section of the rustc guide (you are here) - -### Blog Posts - -* [Lowering Rust traits to logic](http://smallcultfollowing.com/babysteps/blog/2017/01/26/lowering-rust-traits-to-logic/) -* [Unification in Chalk, part 1](http://smallcultfollowing.com/babysteps/blog/2017/03/25/unification-in-chalk-part-1/) -* [Unification in Chalk, part 2](http://smallcultfollowing.com/babysteps/blog/2017/04/23/unification-in-chalk-part-2/) -* [Negative reasoning in Chalk](http://aturon.github.io/blog/2017/04/24/negative-chalk/) -* [Query structure in chalk](http://smallcultfollowing.com/babysteps/blog/2017/05/25/query-structure-in-chalk/) -* [Cyclic queries in chalk](http://smallcultfollowing.com/babysteps/blog/2017/09/12/tabling-handling-cyclic-queries-in-chalk/) -* [An on-demand SLG solver for chalk](http://smallcultfollowing.com/babysteps/blog/2018/01/31/an-on-demand-slg-solver-for-chalk/) +[chalk-tests]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L115 ## Parsing @@ -190,9 +175,20 @@ described above. [This](https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/solve/test.rs#L83-L110) is the function that is ultimately called. -## Solver +## More Resources + +* [Chalk Source Code](https://github.com/rust-lang-nursery/chalk) +* [Chalk Glossary](https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md) + +### Blog Posts -See [The SLG Solver][slg]. +* [Lowering Rust traits to logic](http://smallcultfollowing.com/babysteps/blog/2017/01/26/lowering-rust-traits-to-logic/) +* [Unification in Chalk, part 1](http://smallcultfollowing.com/babysteps/blog/2017/03/25/unification-in-chalk-part-1/) +* [Unification in Chalk, part 2](http://smallcultfollowing.com/babysteps/blog/2017/04/23/unification-in-chalk-part-2/) +* [Negative reasoning in Chalk](http://aturon.github.io/blog/2017/04/24/negative-chalk/) +* [Query structure in chalk](http://smallcultfollowing.com/babysteps/blog/2017/05/25/query-structure-in-chalk/) +* [Cyclic queries in chalk](http://smallcultfollowing.com/babysteps/blog/2017/09/12/tabling-handling-cyclic-queries-in-chalk/) +* [An on-demand SLG solver for chalk](http://smallcultfollowing.com/babysteps/blog/2018/01/31/an-on-demand-slg-solver-for-chalk/) [rustc-issues]: https://github.com/rust-lang-nursery/rustc-guide/issues [chalk]: https://github.com/rust-lang-nursery/chalk From d2238c30b7837ec1bd6411b5a01ef3c8e2807604 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Mon, 29 Oct 2018 22:22:14 -0500 Subject: [PATCH 398/648] Move Crates section down Nest existing content under Chalk Structure. I think it reads better this way. --- src/traits/chalk-overview.md | 73 ++++++++++++++++++++---------------- 1 file changed, 40 insertions(+), 33 deletions(-) diff --git a/src/traits/chalk-overview.md b/src/traits/chalk-overview.md index 20a71ef87..4abe0ac02 100644 --- a/src/traits/chalk-overview.md +++ b/src/traits/chalk-overview.md @@ -17,6 +17,7 @@ expressive, which should behave better in corner cases, and be much easier to extend. ## Chalk Structure + Chalk has two main "products". The first of these is the [`chalk_engine`][doc-chalk-engine] crate, which defines the core [SLG solver][slg]. This is the part rustc uses. @@ -52,37 +53,9 @@ You can see more examples of programs and queries in the [unit tests][chalk-test Next we'll go through each stage required to produce the output above. -### Crates -- [**chalk_engine**][doc-chalk-engine]: Defines the core [SLG solver][slg]. -- [**chalk_ir**][doc-chalk-ir]: Defines chalk's internal representation of - types, lifetimes, and goals. -- [**chalk_solve**][doc-chalk-solve]: Combines `chalk_ir` and `chalk_engine`, - effectively. - - [`chalk_engine::context`][engine-context] provides the necessary hooks. -- [**chalk_parse**][doc-chalk-parse]: Defines the raw AST and a parser. -- [**chalk**][doc-chalk]: Brings everything together. Defines the following - modules: - - [`rust_ir`][doc-chalk-rust-ir], containing the "HIR-like" form of the AST - - `rust_ir::lowering`, which converts AST to `rust_ir` - - `rules`, which implements logic rules - converting `rust_ir` to `chalk_ir` - - `coherence`, which implements coherence rules - - Also includes [chalki][doc-chalki], chalk's REPL. - -[Browse source on GitHub](https://github.com/rust-lang-nursery/chalk) - -[engine-context]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/context/index.html - -[doc-chalk-engine]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/index.html -[doc-chalk-ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/index.html -[doc-chalk-solve]: https://rust-lang-nursery.github.io/chalk/doc/chalk_solve/index.html -[doc-chalk-parse]: https://rust-lang-nursery.github.io/chalk/doc/chalk_parse/index.html -[doc-chalk]: https://rust-lang-nursery.github.io/chalk/doc/chalk/index.html -[doc-chalk-rust-ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk/rules/index.html -[doc-chalki]: https://rust-lang-nursery.github.io/chalk/doc/chalki/index.html [chalk-tests]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L115 -## Parsing +### Parsing Chalk is designed to be incorporated with the Rust compiler, so the syntax and concepts it deals with heavily borrow from Rust. It is convenient for the sake @@ -98,7 +71,7 @@ impls, and struct definitions. Parsing is often the first "phase" of transformation that a program goes through in order to become a format that chalk can understand. -## Lowering +### Lowering After parsing, there is a "lowering" phase. This aims to convert traits/impls into "program clauses". A [`ProgramClause` (source code)][programclause] is @@ -136,7 +109,7 @@ forall { (Vec: Clone) :- (T: Clone) } This rule dictates that `Vec: Clone` is only satisfied if `T: Clone` is also satisfied (i.e. "provable"). -### Well-formedness checks +#### Well-formedness checks As part of lowering from the AST to the internal IR, we also do some "well formedness" checks. See the [source code][well-formedness-checks] for where @@ -144,7 +117,7 @@ those are done. The call to `record_specialization_priorities` checks "coherence" which means that it ensures that two impls of the same trait for the same type cannot exist. -## Intermediate Representation (IR) +### Intermediate Representation (IR) The second intermediate representation in chalk is called, well, the "ir". :) The [IR source code][ir-code] contains the complete definition. The @@ -159,7 +132,7 @@ In addition to `ir::Program` which has "rust-like things", there is also `program_clauses` which contains the `ProgramClause`s that we generated previously. -## Rules +### Rules The `rules` module works by iterating over every trait, impl, etc. and emitting the rules that come from each one. See [Lowering Rules][lowering-rules] for the @@ -167,6 +140,40 @@ most up-to-date reference on that. The `ir::ProgramEnvironment` is created [in this module][rules-environment]. +### Solver + +See [The SLG Solver][slg]. + +## Crates + +Chalk's functionality is broken up into the following crates: +- [**chalk_engine**][doc-chalk-engine]: Defines the core [SLG solver][slg]. +- [**chalk_ir**][doc-chalk-ir]: Defines chalk's internal representation of + types, lifetimes, and goals. +- [**chalk_solve**][doc-chalk-solve]: Combines `chalk_ir` and `chalk_engine`, + effectively. + - [`chalk_engine::context`][engine-context] provides the necessary hooks. +- [**chalk_parse**][doc-chalk-parse]: Defines the raw AST and a parser. +- [**chalk**][doc-chalk]: Brings everything together. Defines the following + modules: + - [`rust_ir`][doc-chalk-rust-ir], containing the "HIR-like" form of the AST + - `rust_ir::lowering`, which converts AST to `rust_ir` + - `rules`, which implements logic rules converting `rust_ir` to `chalk_ir` + - `coherence`, which implements coherence rules + - Also includes [chalki][doc-chalki], chalk's REPL. + +[Browse source on GitHub](https://github.com/rust-lang-nursery/chalk) + +[engine-context]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/context/index.html + +[doc-chalk-engine]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/index.html +[doc-chalk-ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/index.html +[doc-chalk-solve]: https://rust-lang-nursery.github.io/chalk/doc/chalk_solve/index.html +[doc-chalk-parse]: https://rust-lang-nursery.github.io/chalk/doc/chalk_parse/index.html +[doc-chalk]: https://rust-lang-nursery.github.io/chalk/doc/chalk/index.html +[doc-chalk-rust-ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk/rules/index.html +[doc-chalki]: https://rust-lang-nursery.github.io/chalk/doc/chalki/index.html + ## Testing TODO: Basically, [there is a macro](https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/solve/test.rs#L112-L148) From fe9dc61b8935742d40727af66e2ae110701f84fd Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Tue, 30 Oct 2018 21:06:07 +0100 Subject: [PATCH 399/648] Fix a few more things Co-Authored-By: scalexm --- src/traits/wf.md | 59 ++++++++++++++++++++++++++++++++++-------------- 1 file changed, 42 insertions(+), 17 deletions(-) diff --git a/src/traits/wf.md b/src/traits/wf.md index ec391924a..8c4938d33 100644 --- a/src/traits/wf.md +++ b/src/traits/wf.md @@ -233,49 +233,66 @@ impl Foo for f32 { } ``` -So where clauses on associated types work *exactly* like where clauses on -trait methods: in an impl, we must substitute the parameters from the traits -with values provided by the impl, we may omit them if we don't need them, but -we cannot add new where clauses. +> So in Rust, where clauses on associated types work *exactly* like where +> clauses on trait methods: in an impl, we must substitute the parameters from +> the traits with values provided by the impl, we may omit them if we don't +> need them, but we cannot add new where clauses. Now let's see the generated goal for this general impl: ```text forall { - if (FromEnv(WC_impl), FromEnv(InputTypes(SomeType))) { - WellFormed(SomeType: Trait) && - WellFormed(InputTypes(WC_impl)) && + // Well-formedness of types appearing in the impl + if (FromEnv(WC_impl), FromEnv(InputTypes(SomeType: Trait))) { + WellFormed(InputTypes(WC_impl)) && forall { if (FromEnv(WC_assoc)) { - WellFormed(SomeValue: Bounds_assoc) && WellFormed(InputTypes(SomeValue)) } } } + + // Implied bounds checking + if (FromEnv(WC_impl), FromEnv(InputTypes(SomeType: Trait))) { + WellFormed(SomeType: Trait) && + + forall { + if (FromEnv(WC_assoc)) { + WellFormed(SomeValue: Bounds_assoc) + } + } + } } ``` Here is the most complex goal. As always, first, assuming that the various where clauses hold, we prove that every type appearing in the impl -is well-formed, ***except*** types appearing in the receiver type -`SomeType`. Instead, we *assume* that those types are well-formed -(hence the `if (FromEnv(InputTypes(SomeType)))` condition). This is +is well-formed, ***except*** types appearing in the impl header +`SomeType: Trait`. Instead, we *assume* that those types are +well-formed +(hence the `if (FromEnv(InputTypes(SomeType: Trait)))` +conditions). This is part of the implied bounds proposal, so that we can rely on the bounds -written on the definition of the `SomeType` type (and that we don't +written on the definition of e.g. the `SomeType` type (and that we don't need to repeat those bounds). +> Note that we don't need to check well-formedness of types appearing in +> `WC_assoc` because we already did that in the trait decl (they are just +> repeated with some substitutions of values which we already assume to be +> well-formed) -Next, assuming that the where clauses on the impl `WC_impl` hold and that the -input types of `SomeType` are well-formed, we prove that +Next, still assuming that the where clauses on the impl `WC_impl` hold and that +the input types of `SomeType` are well-formed, we prove that `WellFormed(SomeType: Trait)` hold. That is, we want to prove that `SomeType` verify all the where clauses that might transitively come from the `Trait` definition (see [this subsection](./implied-bounds.md#co-inductiveness-of-wellformed)). -Lastly, assuming that the where clauses on the associated type `WC_assoc` hold, +Lastly, assuming in addition that the where clauses on the associated type +`WC_assoc` hold, we prove that `WellFormed(SomeValue: Bounds_assoc)` hold. Again, we are not only proving `Implemented(SomeValue: Bounds_assoc)`, but also -all the facts that might transitively come from `Bounds_assoc`. This is because -we allow the use of implied bounds on associated types: if we have +all the facts that might transitively come from `Bounds_assoc`. We must do this +because we allow the use of implied bounds on associated types: if we have `FromEnv(SomeType: Trait)` in our environment, the lowering rules chapter indicates that we are able to deduce `FromEnv(::Assoc: Bounds_assoc)` without knowing what the @@ -283,6 +300,12 @@ precise value of `::Assoc` is. Some examples for the generated goal: ```rust,ignore +// Trait Program Clauses + +// These are program clauses that come from the trait definitions below +// and that the trait solver can use for its reasonings. I'm just restating +// them here so that we have them in mind. + trait Copy { } // This is a program clause that comes from the trait definition above // and that the trait solver can use for its reasonings. I'm just restating @@ -304,6 +327,8 @@ trait Complete where Self: Partial { } // WellFormed(Self: Partial). // ``` +// Impl WF Goals + impl Partial for T where T: Complete { } // The generated goal is: // ``` From b5024c3a7328eecdccb37edbfe376c6ac99f2004 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Mon, 29 Oct 2018 22:31:59 -0500 Subject: [PATCH 400/648] Chalk Overview: Update old content --- src/traits/chalk-overview.md | 99 +++++++++++++++++++++--------------- 1 file changed, 59 insertions(+), 40 deletions(-) diff --git a/src/traits/chalk-overview.md b/src/traits/chalk-overview.md index 4abe0ac02..5e0f147ca 100644 --- a/src/traits/chalk-overview.md +++ b/src/traits/chalk-overview.md @@ -55,7 +55,7 @@ Next we'll go through each stage required to produce the output above. [chalk-tests]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L115 -### Parsing +### Parsing ([chalk_parse]) Chalk is designed to be incorporated with the Rust compiler, so the syntax and concepts it deals with heavily borrow from Rust. It is convenient for the sake @@ -71,11 +71,23 @@ impls, and struct definitions. Parsing is often the first "phase" of transformation that a program goes through in order to become a format that chalk can understand. -### Lowering +### Rust Intermediate Representation ([rust_ir]) -After parsing, there is a "lowering" phase. This aims to convert traits/impls -into "program clauses". A [`ProgramClause` (source code)][programclause] is -essentially one of the following: +After getting the AST we convert it to a more convenient intermediate +representation called [`rust_ir`][rust_ir]. This is sort of analogous to the +[HIR] in Rust. The process of converting to IR is called *lowering*. + +The [`rust_ir::Program`][rust_ir-program] struct contains some "rust things" +but indexed and accessible in a different way. For example, if you have a +type like `Foo`, we would represent `Foo` as a string in the AST but in +`rust_ir::Program`, we use numeric indices (`ItemId`). + +The [IR source code][ir-code] contains the complete definition. + +### Chalk Intermediate Representation ([chalk_ir]) + +Once we have Rust IR it is time to convert it to "program clauses". A +[`ProgramClause`] is essentially one of the following: * A [clause] of the form `consequence :- conditions` where `:-` is read as "if" and `conditions = cond1 && cond2 && ...` @@ -93,8 +105,8 @@ essentially one of the following: *See also: [Goals and Clauses][goals-and-clauses]* -Lowering is the phase where we encode the rules of the trait system into logic. -For example, if we have the following Rust: +This is where we encode the rules of the trait system into logic. For +example, if we have the following Rust: ```rust,ignore impl Clone for Vec {} @@ -109,54 +121,61 @@ forall { (Vec: Clone) :- (T: Clone) } This rule dictates that `Vec: Clone` is only satisfied if `T: Clone` is also satisfied (i.e. "provable"). -#### Well-formedness checks +Similar to [`rust_ir::Program`][rust_ir-program] which has "rust-like +things", chalk_ir defines [`ProgramEnvironment`] which which is "pure logic". +The main field in that struct is `program_clauses`, which contains the +[`ProgramClause`]s generated by the rules module. -As part of lowering from the AST to the internal IR, we also do some "well -formedness" checks. See the [source code][well-formedness-checks] for where -those are done. The call to `record_specialization_priorities` checks -"coherence" which means that it ensures that two impls of the same trait for the -same type cannot exist. +#### Rules + +The `rules` module ([source code][rules-src]) defines the logic rules we use +for each item in the Rust IR. It works by iterating over every trait, impl, +etc. and emitting the rules that come from each one. + +*See also: [Lowering Rules][lowering-rules]* + +[`ProgramEnvironment`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/struct.ProgramEnvironment.html +[`ProgramClause`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/enum.ProgramClause.html +[rules-src]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules.rs + +#### Well-formedness checks -### Intermediate Representation (IR) +As part of lowering to logic, we also do some "well formedness" checks. See +the [`rules::wf` source code][rules-wf-src] for where those are done. -The second intermediate representation in chalk is called, well, the "ir". :) -The [IR source code][ir-code] contains the complete definition. The -`ir::Program` struct contains some "rust things" but indexed and accessible in -a different way. This is sort of analogous to the [HIR] in Rust. +*See also: [Well-formedness checking][wf-checking]* -For example, if you have a type like `Foo`, we would represent `Foo` as a -string in the AST but in `ir::Program`, we use numeric indices (`ItemId`). +[rules-wf-src]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules/wf.rs -In addition to `ir::Program` which has "rust-like things", there is also -`ir::ProgramEnvironment` which is "pure logic". The main field in that struct is -`program_clauses` which contains the `ProgramClause`s that we generated -previously. +#### Coherence -### Rules +The function `record_specialization_priorities` in the `coherence` module +([source code][coherence-src]) checks "coherence", which means that it +ensures that two impls of the same trait for the same type cannot exist. -The `rules` module works by iterating over every trait, impl, etc. and emitting -the rules that come from each one. See [Lowering Rules][lowering-rules] for the -most up-to-date reference on that. +[coherence-src]: https://github.com/rust-lang-nursery/chalk/blob/master/src/coherence.rs -The `ir::ProgramEnvironment` is created [in this module][rules-environment]. +### Solver ([chalk_solve]) -### Solver +Finally, when we've collected all the program clauses we care about, we want +to perform queries on it. The component that finds the answer to these +queries is called the *solver*. -See [The SLG Solver][slg]. +*See also: [The SLG Solver][slg]* ## Crates Chalk's functionality is broken up into the following crates: - [**chalk_engine**][doc-chalk-engine]: Defines the core [SLG solver][slg]. -- [**chalk_ir**][doc-chalk-ir]: Defines chalk's internal representation of +- [**chalk_ir**][chalk_ir]: Defines chalk's internal representation of types, lifetimes, and goals. - [**chalk_solve**][doc-chalk-solve]: Combines `chalk_ir` and `chalk_engine`, effectively. - [`chalk_engine::context`][engine-context] provides the necessary hooks. -- [**chalk_parse**][doc-chalk-parse]: Defines the raw AST and a parser. +- [**chalk_parse**][chalk_parse]: Defines the raw AST and a parser. - [**chalk**][doc-chalk]: Brings everything together. Defines the following modules: - - [`rust_ir`][doc-chalk-rust-ir], containing the "HIR-like" form of the AST + - [`rust_ir`][rust_ir], containing the "HIR-like" form of the AST - `rust_ir::lowering`, which converts AST to `rust_ir` - `rules`, which implements logic rules converting `rust_ir` to `chalk_ir` - `coherence`, which implements coherence rules @@ -167,11 +186,12 @@ Chalk's functionality is broken up into the following crates: [engine-context]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/context/index.html [doc-chalk-engine]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/index.html -[doc-chalk-ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/index.html +[chalk_ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/index.html [doc-chalk-solve]: https://rust-lang-nursery.github.io/chalk/doc/chalk_solve/index.html -[doc-chalk-parse]: https://rust-lang-nursery.github.io/chalk/doc/chalk_parse/index.html +[chalk_parse]: https://rust-lang-nursery.github.io/chalk/doc/chalk_parse/index.html [doc-chalk]: https://rust-lang-nursery.github.io/chalk/doc/chalk/index.html -[doc-chalk-rust-ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk/rules/index.html +[rust_ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk/rust_ir/index.html +[rust_ir-program]: https://rust-lang-nursery.github.io/chalk/doc/chalk/rust_ir/struct.Program.html [doc-chalki]: https://rust-lang-nursery.github.io/chalk/doc/chalki/index.html ## Testing @@ -201,15 +221,14 @@ is the function that is ultimately called. [chalk]: https://github.com/rust-lang-nursery/chalk [lowering-to-logic]: ./lowering-to-logic.html [lowering-rules]: ./lowering-rules.html +[wf-checking]: ./wf.html [ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree [chalk-ast]: https://github.com/rust-lang-nursery/chalk/blob/master/chalk-parse/src/ast.rs [universal quantification]: https://en.wikipedia.org/wiki/Universal_quantification [lowering-forall]: ./lowering-to-logic.html#type-checking-generic-functions-beyond-horn-clauses -[programclause]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L721 [clause]: https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md#clause [goals-and-clauses]: ./goals-and-clauses.html -[well-formedness-checks]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir/lowering.rs#L230-L232 -[ir-code]: https://github.com/rust-lang-nursery/chalk/tree/master/chalk-ir +[ir-code]: https://github.com/rust-lang-nursery/chalk/blob/master/src/rust_ir.rs [HIR]: ../hir.html [binders-struct]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L661 [rules-environment]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/rules.rs#L9 From 7f14c8254e501a418836f8c1fe979d746e847df2 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Tue, 30 Oct 2018 17:20:29 -0500 Subject: [PATCH 401/648] Chalk Overview: Organize and sort links --- src/traits/chalk-overview.md | 67 +++++++++++++++++------------------- 1 file changed, 32 insertions(+), 35 deletions(-) diff --git a/src/traits/chalk-overview.md b/src/traits/chalk-overview.md index 5e0f147ca..1a6e72bec 100644 --- a/src/traits/chalk-overview.md +++ b/src/traits/chalk-overview.md @@ -19,7 +19,7 @@ extend. ## Chalk Structure Chalk has two main "products". The first of these is the -[`chalk_engine`][doc-chalk-engine] crate, which defines the core [SLG +[`chalk_engine`][chalk_engine] crate, which defines the core [SLG solver][slg]. This is the part rustc uses. The rest of chalk can be considered an elaborate testing harness. Chalk is @@ -53,8 +53,6 @@ You can see more examples of programs and queries in the [unit tests][chalk-test Next we'll go through each stage required to produce the output above. -[chalk-tests]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L115 - ### Parsing ([chalk_parse]) Chalk is designed to be incorporated with the Rust compiler, so the syntax and @@ -134,10 +132,6 @@ etc. and emitting the rules that come from each one. *See also: [Lowering Rules][lowering-rules]* -[`ProgramEnvironment`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/struct.ProgramEnvironment.html -[`ProgramClause`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/enum.ProgramClause.html -[rules-src]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules.rs - #### Well-formedness checks As part of lowering to logic, we also do some "well formedness" checks. See @@ -145,16 +139,12 @@ the [`rules::wf` source code][rules-wf-src] for where those are done. *See also: [Well-formedness checking][wf-checking]* -[rules-wf-src]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules/wf.rs - #### Coherence The function `record_specialization_priorities` in the `coherence` module ([source code][coherence-src]) checks "coherence", which means that it ensures that two impls of the same trait for the same type cannot exist. -[coherence-src]: https://github.com/rust-lang-nursery/chalk/blob/master/src/coherence.rs - ### Solver ([chalk_solve]) Finally, when we've collected all the program clauses we care about, we want @@ -166,10 +156,10 @@ queries is called the *solver*. ## Crates Chalk's functionality is broken up into the following crates: -- [**chalk_engine**][doc-chalk-engine]: Defines the core [SLG solver][slg]. +- [**chalk_engine**][chalk_engine]: Defines the core [SLG solver][slg]. - [**chalk_ir**][chalk_ir]: Defines chalk's internal representation of types, lifetimes, and goals. -- [**chalk_solve**][doc-chalk-solve]: Combines `chalk_ir` and `chalk_engine`, +- [**chalk_solve**][chalk_solve]: Combines `chalk_ir` and `chalk_engine`, effectively. - [`chalk_engine::context`][engine-context] provides the necessary hooks. - [**chalk_parse**][chalk_parse]: Defines the raw AST and a parser. @@ -179,20 +169,9 @@ Chalk's functionality is broken up into the following crates: - `rust_ir::lowering`, which converts AST to `rust_ir` - `rules`, which implements logic rules converting `rust_ir` to `chalk_ir` - `coherence`, which implements coherence rules - - Also includes [chalki][doc-chalki], chalk's REPL. - -[Browse source on GitHub](https://github.com/rust-lang-nursery/chalk) - -[engine-context]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/context/index.html + - Also includes [chalki][chalki], chalk's REPL. -[doc-chalk-engine]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/index.html -[chalk_ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/index.html -[doc-chalk-solve]: https://rust-lang-nursery.github.io/chalk/doc/chalk_solve/index.html -[chalk_parse]: https://rust-lang-nursery.github.io/chalk/doc/chalk_parse/index.html -[doc-chalk]: https://rust-lang-nursery.github.io/chalk/doc/chalk/index.html -[rust_ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk/rust_ir/index.html -[rust_ir-program]: https://rust-lang-nursery.github.io/chalk/doc/chalk/rust_ir/struct.Program.html -[doc-chalki]: https://rust-lang-nursery.github.io/chalk/doc/chalki/index.html +[Browse source code on GitHub](https://github.com/rust-lang-nursery/chalk) ## Testing @@ -217,19 +196,37 @@ is the function that is ultimately called. * [Cyclic queries in chalk](http://smallcultfollowing.com/babysteps/blog/2017/09/12/tabling-handling-cyclic-queries-in-chalk/) * [An on-demand SLG solver for chalk](http://smallcultfollowing.com/babysteps/blog/2018/01/31/an-on-demand-slg-solver-for-chalk/) -[rustc-issues]: https://github.com/rust-lang-nursery/rustc-guide/issues -[chalk]: https://github.com/rust-lang-nursery/chalk -[lowering-to-logic]: ./lowering-to-logic.html +[goals-and-clauses]: ./goals-and-clauses.html +[HIR]: ../hir.html +[lowering-forall]: ./lowering-to-logic.html#type-checking-generic-functions-beyond-horn-clauses [lowering-rules]: ./lowering-rules.html +[lowering-to-logic]: ./lowering-to-logic.html +[slg]: ./slg.html [wf-checking]: ./wf.html + [ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree -[chalk-ast]: https://github.com/rust-lang-nursery/chalk/blob/master/chalk-parse/src/ast.rs +[chalk]: https://github.com/rust-lang-nursery/chalk +[rustc-issues]: https://github.com/rust-lang-nursery/rustc-guide/issues [universal quantification]: https://en.wikipedia.org/wiki/Universal_quantification -[lowering-forall]: ./lowering-to-logic.html#type-checking-generic-functions-beyond-horn-clauses + +[`ProgramClause`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/enum.ProgramClause.html +[`ProgramEnvironment`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/struct.ProgramEnvironment.html +[chalk_engine]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/index.html +[chalk_ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/index.html +[chalk_parse]: https://rust-lang-nursery.github.io/chalk/doc/chalk_parse/index.html +[chalk_solve]: https://rust-lang-nursery.github.io/chalk/doc/chalk_solve/index.html +[doc-chalk]: https://rust-lang-nursery.github.io/chalk/doc/chalk/index.html +[engine-context]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/context/index.html +[rust_ir-program]: https://rust-lang-nursery.github.io/chalk/doc/chalk/rust_ir/struct.Program.html +[rust_ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk/rust_ir/index.html + +[binders-struct]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L661 +[chalk-ast]: https://github.com/rust-lang-nursery/chalk/blob/master/chalk-parse/src/ast.rs +[chalk-tests]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L115 +[chalki]: https://rust-lang-nursery.github.io/chalk/doc/chalki/index.html [clause]: https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md#clause -[goals-and-clauses]: ./goals-and-clauses.html +[coherence-src]: https://github.com/rust-lang-nursery/chalk/blob/master/src/coherence.rs [ir-code]: https://github.com/rust-lang-nursery/chalk/blob/master/src/rust_ir.rs -[HIR]: ../hir.html -[binders-struct]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L661 [rules-environment]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/rules.rs#L9 -[slg]: ./slg.html +[rules-src]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules.rs +[rules-wf-src]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules/wf.rs From cf2682ae86e6f026395e7bdfc3ed1bf3a34e6d5b Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Tue, 30 Oct 2018 18:03:42 -0500 Subject: [PATCH 402/648] Chalk Overview: Fill in testing section --- src/traits/chalk-overview.md | 39 +++++++++++++++++++++++++++++------- 1 file changed, 32 insertions(+), 7 deletions(-) diff --git a/src/traits/chalk-overview.md b/src/traits/chalk-overview.md index 1a6e72bec..59572479d 100644 --- a/src/traits/chalk-overview.md +++ b/src/traits/chalk-overview.md @@ -49,7 +49,8 @@ No possible solution. Ambiguous; no inference guidance ``` -You can see more examples of programs and queries in the [unit tests][chalk-tests]. +You can see more examples of programs and queries in the [unit +tests][chalk-test-example]. Next we'll go through each stage required to produce the output above. @@ -175,11 +176,29 @@ Chalk's functionality is broken up into the following crates: ## Testing -TODO: Basically, [there is a macro](https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/solve/test.rs#L112-L148) -that will take chalk's Rust-like syntax and run it through the full pipeline -described above. -[This](https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/solve/test.rs#L83-L110) -is the function that is ultimately called. +chalk has a test framework for lowering programs to logic, checking the +lowered logic, and performing queries on it. This is how we test the +implementation of chalk itself, and the viability of the [lowering +rules][lowering-rules]. + +The main kind of tests in chalk are **goal tests**. They contain a program, +which is expected to lower to logic successfully, and a set of queries +(goals) along with the expected output. Here's an +[example][chalk-test-example]. Since chalk's output can be quite long, goal +tests support specifying only a prefix of the output. + +**Lowering tests** check the stages that occur before we can issue queries +to the solver: the [lowering to rust_ir][chalk-test-lowering], and the +[well-formedness checks][chalk-test-wf] that occur after that. + +### Testing internals + +Goal tests use a [`test!` macro][test-macro] that takes chalk's Rust-like +syntax and runs it through the full pipeline described above. The macro +ultimately calls the [`solve_goal` function][solve_goal]. + +Likewise, lowering tests use the [`lowering_success!` and +`lowering_error!` macros][test-lowering-macros]. ## More Resources @@ -222,7 +241,10 @@ is the function that is ultimately called. [binders-struct]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L661 [chalk-ast]: https://github.com/rust-lang-nursery/chalk/blob/master/chalk-parse/src/ast.rs -[chalk-tests]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L115 +[chalk-test-example]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L115 +[chalk-test-lowering-example]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rust_ir/lowering/test.rs#L8-L31 +[chalk-test-lowering]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rust_ir/lowering/test.rs +[chalk-test-wf]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules/wf/test.rs#L1 [chalki]: https://rust-lang-nursery.github.io/chalk/doc/chalki/index.html [clause]: https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md#clause [coherence-src]: https://github.com/rust-lang-nursery/chalk/blob/master/src/coherence.rs @@ -230,3 +252,6 @@ is the function that is ultimately called. [rules-environment]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/rules.rs#L9 [rules-src]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules.rs [rules-wf-src]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules/wf.rs +[solve_goal]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L85 +[test-lowering-macros]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test_util.rs#L21-L54 +[test-macro]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L33 From 66e831eeb0de5ca3cea50c12255a09278e7920c8 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Wed, 31 Oct 2018 15:33:20 -0500 Subject: [PATCH 403/648] Missing link --- src/borrow_check/moves_and_initialization/move_paths.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/borrow_check/moves_and_initialization/move_paths.md b/src/borrow_check/moves_and_initialization/move_paths.md index c9e22a81c..8fd7b3f19 100644 --- a/src/borrow_check/moves_and_initialization/move_paths.md +++ b/src/borrow_check/moves_and_initialization/move_paths.md @@ -47,6 +47,7 @@ move_data.move_paths[mpi].place [move_paths]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MoveData.html#structfield.move_paths [`MovePath::place`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MovePath.html#structfield.place +[`MovePathIndex`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/indexes/struct.MovePathIndex.html ## Building move paths From 9805434b94e175beca3d384540714469d3431127 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Mon, 22 Oct 2018 12:37:13 -0500 Subject: [PATCH 404/648] Clarify why 'static is free --- src/traits/canonicalization.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/traits/canonicalization.md b/src/traits/canonicalization.md index 62c4d9f74..ca81d6fd1 100644 --- a/src/traits/canonicalization.md +++ b/src/traits/canonicalization.md @@ -42,7 +42,11 @@ This query contains two unbound variables, but it also contains the lifetime `'static`. The trait system generally ignores all lifetimes and treats them equally, so when canonicalizing, we will *also* replace any [free lifetime](../appendix/background.html#free-vs-bound) with a -canonical variable. Therefore, we get the following result: +canonical variable (Note that `'static` is actually a _free_ lifetime +variable here. We are not considering it in the typing context of the whole +program but only in the context of this trait reference. Mathematically, we +are not quantifying over the whole program, but only this obligation). +Therefore, we get the following result: ```text ?0: Foo<'?1, ?2> From cbb021e93f31d2b4c7f153232df03a7dd2859c14 Mon Sep 17 00:00:00 2001 From: scalexm Date: Fri, 2 Nov 2018 21:50:36 +0100 Subject: [PATCH 405/648] Fix nits --- src/traits/wf.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/src/traits/wf.md b/src/traits/wf.md index 8c4938d33..f0cb03caa 100644 --- a/src/traits/wf.md +++ b/src/traits/wf.md @@ -27,18 +27,18 @@ In addition to the notations introduced in the chapter about lowering rules, we'll introduce another notation: when checking WF of a declaration, we'll often have to prove that all types that appear are well-formed, except type parameters that we always assume to be WF. Hence, -we'll use the following notation: for a type `SomeType<...>`, we denote -`InputTypes(SomeType<...>)` the set of all non-parameter types appearing in -`SomeType<...>`, including `SomeType<...>` itself. +we'll use the following notation: for a type `SomeType<...>`, we define +`InputTypes(SomeType<...>)` to be the set of all non-parameter types appearing +in `SomeType<...>`, including `SomeType<...>` itself. Examples: * `InputTypes((u32, f32)) = [u32, f32, (u32, f32)]` -* `InputTypes(Box) = [Box]` +* `InputTypes(Box) = [Box]` (assuming that `T` is a type parameter) * `InputTypes(Box>) = [Box, Box>]` -We may naturally extend the `InputTypes` notation to where clauses, for example -`InputTypes(A0: Trait)` is the union of `InputTypes(A0)`, -`InputTypes(A1)`, ..., `InputTypes(An)`. +We also extend the `InputTypes` notation to where clauses in the natural way. +So, for example `InputTypes(A0: Trait)` is the union of +`InputTypes(A0)`, `InputTypes(A1)`, ..., `InputTypes(An)`. # Type definitions @@ -51,7 +51,7 @@ struct Type where WC_type { } ``` -we generate the following goal: +we generate the following goal, which represents its well-formedness condition: ```text forall { if (FromEnv(WC_type)) { @@ -63,7 +63,7 @@ forall { } ``` -which in English gives: assuming that the where clauses defined on the type +which in English states: assuming that the where clauses defined on the type hold, prove that every type appearing in the type definition is well-formed. Some examples: @@ -284,7 +284,7 @@ Next, still assuming that the where clauses on the impl `WC_impl` hold and that the input types of `SomeType` are well-formed, we prove that `WellFormed(SomeType: Trait)` hold. That is, we want to prove that `SomeType` verify all the where clauses that might transitively -come from the `Trait` definition (see +be required by the `Trait` definition (see [this subsection](./implied-bounds.md#co-inductiveness-of-wellformed)). Lastly, assuming in addition that the where clauses on the associated type From 76a2c5eb86f0db7217a6b94d04ac3f082c8ebd4e Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Fri, 2 Nov 2018 15:42:21 -0500 Subject: [PATCH 406/648] Copy chalk_engine README --- src/traits/slg.md | 279 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 277 insertions(+), 2 deletions(-) diff --git a/src/traits/slg.md b/src/traits/slg.md index 1dc56e14c..8e4e451c4 100644 --- a/src/traits/slg.md +++ b/src/traits/slg.md @@ -1,3 +1,278 @@ -# The SLG solver +# The On-Demand SLG solver -TODO: +## Description of how it works + +The basis of the solver is the `Forest` type. A *forest* stores a +collection of *tables* as well as a *stack*. Each *table* represents +the stored results of a particular query that is being performed, as +well as the various *strands*, which are basically suspended +computations that may be used to find more answers. Tables are +interdependent: solving one query may require solving others. + +### Walkthrough + +Perhaps the easiest way to explain how the solver works is to walk +through an example. Let's imagine that we have the following program: + +```rust +trait Debug { } + +struct u32 { } +impl Debug for u32 { } + +struct Rc { } +impl Debug for Rc { } + +struct Vec { } +impl Debug for Vec { } +``` + +Now imagine that we want to find answers for the query `exists { +Rc: Debug }`. The first step would be to u-canonicalize this query; this +is the act of giving canonical names to all the unbound inference variables based on the +order of their left-most appearance, as well as canonicalizing the universes of any +universally bound names (e.g., the `T` in `forall { ... }`). In this case, there are no +universally bound names, but the canonical form Q of the query might look something like: + + Rc: Debug + +where `?0` is a variable in the root universe U0. We would then go and +look for a table with this as the key: since the forest is empty, this +lookup will fail, and we will create a new table T0, corresponding to +the u-canonical goal Q. + +**Ignoring negative reasoning and regions.** To start, we'll ignore +the possibility of negative goals like `not { Foo }`. We'll phase them +in later, as they bring several complications. + +**Creating a table.** When we first create a table, we also initialize +it with a set of *initial strands*. A "strand" is kind of like a +"thread" for the solver: it contains a particular way to produce an +answer. The initial set of strands for a goal like `Rc: Debug` +(i.e., a "domain goal") is determined by looking for *clauses* in the +environment. In Rust, these clauses derive from impls, but also from +where-clauses that are in scope. In the case of our example, there +would be three clauses, each coming from the program. Using a +Prolog-like notation, these look like: + +``` +(u32: Debug). +(Rc: Debug) :- (T: Debug). +(Vec: Debug) :- (T: Debug). +``` + +To create our initial strands, then, we will try to apply each of +these clauses to our goal of `Rc: Debug`. The first and third +clauses are inapplicable because `u32` and `Vec` cannot be unified +with `Rc`. The second clause, however, will work. + +**What is a strand?** Let's talk a bit more about what a strand *is*. In the code, a strand +is the combination of an inference table, an X-clause, and (possibly) +a selected subgoal from that X-clause. But what is an X-clause +(`ExClause`, in the code)? An X-clause pulls together a few things: + +- The current state of the goal we are trying to prove; +- A set of subgoals that have yet to be proven; +- There are also a few things we're ignoring for now: + - delayed literals, region constraints + +The general form of an X-clause is written much like a Prolog clause, +but with somewhat different semantics. Since we're ignoring delayed +literals and region constraints, an X-clause just looks like this: + + G :- L + +where G is a goal and L is a set of subgoals that must be proven. +(The L stands for *literal* -- when we address negative reasoning, a +literal will be either a positive or negative subgoal.) The idea is +that if we are able to prove L then the goal G can be considered true. + +In the case of our example, we would wind up creating one strand, with +an X-clause like so: + + (Rc: Debug) :- (?T: Debug) + +Here, the `?T` refers to one of the inference variables created in the +inference table that accompanies the strand. (I'll use named variables +to refer to inference variables, and numbered variables like `?0` to +refer to variables in a canonicalized goal; in the code, however, they +are both represented with an index.) + +For each strand, we also optionally store a *selected subgoal*. This +is the subgoal after the turnstile (`:-`) that we are currently trying +to prove in this strand. Initally, when a strand is first created, +there is no selected subgoal. + +**Activating a strand.** Now that we have created the table T0 and +initialized it with strands, we have to actually try and produce an +answer. We do this by invoking the `ensure_answer` operation on the +table: specifically, we say `ensure_answer(T0, A0)`, meaning "ensure +that there is a 0th answer". + +Remember that tables store not only strands, but also a vector of +cached answers. The first thing that `ensure_answer` does is to check +whether answer 0 is in this vector. If so, we can just return +immediately. In this case, the vector will be empty, and hence that +does not apply (this becomes important for cyclic checks later on). + +When there is no cached answer, `ensure_answer` will try to produce +one. It does this by selecting a strand from the set of active +strands -- the strands are stored in a `VecDeque` and hence processed +in a round-robin fashion. Right now, we have only one strand, storing +the following X-clause with no selected subgoal: + + (Rc: Debug) :- (?T: Debug) + +When we activate the strand, we see that we have no selected subgoal, +and so we first pick one of the subgoals to process. Here, there is only +one (`?T: Debug`), so that becomes the selected subgoal, changing +the state of the strand to: + + (Rc: Debug) :- selected(?T: Debug, A0) + +Here, we write `selected(L, An)` to indicate that (a) the literal `L` +is the selected subgoal and (b) which answer `An` we are looking for. We +start out looking for `A0`. + +**Processing the selected subgoal.** Next, we have to try and find an +answer to this selected goal. To do that, we will u-canonicalize it +and try to find an associated table. In this case, the u-canonical +form of the subgoal is `?0: Debug`: we don't have a table yet for +that, so we can create a new one, T1. As before, we'll initialize T1 +with strands. In this case, there will be three strands, because all +the program clauses are potentially applicable. Those three strands +will be: + +- `(u32: Debug) :-`, derived from the program clause `(u32: Debug).`. + - Note: This strand has no subgoals. +- `(Vec: Debug) :- (?U: Debug)`, derived from the `Vec` impl. +- `(Rc: Debug) :- (?U: Debug)`, derived from the `Rc` impl. + +We can thus summarize the state of the whole forest at this point as +follows: + +``` +Table T0 [Rc: Debug] + Strands: + (Rc: Debug) :- selected(?T: Debug, A0) + +Table T1 [?0: Debug] + Strands: + (u32: Debug) :- + (Vec: Debug) :- (?U: Debug) + (Rc: Debug) :- (?V: Debug) +``` + +**Delegation between tables.** Now that the active strand from T0 has +created the table T1, it can try to extract an answer. It does this +via that same `ensure_answer` operation we saw before. In this case, +the strand would invoke `ensure_answer(T1, A0)`, since we will start +with the first answer. This will cause T1 to activate its first +strand, `u32: Debug :-`. + +This strand is somewhat special: it has no subgoals at all. This means +that the goal is proven. We can therefore add `u32: Debug` to the set +of *answers* for our table, calling it answer A0 (it is the first +answer). The strand is then removed from the list of strands. + +The state of table T1 is therefore: + +``` +Table T1 [?0: Debug] + Answers: + A0 = [?0 = u32] + Strand: + (Vec: Debug) :- (?U: Debug) + (Rc: Debug) :- (?V: Debug) +``` + +Note that I am writing out the answer A0 as a substitution that can be +applied to the table goal; actually, in the code, the goals for each +X-clause are also represented as substitutions, but in this exposition +I've chosen to write them as full goals, following NFTD. + +Since we now have an answer, `ensure_answer(T1, A0)` will return `Ok` +to the table T0, indicating that answer A0 is available. T0 now has +the job of incorporating that result into its active strand. It does +this in two ways. First, it creates a new strand that is looking for +the next possible answer of T1. Next, it incorpoates the answer from +A0 and removes the subgoal. The resulting state of table T0 is: + +``` +Table T0 [Rc: Debug] + Strands: + (Rc: Debug) :- selected(?T: Debug, A1) + (Rc: Debug) :- +``` + +We then immediately activate the strand that incorporated the answer +(the `Rc: Debug` one). In this case, that strand has no further +subgoals, so it becomes an answer to the table T0. This answer can +then be returned up to our caller, and the whole forest goes quiescent +at this point (remember, we only do enough work to generate *one* +answer). The ending state of the forest at this point will be: + +``` +Table T0 [Rc: Debug] + Answer: + A0 = [?0 = u32] + Strands: + (Rc: Debug) :- selected(?T: Debug, A1) + +Table T1 [?0: Debug] + Answers: + A0 = [?0 = u32] + Strand: + (Vec: Debug) :- (?U: Debug) + (Rc: Debug) :- (?V: Debug) +``` + +Here you can see how the forest captures both the answers we have +created thus far *and* the strands that will let us try to produce +more answers later on. + +## Heritage and acroynms + +This solver implements the SLG solving technique, though extended to +accommodate hereditary harrop (HH) predicates, as well as the needs of +lazy normalization. + +Its design is kind of a fusion of [MiniKanren] and the following +papers, which I will refer to as EWFS and NTFD respectively: + +> Efficient Top-Down Computation of Queries Under the Well-formed Semantics +> (Chen, Swift, and Warren; Journal of Logic Programming '95) + +> A New Formulation of Tabled resolution With Delay +> (Swift; EPIA '99) + +[MiniKanren]: http://minikanren.org/ + +In addition, I incorporated extensions from the following papers, +which I will refer to as SA and RR respectively, that describes how to +do introduce approximation when processing subgoals and so forth: + +> Terminating Evaluation of Logic Programs with Finite Three-Valued Models +> Riguzzi and Swift; ACM Transactions on Computational Logic 2013 +> (Introduces "subgoal abstraction", hence the name SA) +> +> Radial Restraint +> Grosof and Swift; 2013 + +Another useful paper that gives a kind of high-level overview of +concepts at play is the following: + +> XSB: Extending Prolog with Tabled Logic Programming +> (Swift and Warren; Theory and Practice of Logic Programming '10) + +There are a places where I intentionally diverged from the semantics +as described in the papers -- e.g. by more aggressively approximating +-- which I marked them with a comment DIVERGENCE. Those places may +want to be evaluated in the future. + +A few other acronyms that I use: + +- WAM: Warren abstract machine, an efficient way to evaluate Prolog programs. + See . +- HH: Hereditary harrop predicates. What Chalk deals in. + Popularized by Lambda Prolog. From 69d42aa4ce7ff9be6edb760dd436d06967fb6838 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Sun, 4 Nov 2018 09:38:56 -0600 Subject: [PATCH 407/648] SLG: Add intro --- src/traits/slg.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/src/traits/slg.md b/src/traits/slg.md index 8e4e451c4..d151377c5 100644 --- a/src/traits/slg.md +++ b/src/traits/slg.md @@ -1,5 +1,43 @@ # The On-Demand SLG solver +Given a set of program clauses (provided by our [lowering rules][lowering]) +and a query, we need to return the result of the query and the value of any +type variables we can determine. This is the job of the solver. + +For example, `exists { Vec: FromIterator }` has one solution, so +its result is `Unique; substitution [?T := u32]`. A solution also comes with +a set of region constraints, which we'll ignore in this introduction. + +[lowering]: ./lowering-rules.html + +## Goals of the Solver + +### On demand + +There are often many, or even infinitely many, solutions to a query. For +example, say we want to prove that `exists { Vec: Debug }` for _some_ +type `?T`. Our solver should be capable of iterating over each answer one at +a time, say `?T = u32`, then `?T = i32`, and so on, rather than iterating +over every type in the type system. If we need more answers, we can request +more until we are done. This is similar to how Prolog works. + +*See also: [The traditional, interactive Prolog query][pq]* + +[pq]: ./canonical-queries.html#the-traditional-interactive-prolog-query + +### Breadth-first + +`Vec: Debug` is true if `?T: Debug`. This leads to a cycle: `[Vec , +Vec>, Vec>>]`, and so on all implement `Debug`. Our +solver ought to be breadth first and consider answers like `[Vec: Debug, +Vec: Debug, ...]` before it recurses, or we may never find the answer +we're looking for. + +### Cache friendly + +To speed up compilation, we need to cache results, including partial results +left over from past solver queries. + ## Description of how it works The basis of the solver is the `Forest` type. A *forest* stores a From 62c8d0465b77b8580fe3cc048cac9a89268720ac Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Sun, 4 Nov 2018 09:39:31 -0600 Subject: [PATCH 408/648] SLG: Replace heritage section with links to it elsewhere Maybe we want to include this in the bibliography or something, though. --- src/traits/slg.md | 57 ++++++++++------------------------------------- 1 file changed, 12 insertions(+), 45 deletions(-) diff --git a/src/traits/slg.md b/src/traits/slg.md index d151377c5..812afd1ec 100644 --- a/src/traits/slg.md +++ b/src/traits/slg.md @@ -269,48 +269,15 @@ Here you can see how the forest captures both the answers we have created thus far *and* the strands that will let us try to produce more answers later on. -## Heritage and acroynms - -This solver implements the SLG solving technique, though extended to -accommodate hereditary harrop (HH) predicates, as well as the needs of -lazy normalization. - -Its design is kind of a fusion of [MiniKanren] and the following -papers, which I will refer to as EWFS and NTFD respectively: - -> Efficient Top-Down Computation of Queries Under the Well-formed Semantics -> (Chen, Swift, and Warren; Journal of Logic Programming '95) - -> A New Formulation of Tabled resolution With Delay -> (Swift; EPIA '99) - -[MiniKanren]: http://minikanren.org/ - -In addition, I incorporated extensions from the following papers, -which I will refer to as SA and RR respectively, that describes how to -do introduce approximation when processing subgoals and so forth: - -> Terminating Evaluation of Logic Programs with Finite Three-Valued Models -> Riguzzi and Swift; ACM Transactions on Computational Logic 2013 -> (Introduces "subgoal abstraction", hence the name SA) -> -> Radial Restraint -> Grosof and Swift; 2013 - -Another useful paper that gives a kind of high-level overview of -concepts at play is the following: - -> XSB: Extending Prolog with Tabled Logic Programming -> (Swift and Warren; Theory and Practice of Logic Programming '10) - -There are a places where I intentionally diverged from the semantics -as described in the papers -- e.g. by more aggressively approximating --- which I marked them with a comment DIVERGENCE. Those places may -want to be evaluated in the future. - -A few other acronyms that I use: - -- WAM: Warren abstract machine, an efficient way to evaluate Prolog programs. - See . -- HH: Hereditary harrop predicates. What Chalk deals in. - Popularized by Lambda Prolog. +## See also + +- [chalk_solve README][readme], which contains links to papers used and + acronyms referenced in the code +- This section is a lightly adapted version of the blog post [An on-demand + SLG solver for chalk][slg-blog] +- [Negative Reasoning in Chalk][negative-reasoning-blog] explains the need + for negative reasoning, but not how the SLG solver does it + +[readme]: https://github.com/rust-lang-nursery/chalk/blob/239e4ae4e69b2785b5f99e0f2b41fc16b0b4e65e/chalk-engine/src/README.md +[slg-blog]: http://smallcultfollowing.com/babysteps/blog/2018/01/31/an-on-demand-slg-solver-for-chalk/ +[negative-reasoning-blog]: http://aturon.github.io/blog/2017/04/24/negative-chalk/ From 351f75b4eb7eada41426dcc7a81cb73745154792 Mon Sep 17 00:00:00 2001 From: Tom Tromey Date: Tue, 6 Nov 2018 14:47:56 -0700 Subject: [PATCH 409/648] Document ignore-llvm-version This documents the ignore-llvm-version directive. --- src/tests/adding.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/tests/adding.md b/src/tests/adding.md index 580741ccf..418287355 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -167,6 +167,11 @@ source. required for this test; the test is ignored if the system llvm is in use and it doesn't meet the minimum version. This is useful when an llvm feature has been backported to rust-llvm +* `ignore-llvm-version` can be used to skip the test when certain LLVM + versions are used. This takes one or two arguments; the first + argument is the first version to ignore. If no second argument is + given, all subsequent versions are ignored; otherwise, the second + argument is the last version to ignore. * `compile-pass` for UI tests, indicates that the test is supposed to compile, as opposed to the default where the test is supposed to error out. From 253affdfc5532258b2180d8f60b985ba08676093 Mon Sep 17 00:00:00 2001 From: kenta7777 Date: Tue, 6 Nov 2018 14:49:01 +0900 Subject: [PATCH 410/648] revised a jemalloc statement in how-to-build-and-run.md --- src/how-to-build-and-run.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 62461d3ef..863536a06 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -42,9 +42,9 @@ debuginfo = true # Gives you line numbers for backtraces. debuginfo-lines = true -# Using the system allocator (instead of jemalloc) means that tools -# like valgrind and memcache work better. -use-jemalloc = false +# Link the compiler against `jemalloc`, where on Linux and OSX it should +# override the default allocator for rustc and LLVM. +jemalloc = false ``` ### What is x.py? @@ -335,4 +335,4 @@ everything up then you only need to run one command! The documentation for the rust components are found at [rustc doc]. -[rustc doc]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ \ No newline at end of file +[rustc doc]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ From dc9ceca1036650e1f7518e87b1197287bcdb336d Mon Sep 17 00:00:00 2001 From: kenta7777 Date: Wed, 7 Nov 2018 08:58:21 +0900 Subject: [PATCH 411/648] removed jemalloc statements. --- src/how-to-build-and-run.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 863536a06..d89457257 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -41,10 +41,6 @@ debuginfo = true # Gives you line numbers for backtraces. debuginfo-lines = true - -# Link the compiler against `jemalloc`, where on Linux and OSX it should -# override the default allocator for rustc and LLVM. -jemalloc = false ``` ### What is x.py? From 6d01e7ceb2df04cad9bd6e85663dbdd1eb17e03b Mon Sep 17 00:00:00 2001 From: Aleksey Kladov Date: Thu, 8 Nov 2018 13:42:44 +0300 Subject: [PATCH 412/648] Rename trans to codegen --- src/how-to-build-and-run.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index d89457257..89ad47199 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -24,7 +24,7 @@ settings (and possibly others, such as `llvm.ccache`): [llvm] # Enables LLVM assertions, which will check that the LLVM bitcode generated # by the compiler is internally consistent. These are particularly helpful -# if you edit `trans`. +# if you edit `codegen`. assertions = true [rust] From 76be9f99d706d7ac4d10104f44483dd78d6d947a Mon Sep 17 00:00:00 2001 From: Aleksey Kladov Date: Thu, 8 Nov 2018 15:29:17 +0300 Subject: [PATCH 413/648] Remove reference to copyright comment We don't need them anymore --- src/tests/adding.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/src/tests/adding.md b/src/tests/adding.md index 418287355..5b8674814 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -72,10 +72,9 @@ directory structure to use is actually an area of active debate.) ## Comment explaining what the test is about When you create a test file, **include a comment summarizing the point -of the test immediately after the copyright notice**. This should -highlight which parts of the test are more important, and what the bug -was that the test is fixing. Citing an issue number is often very -helpful. +of the test at the start of the file**. This should highlight which +parts of the test are more important, and what the bug was that the +test is fixing. Citing an issue number is often very helpful. This comment doesn't have to be super extensive. Just something like "Regression test for #18060: match arms were matching in the wrong From 97e853022f2e6be88443e1bbb93942aa3a16c813 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Thu, 8 Nov 2018 20:29:45 -0600 Subject: [PATCH 414/648] minor improvements --- src/borrow_check.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/borrow_check.md b/src/borrow_check.md index 40858b1b4..b3eeaa387 100644 --- a/src/borrow_check.md +++ b/src/borrow_check.md @@ -51,13 +51,14 @@ the [`mir_borrowck`] query. the purpose of this type check is to determine all of the constraints between different regions. - Next, we do [region inference](borrow_check/region_inference.html), which computes - the values of each region — basically, points in the control-flow graph. + the values of each region — basically, the points in the control-flow graph where + each lifetime must be valid according to the constraints we collected. - At this point, we can compute the "borrows in scope" at each point. - Finally, we do a second walk over the MIR, looking at the actions it does and reporting errors. For example, if we see a statement like `*a + 1`, then we would check that the variable `a` is initialized and that it is not mutably borrowed, as either of those would - require an error to be reported. - - Doing this check requires the results of all the previous analyses. + require an error to be reported. Doing this check requires the results of all + the previous analyses. [`replace_regions_in_mir`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/fn.replace_regions_in_mir.html From 5194978cc7bea075583e75a77ac3d730ffda86f1 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Thu, 8 Nov 2018 20:38:14 -0600 Subject: [PATCH 415/648] add a few links to rustdocs --- src/borrow_check/region_inference.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index 95c2bc804..e35849ef6 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -9,7 +9,7 @@ deprecated once they become the standard kind of lifetime.) The MIR-based region analysis consists of two major functions: -- `replace_regions_in_mir`, invoked first, has two jobs: +- [`replace_regions_in_mir`], invoked first, has two jobs: - First, it finds the set of regions that appear within the signature of the function (e.g., `'a` in `fn foo<'a>(&'a u32) { ... }`). These are called the "universal" or "free" regions – in @@ -21,21 +21,28 @@ The MIR-based region analysis consists of two major functions: not of much interest. The intention is that – eventually – they will be "erased regions" (i.e., no information at all), since we won't be doing lexical region inference at all. -- `compute_regions`, invoked second: this is given as argument the +- [`compute_regions`], invoked second: this is given as argument the results of move analysis. It has the job of computing values for all the inference variables that `replace_regions_in_mir` introduced. - To do that, it first runs the [MIR type checker](#mirtypeck). This is basically a normal type-checker but specialized to MIR, which - is much simpler than full Rust of course. Running the MIR type + is much simpler than full Rust, of course. Running the MIR type checker will however create **outlives constraints** between region variables (e.g., that one variable must outlive another one) to reflect the subtyping relationships that arise. - It also adds **liveness constraints** that arise from where variables are used. - - More details to come, though the [NLL RFC] also includes fairly thorough - (and hopefully readable) coverage. + - After this, we create a [`RegionInferenceContext`] with the constraints we + have computed and the inference variables we introduced and use the + [`solve`] method to infer values for all region inference varaibles. + - The [NLL RFC] also includes fairly thorough (and hopefully readable) + coverage. [fvb]: ../appendix/background.html#free-vs-bound +[`replace_regions_in_mir`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/fn.replace_regions_in_mir.html +[`compute_regions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/fn.compute_regions.html +[`RegionInferenceContext`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html +[`solve`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.solve [NLL RFC]: http://rust-lang.github.io/rfcs/2094-nll.html ## Universal regions From 9f435099c106c7c26a9b9ed2007b4a7a0ea9c708 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Thu, 8 Nov 2018 21:34:17 -0600 Subject: [PATCH 416/648] fill out the borrowck chapter a bit more --- src/borrow_check/region_inference.md | 109 ++++++++++++++++++++++----- 1 file changed, 92 insertions(+), 17 deletions(-) diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index e35849ef6..d08e06a80 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -24,7 +24,7 @@ The MIR-based region analysis consists of two major functions: - [`compute_regions`], invoked second: this is given as argument the results of move analysis. It has the job of computing values for all the inference variables that `replace_regions_in_mir` introduced. - - To do that, it first runs the [MIR type checker](#mirtypeck). This + - To do that, it first runs the [MIR type checker]. This is basically a normal type-checker but specialized to MIR, which is much simpler than full Rust, of course. Running the MIR type checker will however create **outlives constraints** between @@ -44,29 +44,26 @@ The MIR-based region analysis consists of two major functions: [`RegionInferenceContext`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html [`solve`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.solve [NLL RFC]: http://rust-lang.github.io/rfcs/2094-nll.html +[MIR type checker]: ./type_check.md ## Universal regions -*to be written* – explain the `UniversalRegions` type +The [`UnversalRegions`] type represents a collection of _unversal_ regions +corresponding to some MIR `DefId`. It is constructed in +[`replace_regions_in_mir`] when we replace all regions with fresh inference +variables. [`UniversalRegions`] contains indices for all the free regions in +the given MIR along with any relationships that are _known_ to hold between +them (e.g. implied bounds, where clauses, etc.). -## Region variables and constraints +TODO: is there more to write here? -*to be written* – describe the `RegionInferenceContext` and -the role of `liveness_constraints` vs other `constraints`, plus +[`UniversalRegions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/universal_regions/struct.UniversalRegions.html -## Closures - -*to be written* - - +## Region variables -## The MIR type-check - -## Representing the "values" of a region variable - -The value of a region can be thought of as a **set**; we call the -domain of this set a `RegionElement`. In the code, the value for all -regions is maintained in +The value of a region can be thought of as a **set** of points in the MIR where +the region is valid; we call the domain of this set a `RegionElement`. In the +code, the value for all regions is maintained in [the `rustc_mir::borrow_check::nll::region_infer` module][ri]. For each region we maintain a set storing what elements are present in its value (to make this efficient, we give each kind of element an index, @@ -90,11 +87,86 @@ The kinds of region elements are as follows: for details on placeholders, see the section [placeholders and universes](#placeholder). +## Constraints + +Before we can infer the value of regions, we need to collect constraints on the +regions. There are two primary types of constraints. + +1. Outlives constraints. These are constraints that one region outlives another + (e.g. `'a: 'b`). Outlives constraints are generated by the [MIR type + checker]. +2. Liveness constraints. Each region needs to be live at points where it can be + used. These constraints are collected by [`generate_constraints`]. + +[`generate_constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/constraint_generation/fn.generate_constraints.html + +## Inference Overview + +So how do we compute the contents of a region? This process is called _region +inference_. The high-level idea is pretty simple, but there are some details we +need to take care of. + +The [`RegionInferenceContext`] type contains all of the information needed to +do inference, including the universal regions from `replace_regions_in_mir` and +the constraints computed for each region. It is constructed just after we +compute the liveness constraints. + +Here are some of the fields of the struct: + +- `constraints`: contains all the outlives constraints. +- `liveness_constraints`: contains all the liveness constraints. +- `universal_regions`: contains the `UniversalRegions` returned by + `replace_regions_in_mir`. +- `universal_region_relations`: contains relations known to be true about + universal regions. For example, if we have a where clause that `'a: 'b`, that + relation is assumed to be true while borrow checking the implementation (it + is checked at the caller), so `universal_region_relations` would contain `'a: + 'b`. +- `type_tests`: contains some constraints on types that we must check after + inference (e.g. `T: 'a`). +- `closure_bounds_mapping`: used for propagating region constraints from + closures back out to the creater of the closure. + +TODO: should we discuss any of the others fields? What about the SCCs? + +Ok, now that we have constructed a `RegionInferenceContext`, we can do +inference. This is done by calling the [`solve`] method on the context. + +We will start off the value of each region with the liveness constraints (the +places we already know must be in the region). We will then use the outlives +constraints to widen each region until all constraints are met. This is done in +[`propagate_constraints`]. For each region, if `'a: 'b`, we add all elements of +`'b` to `'a`. + +Then, we will check for errors. We first check that type tests are satisfied by +calling [`check_type_tests`]. This checks constraints like `T: 'a`. Second, we +check that universal regions are not "too big". This is done by calling +[`check_universal_regions`]. This checks that for each region `'a` if `'a` +contains the element `end('b)`, then we must already know that `'a: 'b` holds +(e.g. from a where clause). If we don't already know this, that is an error... +well, almost. + +[`propagate_constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.propagate_constraints +[`check_type_tests`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.check_type_tests +[`check_universal_regions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.check_universal_regions + +## Closures + +When we are checking the type tests and universal regions, we may come across a +constraint that we can't prove yet if we are in a closure body! However, the +necessary constraints may actually hold (we just don't know it yet). Thus, if +we are inside a closure, we just collect all the constraints we can't prove yet +and return them. Later, when we are borrow check the MIR node that created the +closure, we can also check that these constraints hold. At that time, if we +can't prove they hold, we report an error. + ## Causal tracking *to be written* – describe how we can extend the values of a variable with causal tracking etc +TODO: is this what I described above or something else? + ## Placeholders and universes @@ -541,3 +613,6 @@ Now constraint propagation is done, but when we check the outlives relationships, we find that `V2` includes this new element `placeholder(1)`, so we report an error. +## Borrow Checker Errors + +TODO: we should discuss how to generate errors from the results of these analyses. From 9d2e781896af6a86b28e5c1a464d4c6872fc7146 Mon Sep 17 00:00:00 2001 From: Gareth Ellis Date: Thu, 8 Nov 2018 19:55:46 -0800 Subject: [PATCH 417/648] Fixed Spelling Mistake In MIR/index.md - "alloated" --- src/mir/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/mir/index.md b/src/mir/index.md index d69606d66..baaf23c36 100644 --- a/src/mir/index.md +++ b/src/mir/index.md @@ -38,7 +38,7 @@ This section introduces the key concepts of MIR, summarized here: the end of a block - (if you're not familiar with the term *basic block*, see the [background chapter][cfg]) -- **Locals:** Memory locations alloated on the stack (conceptually, at +- **Locals:** Memory locations allocated on the stack (conceptually, at least), such as function arguments, local variables, and temporaries. These are identified by an index, written with a leading underscore, like `_1`. There is also a special "local" From bed7044bbd774bbf0d9d3c6193298489c1810192 Mon Sep 17 00:00:00 2001 From: Aleksey Kladov Date: Thu, 8 Nov 2018 16:46:39 +0300 Subject: [PATCH 418/648] document the interaction between macros and nameres "The name resolution is a separate pass" is a wishful thinking :( --- src/name-resolution.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/src/name-resolution.md b/src/name-resolution.md index bba3142fc..3117217e5 100644 --- a/src/name-resolution.md +++ b/src/name-resolution.md @@ -1,12 +1,17 @@ # Name resolution -The name resolution is a separate pass in the compiler. Its input is the syntax -tree, produced by parsing input files. It produces links from all the names in -the source to relevant places where the name was introduced. It also generates +The name resolution is a two-phase process. In the first phase, which runs +during macro expansion, we build a tree of modules and resolve imports. Macro +expansion and name resolution communicate with each other via the `Resolver` +trait, defined in `libsyntax`. + +The input to the second phase is the syntax tree, produced by parsing input +files and expanding macros. This phase produces links from all the names in the +source to relevant places where the name was introduced. It also generates helpful error messages, like typo suggestions, traits to import or lints about unused items. -A successful run of the name resolution (`Resolver::resolve_crate`) creates kind +A successful run of the second phase (`Resolver::resolve_crate`) creates kind of an index the rest of the compilation may use to ask about the present names (through the `hir::lowering::Resolver` interface). From 80b656fa7c2e2c813a63884d6f3b4d5d073456c2 Mon Sep 17 00:00:00 2001 From: Oliver Scherer Date: Fri, 2 Nov 2018 12:43:36 +0100 Subject: [PATCH 419/648] Document MIR building somewhat --- src/mir/construction.md | 110 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/src/mir/construction.md b/src/mir/construction.md index 898fcd529..606301fdb 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -1 +1,111 @@ # MIR construction + +The lowering of [HIR] to [MIR] occurs for the following (probably incomplete) +list of items: + +* Function and Closure bodies +* Initializers of `static` and `const` items +* Initializers of enum discriminants +* Glue and Shims of any kind + * Tuple struct initializer functions + * Drop code (the `Drop::drop` function is not called directly) + * Drop implementations of types without an explicit `Drop` implementation + +The lowering is triggered by calling the `mir_built` query. The entire lowering +code lives in `src/librustc_mir/build`. There is an intermediate representation +between [HIR] and [MIR] called the `HAIR` that is only used during the lowering. +The `HAIR` has datatypes that mirror the [HIR] datatypes, but instead of e.g. `-x` +being a `hair::ExprKind::Neg(hair::Expr)` it is a `hair::ExprKind::Neg(hir::Expr)`. + +This shallowness enables the `HAIR` to represent all datatypes that [HIR] has, but +without having to create an in-memory copy of the entire [HIR]. The `HAIR` also +does a few simplifications, e.g. method calls and function calls have been merged +into a single variant. + +The lowering creates local variables for every argument as specified in the signature. +Next it creates local variables for every binding specified (e.g. `(a, b): (i32, String)`) +produces 3 bindings, one for the argument, and two for the bindings. Next it generates +field accesses that read the fields from the argument and writes the value to the binding +variable. + +With this initialization out of the way, the lowering triggers a recursive call +to a function that generates the MIR for the body (a `Block` expression) and +writes the result into the `RETURN_PLACE`. + +## `unpack!` all the things + +One important thing of note is the `unpack!` macro, which accompanies all recursive +calls. The macro ensures, that you get the result of the recursive call while updating +the basic block that you are now in. As an example: lowering `a + b` will need to do +three somewhat independent things: + +* give you an `Rvalue` referring to the result of the operation +* insert an assertion ensuring that the operation does not overflow +* tell you in which basic block you should write further operations into, because + the basic block has changed due to the inserted assertion (assertions terminate + blocks and jump either to a panic block or a newly created block, the latter being + the one you get back). + +The `unpack!` macro will call the recursive function you pass it, return the `Rvalue` +and update the basic block by mutating the basic block variable you pass to it. + +## Lowering expressions into the desired MIR + +There are essentially four kinds of representations one might want of a value: + +* `Place` refers to a (or a part of) preexisting memory location (local, static, promoted) +* `Rvalue` is something that can be assigned to a `Place` +* `Operand` is an argument to e.g. a `+` operation or a function call +* a temporary variable containing a copy of the value + +Since we start out with lowering the function body to an `Rvalue` so we can create an +assignment to `RETURN_PLACE`, that `Rvalue` lowering will in turn trigger lowering to +`Operand` for its arguments (if any). `Operand` lowering either produces a `const` +operand, or moves/copies out of a `Place`, thus triggering a `Place` lowering. An +expression being lowered to a `Place` can in turn trigger a temporary to be created +if the expression being lowered contains operations. This is where the snake bites its +own tail and we need to trigger an `Rvalue` lowering for the expression to be written +into the local. + +## Operator lowering + +Operators on builtin types are not lowered to function calls (which would end up being +infinite recursion calls, because the trait impls just contain the operation itself +again). Instead there are `Rvalue`s for binary and unary operators and index operations. +These `Rvalue`s later get codegened to llvm primitive operations or llvm intrinsics. + +Operators on all other types get lowered to a function call to their `impl` of the +Operator's corresponding trait. + +Irrelevant of the lowering kind, the arguments to the operator are lowered to `Operand`s. +This means all arguments are either constants, or refer to an already existing value +somewhere in a local or static. + +## Method call lowering + +Method calls are lowered to the same `TerminatorKind` that function calls are. +In [MIR] there is no difference between method calls and function calls anymore. + +## Conditions + +`if` conditions and `match` statements for `enum`s without variants with fields are +lowered to `TerminatorKind::SwitchInt`. Each possible value (so `0` and `1` for `if` +conditions) has a corresponding `BasicBlock` to which the code continues. +The argument being branched on is again an `Operand`. + +### Pattern matching + +`match` statements for `enum`s with variants that have fields are lowered to +`TerminatorKind::SwitchInt`, too, but the `Operand` refers to a `Place` where the +discriminant of the value can be found. This often involves reading the discriminant +to a new temporary variable. + +## Aggregate construction + +Aggregate values of any kind are built via `Rvalue::Aggregate`. All fields are +lowered to `Operator`s. This is essentially equivalent to one assignment +statement per aggregate field plus an assignment to the discriminant in the +case of `enum`s. + +[MIR]: ./index.html +[HIR]: ../hir.html \ No newline at end of file From a7c23651b71f2799d0aba15721e3cae717b80f64 Mon Sep 17 00:00:00 2001 From: Oliver Scherer Date: Mon, 5 Nov 2018 07:37:14 +0100 Subject: [PATCH 420/648] Link to docs and address some review comments --- src/mir/construction.md | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index 606301fdb..d329da3c1 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -11,16 +11,22 @@ list of items: * Drop code (the `Drop::drop` function is not called directly) * Drop implementations of types without an explicit `Drop` implementation -The lowering is triggered by calling the `mir_built` query. The entire lowering -code lives in `src/librustc_mir/build`. There is an intermediate representation -between [HIR] and [MIR] called the `HAIR` that is only used during the lowering. -The `HAIR` has datatypes that mirror the [HIR] datatypes, but instead of e.g. `-x` +The lowering is triggered by calling the [`mir_built`] query. +There is an intermediate representation +between [HIR] and [MIR] called the [HAIR] that is only used during the lowering. +The [HAIR]'s most important feature is that the various adjustments that happen +without explicit syntax (coercion, autoderef, autoref, ...) have become explicit +casts, deref operations or reference expressions. + +The [HAIR] has datatypes that mirror the [HIR] datatypes, but instead of e.g. `-x` being a `hair::ExprKind::Neg(hair::Expr)` it is a `hair::ExprKind::Neg(hir::Expr)`. - This shallowness enables the `HAIR` to represent all datatypes that [HIR] has, but -without having to create an in-memory copy of the entire [HIR]. The `HAIR` also +without having to create an in-memory copy of the entire [HIR]. The [HAIR] also does a few simplifications, e.g. method calls and function calls have been merged -into a single variant. +into a single variant. [MIR] lowering will first convert the topmost expression from +[HIR] to [HAIR] (in +[https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/hair/cx/expr/index.html]) +and then process the [HAIR] expressions recursively. The lowering creates local variables for every argument as specified in the signature. Next it creates local variables for every binding specified (e.g. `(a, b): (i32, String)`) @@ -35,7 +41,7 @@ writes the result into the `RETURN_PLACE`. ## `unpack!` all the things One important thing of note is the `unpack!` macro, which accompanies all recursive -calls. The macro ensures, that you get the result of the recursive call while updating +calls. The macro ensures that you get the result of the recursive call while updating the basic block that you are now in. As an example: lowering `a + b` will need to do three somewhat independent things: @@ -108,4 +114,6 @@ statement per aggregate field plus an assignment to the discriminant in the case of `enum`s. [MIR]: ./index.html -[HIR]: ../hir.html \ No newline at end of file +[HIR]: ../hir.html +[HAIR]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/hair/index.html +[`mir_built`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/transform/fn.mir_built.html From 02ce4af2b066033eef6887d4072d1e7aa15a00fd Mon Sep 17 00:00:00 2001 From: Oliver Scherer Date: Mon, 5 Nov 2018 07:39:18 +0100 Subject: [PATCH 421/648] Split overcomplicated sentence into two --- src/mir/construction.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index d329da3c1..a82a5e3a4 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -64,8 +64,8 @@ There are essentially four kinds of representations one might want of a value: * `Operand` is an argument to e.g. a `+` operation or a function call * a temporary variable containing a copy of the value -Since we start out with lowering the function body to an `Rvalue` so we can create an -assignment to `RETURN_PLACE`, that `Rvalue` lowering will in turn trigger lowering to +We start out with lowering the function body to an `Rvalue` so we can create an +assignment to `RETURN_PLACE`, This `Rvalue` lowering will in turn trigger lowering to `Operand` for its arguments (if any). `Operand` lowering either produces a `const` operand, or moves/copies out of a `Place`, thus triggering a `Place` lowering. An expression being lowered to a `Place` can in turn trigger a temporary to be created From 9660fbada5cd180aa13d424fcc6b82f948f56587 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 5 Nov 2018 07:46:24 +0100 Subject: [PATCH 422/648] Explain the unpack! macro --- src/mir/construction.md | 56 ++++++++++++++++++++++++++++++----------- 1 file changed, 42 insertions(+), 14 deletions(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index a82a5e3a4..3e300d43d 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -40,20 +40,48 @@ writes the result into the `RETURN_PLACE`. ## `unpack!` all the things -One important thing of note is the `unpack!` macro, which accompanies all recursive -calls. The macro ensures that you get the result of the recursive call while updating -the basic block that you are now in. As an example: lowering `a + b` will need to do -three somewhat independent things: - -* give you an `Rvalue` referring to the result of the operation -* insert an assertion ensuring that the operation does not overflow -* tell you in which basic block you should write further operations into, because - the basic block has changed due to the inserted assertion (assertions terminate - blocks and jump either to a panic block or a newly created block, the latter being - the one you get back). - -The `unpack!` macro will call the recursive function you pass it, return the `Rvalue` -and update the basic block by mutating the basic block variable you pass to it. +Functions that generate MIR tend to fall into one of two patterns. +First, if the function generates only statements, then it will take a +basic block as argument onto which those statements should be appended. +It can then return a result as normal: + +```rust +fn generate_some_mir(&mut self, block: BasicBlock) -> ResultType { + ... +} +``` + +But there are other functions that may generate new basic blocks as well. +For example, lowering an expression like `if foo { 22 } else { 44 }` +requires generating a small "diamond-shaped graph". +In this case, the functions take a basic block where their code starts +and return a (potentially) new basic block where the code generation ends. +The `BlockAnd` type is used to represent this: + +```rust +fn generate_more_mir(&mut self, block: BasicBlock) -> BlockAnd { + ... +} +``` + +When you invoke these functions, it is common to have a local variable `block` that is effectively a "cursor". It represents the point at which we are adding new MIR. When you invoke `generate_more_mir`, you want to update this cursor. You can do this manually, but it's tedious: + +```rust +let mut block; +let v = match self.generate_more_mir(..) { + BlockAnd { block: new_block, value: v } => { + block = new_block; + v + } +}; +``` + +For this reason, we offer a macro that lets you write +`let v = unpack!(block = self.generate_more_mir(...))`. +It simply extracts the new block and overwrites the +variable `block` that you named in the `unpack!`. +MIR functions that generate statements always take as argument a +basic block onto which the statements should be appended. ## Lowering expressions into the desired MIR From 1dd32368273fe67b74715a192e2867245fcb1e00 Mon Sep 17 00:00:00 2001 From: Oliver Scherer Date: Mon, 5 Nov 2018 07:52:09 +0100 Subject: [PATCH 423/648] Address most other review comments --- src/mir/construction.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index 3e300d43d..b7bd6bec6 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -14,16 +14,16 @@ list of items: The lowering is triggered by calling the [`mir_built`] query. There is an intermediate representation between [HIR] and [MIR] called the [HAIR] that is only used during the lowering. -The [HAIR]'s most important feature is that the various adjustments that happen -without explicit syntax (coercion, autoderef, autoref, ...) have become explicit -casts, deref operations or reference expressions. +The [HAIR]'s most important feature is that the various adjustments (which happen +without explicit syntax) like coercions, autoderef, autoref and overloaded method +calls have become explicit casts, deref operations, reference expressions or +concrete function calls. The [HAIR] has datatypes that mirror the [HIR] datatypes, but instead of e.g. `-x` being a `hair::ExprKind::Neg(hair::Expr)` it is a `hair::ExprKind::Neg(hir::Expr)`. This shallowness enables the `HAIR` to represent all datatypes that [HIR] has, but -without having to create an in-memory copy of the entire [HIR]. The [HAIR] also -does a few simplifications, e.g. method calls and function calls have been merged -into a single variant. [MIR] lowering will first convert the topmost expression from +without having to create an in-memory copy of the entire [HIR]. +[MIR] lowering will first convert the topmost expression from [HIR] to [HAIR] (in [https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/hair/cx/expr/index.html]) and then process the [HAIR] expressions recursively. @@ -85,7 +85,7 @@ basic block onto which the statements should be appended. ## Lowering expressions into the desired MIR -There are essentially four kinds of representations one might want of a value: +There are essentially four kinds of representations one might want of an expression: * `Place` refers to a (or a part of) preexisting memory location (local, static, promoted) * `Rvalue` is something that can be assigned to a `Place` From 8e708929c3bfe98ca3e1c103192fdba7c08f7818 Mon Sep 17 00:00:00 2001 From: Oliver Scherer Date: Mon, 5 Nov 2018 12:59:19 +0100 Subject: [PATCH 424/648] Fix tidy --- src/mir/construction.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index b7bd6bec6..fec1df677 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -64,7 +64,10 @@ fn generate_more_mir(&mut self, block: BasicBlock) -> BlockAnd { } ``` -When you invoke these functions, it is common to have a local variable `block` that is effectively a "cursor". It represents the point at which we are adding new MIR. When you invoke `generate_more_mir`, you want to update this cursor. You can do this manually, but it's tedious: +When you invoke these functions, it is common to have a local variable `block` +that is effectively a "cursor". It represents the point at which we are adding new MIR. +When you invoke `generate_more_mir`, you want to update this cursor. +You can do this manually, but it's tedious: ```rust let mut block; From 38947d5d2f8028db3d87e787a281da3b2e7c1328 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 8 Nov 2018 18:58:29 +0100 Subject: [PATCH 425/648] Grammar nit Co-Authored-By: oli-obk --- src/mir/construction.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index fec1df677..3c0fbd11a 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -90,7 +90,7 @@ basic block onto which the statements should be appended. There are essentially four kinds of representations one might want of an expression: -* `Place` refers to a (or a part of) preexisting memory location (local, static, promoted) +* `Place` refers to a (or part of a) preexisting memory location (local, static, promoted) * `Rvalue` is something that can be assigned to a `Place` * `Operand` is an argument to e.g. a `+` operation or a function call * a temporary variable containing a copy of the value From 9681f2cb88c1050b4670304c979039fef382eb29 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 8 Nov 2018 19:01:59 +0100 Subject: [PATCH 426/648] caps nit Co-Authored-By: oli-obk --- src/mir/construction.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index 3c0fbd11a..b22b67a36 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -112,7 +112,7 @@ again). Instead there are `Rvalue`s for binary and unary operators and index ope These `Rvalue`s later get codegened to llvm primitive operations or llvm intrinsics. Operators on all other types get lowered to a function call to their `impl` of the -Operator's corresponding trait. +operator's corresponding trait. Irrelevant of the lowering kind, the arguments to the operator are lowered to `Operand`s. This means all arguments are either constants, or refer to an already existing value From 33558bdac6cfb0a4fa4a0504f154b74e778012db Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 8 Nov 2018 19:02:14 +0100 Subject: [PATCH 427/648] english is hard Co-Authored-By: oli-obk --- src/mir/construction.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index b22b67a36..4217a58a2 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -114,7 +114,7 @@ These `Rvalue`s later get codegened to llvm primitive operations or llvm intrins Operators on all other types get lowered to a function call to their `impl` of the operator's corresponding trait. -Irrelevant of the lowering kind, the arguments to the operator are lowered to `Operand`s. +Regardless of the lowering kind, the arguments to the operator are lowered to `Operand`s. This means all arguments are either constants, or refer to an already existing value somewhere in a local or static. From 030670699a10be13910d66d978f50ced08256681 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 8 Nov 2018 19:02:49 +0100 Subject: [PATCH 428/648] Missing connection between two consecutive sentences Co-Authored-By: oli-obk --- src/mir/construction.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index 4217a58a2..5f54e2180 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -128,7 +128,8 @@ In [MIR] there is no difference between method calls and function calls anymore. `if` conditions and `match` statements for `enum`s without variants with fields are lowered to `TerminatorKind::SwitchInt`. Each possible value (so `0` and `1` for `if` conditions) has a corresponding `BasicBlock` to which the code continues. -The argument being branched on is again an `Operand`. +The argument being branched on is (again) an `Operand` representing the value of +the if condition. ### Pattern matching From f4978b56469382f37e8717f58069634267b2371b Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 8 Nov 2018 19:03:29 +0100 Subject: [PATCH 429/648] Clarify aggregates Co-Authored-By: oli-obk --- src/mir/construction.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index 5f54e2180..d2ce16449 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -140,7 +140,7 @@ to a new temporary variable. ## Aggregate construction -Aggregate values of any kind are built via `Rvalue::Aggregate`. All fields are +Aggregate values of any kind (e.g. structs or tuples) are built via `Rvalue::Aggregate`. All fields are lowered to `Operator`s. This is essentially equivalent to one assignment statement per aggregate field plus an assignment to the discriminant in the case of `enum`s. From 21beb7738a0fd2e1870e0925f04c91144b76d94c Mon Sep 17 00:00:00 2001 From: Oliver Scherer Date: Thu, 8 Nov 2018 18:57:59 +0100 Subject: [PATCH 430/648] Remove a sentence from the end of a chapter because it also appears right at the start of the chapter --- src/mir/construction.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index d2ce16449..1e52730a2 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -83,8 +83,6 @@ For this reason, we offer a macro that lets you write `let v = unpack!(block = self.generate_more_mir(...))`. It simply extracts the new block and overwrites the variable `block` that you named in the `unpack!`. -MIR functions that generate statements always take as argument a -basic block onto which the statements should be appended. ## Lowering expressions into the desired MIR From ce79098344fa85a5d02a2adbcac983cb282559dd Mon Sep 17 00:00:00 2001 From: Oliver Scherer Date: Thu, 8 Nov 2018 19:29:37 +0100 Subject: [PATCH 431/648] Line length checks --- src/mir/construction.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index 1e52730a2..314e0daa1 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -126,7 +126,7 @@ In [MIR] there is no difference between method calls and function calls anymore. `if` conditions and `match` statements for `enum`s without variants with fields are lowered to `TerminatorKind::SwitchInt`. Each possible value (so `0` and `1` for `if` conditions) has a corresponding `BasicBlock` to which the code continues. -The argument being branched on is (again) an `Operand` representing the value of +The argument being branched on is (again) an `Operand` representing the value of the if condition. ### Pattern matching @@ -138,7 +138,8 @@ to a new temporary variable. ## Aggregate construction -Aggregate values of any kind (e.g. structs or tuples) are built via `Rvalue::Aggregate`. All fields are +Aggregate values of any kind (e.g. structs or tuples) are built via `Rvalue::Aggregate`. +All fields are lowered to `Operator`s. This is essentially equivalent to one assignment statement per aggregate field plus an assignment to the discriminant in the case of `enum`s. From 6569862a802ade528727fbc473ba721f2b91360c Mon Sep 17 00:00:00 2001 From: Oliver Scherer Date: Fri, 9 Nov 2018 10:29:56 +0100 Subject: [PATCH 432/648] Don't try to build some example code snippets --- src/mir/construction.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index 314e0daa1..03e220cb4 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -45,7 +45,7 @@ First, if the function generates only statements, then it will take a basic block as argument onto which those statements should be appended. It can then return a result as normal: -```rust +```rust,ignore fn generate_some_mir(&mut self, block: BasicBlock) -> ResultType { ... } @@ -58,7 +58,7 @@ In this case, the functions take a basic block where their code starts and return a (potentially) new basic block where the code generation ends. The `BlockAnd` type is used to represent this: -```rust +```rust,ignore fn generate_more_mir(&mut self, block: BasicBlock) -> BlockAnd { ... } @@ -69,7 +69,7 @@ that is effectively a "cursor". It represents the point at which we are adding n When you invoke `generate_more_mir`, you want to update this cursor. You can do this manually, but it's tedious: -```rust +```rust,ignore let mut block; let v = match self.generate_more_mir(..) { BlockAnd { block: new_block, value: v } => { From ade4a4ec4715fcf35ffe1d87c6c246147f2657f2 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Fri, 9 Nov 2018 19:31:47 -0600 Subject: [PATCH 433/648] Don't keep recompiling! --- ci/install.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ci/install.sh b/ci/install.sh index 80f8de4e8..947d751b4 100755 --- a/ci/install.sh +++ b/ci/install.sh @@ -20,5 +20,5 @@ function cargo_install() { fi } -cargo_install mdbook 0.2 -cargo_install mdbook-linkcheck 0.2 +cargo_install mdbook 0.2.2 +cargo_install mdbook-linkcheck 0.2.3 From 07eb29fb1e78942981bcae5ca2730d86299e40f2 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sat, 10 Nov 2018 22:06:15 -0600 Subject: [PATCH 434/648] start on walkthrough --- src/walkthrough.md | 147 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 147 insertions(+) diff --git a/src/walkthrough.md b/src/walkthrough.md index ab158177a..680f2b831 100644 --- a/src/walkthrough.md +++ b/src/walkthrough.md @@ -1 +1,148 @@ # Walkthrough: a typical contribution + +There are _a lot_ of ways to contribute to the rust compiler, including fixing +bugs, improving performance, helping design features, providing feedback on +existing features, etc. This chapter does not claim to scratch the surface. +Instead, it walks through the design and implementation of a new feature. Not +all of the steps and processes described here are needed for every +contribution, and I will try to point those out as they arise. + +In general, if you are interested in making a contribution and aren't sure +where to start, please feel free to ask! + +## Overview + +The feature I will discuss in this chapter is the `?` Kleene operator for +macros. Basically, we want to be able to write something like this: + +```rust +macro_rules! foo { + ($arg:ident $(, $optional_arg:ident)?) => { + println!("{}", $arg); + + $( + println!("{}", $optional_arg); + )? + } +} + +fn main() { + let x = 0; + foo!(x); // ok! prints "0" + foo!(x, x); // ok! prints "0 0" +} +``` + +So basically, the `$(pat)?` matcher in the macro means "this pattern can occur +0 or 1 times", similar to other regex syntaxes. + +There were a number of steps to go from an idea to stable rust feature. Here is +a quick list. We will go through each of these in order below. As I mentioned +before, not all of these are needed for every type of contribution. + +- **Idea discussion/Pre-RFC** A Pre-RFC is an early draft or design discussion + of a feature. This stage is intended to flesh out the design space a bit and + get a grasp on the different merits and problems with an idea. It's a great + way to get early feedback on your idea before presenting it the wider + audience. You can find the original discussion [here][prerfc]. +- **RFC** This is when you formally present your idea to the community for + consideration. You can find the RFC [here][rfc]. +- **Implementation** Implement your idea unstabley in the compiler. You can + find the original implementation [here][impl1]. +- **Possibly iterate/refine** As the community gets experience with your + feature on the nightly compiler and in `libstd`, there may be additional + feedback about design choice that might be adjusted. This particular feature + went [through][impl2] a [number][impl3] of [iterations][impl4]. +- **Stabilization** When your feature has baked enough, a rust team member may + [propose to stabilize it][merge]. If there is consensus, this is done. +- **Relax** Your feature is now a stable rust feature! + +[prerfc]: https://internals.rust-lang.org/t/pre-rfc-at-most-one-repetition-macro-patterns/6557 +[rfc]: https://github.com/rust-lang/rfcs/pull/2298 +[impl1]: https://github.com/rust-lang/rust/pull/47752 +[impl2]: https://github.com/rust-lang/rust/pull/49719 +[impl3]: https://github.com/rust-lang/rust/pull/51336 +[impl4]: https://github.com/rust-lang/rust/pull/51587 +[merge]: https://github.com/rust-lang/rust/issues/48075#issuecomment-433177613 + +## Pre-RFC and RFC + +> NOTE: In general, if you are not proposing a _new_ feature or substantial +> change to rust or the ecosystem, you don't need to follow the RFC process. +> Instead, you can just jump to [implementation](#impl). +> +> You can find the official guidelines for when to open an RFC [here][rfcwhen]. + +[rfcwhen]: https://github.com/rust-lang/rfcs#when-you-need-to-follow-this-process + +An RFC is a document that describes the feature or change you are proposing in +detail. Anyone can write an RFC; the process is the same for everyone, +including rust team members. + +To open an RFC, open a PR on the +[rust-lang/rfcs](https://github.com/rust-lang/rfcs) repo on GitHub. You can +find detailed instructions in the +[README](https://github.com/rust-lang/rfcs#what-the-process-is). + +Before opening an RFC, you should do the research to "flesh out" your idea. +Hastily-proposed RFCs tend not to be accepted. You should generally have a good +description of the motivation, impact, disadvantages, and potential +interactions with other features. + +If that sounds like a lot of work, it's because it is. But no fear! Even if +you're not a compiler hacker, you can get great feedback by doing a _pre-RFC_. +This is an _informal_ discussion of the idea. The best place to do this is +internals.rust-lang.org. Your post doesn't have to follow any particular +structure. It doesn't even need to be a cohesive idea. Generally, you will get +tons of feedback that you can integrate back to produce a good RFC. + +(Another pro-tip: try searching the RFCs repo and internals for prior related +ideas. A lot of times an idea has already been considered and was either +rejected or postponed to be tried again later. This can save you and everybody +else some time) + +In the case of our example, a participant in the pre-RFC thread pointed out a +syntax ambiguity and a potential resolution. Also, the overall feedback seemed +positive. In this case, the discussion converged pretty quickly, but for some +ideas, a lot more discussion can happen (e.g. see [this RFC][nonascii] which +received a whopping 684 comments!). If that happens, don't be discouraged; it +means the community is interested in your idea, but it perhaps needs some +adjustments. + +[nonascii]: https://github.com/rust-lang/rfcs/pull/2457 + +The RFC for our `?` macro feature did receive some discussion on the RFC thread +too. As with most RFCs, there were a few questions that we couldn't answer by +discussion: we needed experience using the feature to decide. Such questions +are listed in the "Unresolved Questions" section of the RFC. Also, over the +course of the RFC discussion, you will probably want to update the RFC document +itself to reflect the course of the discussion (e.g. new alternatives or prior +work may be added or you may decide to change parts of the proposal itself). + +In the end, when the discussion seems to reach a consensus and die down a bit, +a rust team member may propose [to merge the RFC][rfcmerge]. This means that +they want the other members of the appropriate teams to review and comment on +the RFC. More changes may be proposed. At some point, when everyone is +satisfied, the RFC enters the "final comment period" (FCP), which is the last +chance for people to bring up objections. When the FCP is over, the RFC is +"merged" (or accepted). + +[rfcmerge]: https://github.com/rust-lang/rfcs/pull/2298#issuecomment-360582667 + +Some other possible outcomes might be for a team member to propose to + +- _Close_: this feature in its current form is not a good fit for rust. Don't + be discouraged if this happens to your RFC, and don't take it personally. + This is not a reflection on you, but rather a community decision that rust + will go a different direction. +- _Postpone_: there is interest in going this direction but not at the moment. + This happens most often because the appropriate rust team doesn't have the + bandwidth to shepherd the feature through the process to stabilization. Often + this is the case when the feature doesn't fit into the team's roadmap. + Postponed ideas may be revisited later. + + + +## Implementation + +TODO From a5b158f044cac668197588da5fb4694c2f8c38c3 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sat, 10 Nov 2018 22:41:27 -0600 Subject: [PATCH 435/648] add a bit more --- src/walkthrough.md | 67 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 66 insertions(+), 1 deletion(-) diff --git a/src/walkthrough.md b/src/walkthrough.md index 680f2b831..738828bbe 100644 --- a/src/walkthrough.md +++ b/src/walkthrough.md @@ -15,7 +15,7 @@ where to start, please feel free to ask! The feature I will discuss in this chapter is the `?` Kleene operator for macros. Basically, we want to be able to write something like this: -```rust +```rust,ignore macro_rules! foo { ($arg:ident $(, $optional_arg:ident)?) => { println!("{}", $arg); @@ -145,4 +145,69 @@ Some other possible outcomes might be for a team member to propose to ## Implementation +To make a change to the compiler, open a PR against the [rust-lang/rust] repo. + +[rust-lang/rust]: https://github.com/rust-lang/rust + +Depending on the feature/change/bug fix/improvement, implementation may be +relatively-straightforward or it may be a major undertaking. You can always ask +for help or mentorship from more experienced compiler devs. Also, you don't +have to be the one to implement your feature; but keep in mind that if you +don't it might be a while before someone else does. + +For the `?` macro feature, I needed to go understand the relevant parts of +macro expansion in the compiler. Personally, I find that [improving the +comments][comments] in the code is a helpful way of making sure I understand +it, but you don't have to do that if you don't want to. + +[comments]: https://github.com/rust-lang/rust/pull/47732 + +I then [implemented][impl1] the original feature, as described in the RFC. When +a new feature is implemented, it goes behind a _feature gate_, which means that +you have to use `#![feature(my_feature_name)]` to use the feature. The feature +gate is removed when the feature is stabilized. + +**Most bug fixes and improvements** don't require a feature gate. You can just +make your changes/improvements. + +When you open a PR on the [rust-lang/rust], a bot will assign your PR to a +review. If there is a particular rust team member you are working with, you can +request that reviewer by leaving a comment on the thread with `r? +@reviewer-github-id` (e.g. `r? @eddyb`). If you don't know who to request, +don't request anyone; the bot will assign someone automatically. + +The reviewer may request changes before they approve your PR. Feel free to ask +questions or discuss things you don't understand or disagree with. However, +recognize that the PR won't be merged unless someone on the rust team approves +it. + +When your review approves the PR, it will go into a queue for yet another bot +called `@bors`. `@bors` manages the CI build/merge queue. When your PR reaches +the head of the `@bors` queue, `@bors` will test out the merge by running all +tests against your PR on Travis CI. This takes about 2 hours as of this +writing. If all tests pass, the PR is merged and becomes part of the next +nightly compiler! + +There are a couple of things that may happen for some PRs during the review process + +- If the change is substantial enough, the reviewer may request an FCP on + the PR. This gives all members of the appropriate team a chance to review the + changes. +- If the change may cause breakage, the reviewer may request a [crater] run. + This compiles the compiler with your changes and then attempts to compile all + crates on crates.io with your modified compiler. This is a great smoke test + to check if you introduced a change to compiler behavior that affects a large + portion of the ecosystem. +- If the diff of your PR is large or the reviewer is busy, your PR may have + some merge conflicts with other PRs. You should fix these merge conflicts + using the normal git procedures. + +[crater]: ./tests/intro.html#crater + +## Refining your implementation + +TODO + +## Stabilization + TODO From 0f3cb5ef9849f29166e8053052d16d112871c3a5 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sun, 11 Nov 2018 16:32:11 -0600 Subject: [PATCH 436/648] add a bit more --- src/walkthrough.md | 45 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 34 insertions(+), 11 deletions(-) diff --git a/src/walkthrough.md b/src/walkthrough.md index 738828bbe..8cd511d92 100644 --- a/src/walkthrough.md +++ b/src/walkthrough.md @@ -120,17 +120,16 @@ itself to reflect the course of the discussion (e.g. new alternatives or prior work may be added or you may decide to change parts of the proposal itself). In the end, when the discussion seems to reach a consensus and die down a bit, -a rust team member may propose [to merge the RFC][rfcmerge]. This means that -they want the other members of the appropriate teams to review and comment on -the RFC. More changes may be proposed. At some point, when everyone is +a rust team member may propose to move to FCP with one of three possible dispositions. +This means that they want the other members of the appropriate teams to review +and comment on the RFC. More discussion may ensue, which may result in more changes +or unresolved questions being added. At some point, when everyone is satisfied, the RFC enters the "final comment period" (FCP), which is the last -chance for people to bring up objections. When the FCP is over, the RFC is -"merged" (or accepted). - -[rfcmerge]: https://github.com/rust-lang/rfcs/pull/2298#issuecomment-360582667 - -Some other possible outcomes might be for a team member to propose to +chance for people to bring up objections. When the FCP is over, the disposition is +adopted. Here are the three possible dispositions: +- _Merge_: accept the feature. Here is the proposal to merge for our [`?` macro + feature][rfcmerge]. - _Close_: this feature in its current form is not a good fit for rust. Don't be discouraged if this happens to your RFC, and don't take it personally. This is not a reflection on you, but rather a community decision that rust @@ -141,6 +140,15 @@ Some other possible outcomes might be for a team member to propose to this is the case when the feature doesn't fit into the team's roadmap. Postponed ideas may be revisited later. +[rfcmerge]: https://github.com/rust-lang/rfcs/pull/2298#issuecomment-360582667 + +When an RFC is merged, the PR is merged into the RFCs repo. A new _tracking +issue_ is created in the [rust-lang/rust] repo to track progress on the feature +and discuss unresolved questions, implementation progress and blockers, etc. +Here is the tracking issue on for our [`?` macro feature][tracking]. + +[tracking]: https://github.com/rust-lang/rust/issues/48075 + ## Implementation @@ -199,15 +207,30 @@ There are a couple of things that may happen for some PRs during the review proc to check if you introduced a change to compiler behavior that affects a large portion of the ecosystem. - If the diff of your PR is large or the reviewer is busy, your PR may have - some merge conflicts with other PRs. You should fix these merge conflicts - using the normal git procedures. + some merge conflicts with other PRs that happen to get merged first. You + should fix these merge conflicts using the normal git procedures. [crater]: ./tests/intro.html#crater +If you are not doing a new feature or something like that (e.g. if you are +fixing a bug), then that's it! Thanks for your contribution :) + ## Refining your implementation +As people get experience with your new feature on nightly, slight changes may +be proposed and unresolved questions may become resolved. Updates/changes go +through the same process for implementing any other changes, as described +above (i.e. submit a PR, go through review, wait for `@bors`, etc). + +Some changes may be major enough to require an FCP and some review by rust team +members. + +For the `?` macro feature, we went through a few different iterations after the +original implementation: [1][impl2], [2][impl3], [3][impl4]. + TODO + ## Stabilization TODO From a2404ba0716fe22a4d3c362f868ba0992e5a00ff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Fischer?= Date: Thu, 15 Nov 2018 00:46:25 -0300 Subject: [PATCH 437/648] Fix typo (#238) "is is" ~> "it is" --- src/traits/lowering-to-logic.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/traits/lowering-to-logic.md b/src/traits/lowering-to-logic.md index a0490adf3..e1a6c1361 100644 --- a/src/traits/lowering-to-logic.md +++ b/src/traits/lowering-to-logic.md @@ -50,7 +50,7 @@ so by applying the rules recursively: - `Clone(Vec>)` is provable if: - `Clone(Vec)` is provable if: - - `Clone(usize)` is provable. (Which is is, so we're all good.) + - `Clone(usize)` is provable. (Which it is, so we're all good.) But now suppose we tried to prove that `Clone(Vec)`. This would fail (after all, I didn't give an impl of `Clone` for `Bar`): @@ -130,8 +130,8 @@ Ok, so far so good. Let's move on to type-checking a more complex function. In the last section, we used standard Prolog horn-clauses (augmented with Rust's notion of type equality) to type-check some simple Rust functions. But that only works when we are type-checking non-generic functions. If we want to type-check -a generic function, it turns out we need a stronger notion of goal than Prolog -can be provide. To see what I'm talking about, let's revamp our previous +a generic function, it turns out we need a stronger notion of goal than what Prolog +can provide. To see what I'm talking about, let's revamp our previous example to make `foo` generic: ```rust,ignore From ad8ebf263ce6d42c05c7dbb7be05e01cb3887ef3 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 19 Nov 2018 15:34:02 -0600 Subject: [PATCH 438/648] finish walkthrough --- src/walkthrough.md | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/src/walkthrough.md b/src/walkthrough.md index 8cd511d92..4eae1397d 100644 --- a/src/walkthrough.md +++ b/src/walkthrough.md @@ -228,9 +228,34 @@ members. For the `?` macro feature, we went through a few different iterations after the original implementation: [1][impl2], [2][impl3], [3][impl4]. -TODO - +Along the way, we decided that `?` should not take a separator, which was +previously an unresolved question listed in the RFC. We also changed the +disambiguation strategy: we decided to remove the ability to use `?` as a +separator token for other repetition operators (e.g. `+` or `*`). However, +since this was a breaking change, we decided to do it over an edition boundary. +Thus, the new feature can be enabled only in edition 2018. These deviations +from the original RFC required [another +FCP](https://github.com/rust-lang/rust/issues/51934). ## Stabilization -TODO +Finally, after the feature had baked for a while on nightly, a language team member +[moved to stabilize it][stabilizefcp]. + +[stabilizefcp]: https://github.com/rust-lang/rust/issues/48075#issuecomment-433177613 + +A _stabilization report_ needs to be written that includes + +- brief description of the behavior and any deviations from the RFC +- which edition(s) are affected and how +- links to a few tests to show the interesting aspects + +The stabilization report for our feature is [here][stabrep]. + +[stabrep]: https://github.com/rust-lang/rust/issues/48075#issuecomment-433243048 + +After this, a PR is made to remove the feature gate, enabling the feature by +default (on the 2018 edition). A note is added to the [Release notes][relnotes] +about the feature. + +[relnotes]: https://github.com/rust-lang/rust/blob/master/RELEASES.md From 9b6d561a33d08b546e2e981aa72983343964cdb6 Mon Sep 17 00:00:00 2001 From: Tyler Mandry Date: Mon, 19 Nov 2018 18:28:10 -0600 Subject: [PATCH 439/648] SLG: Add links and mark-i-m's suggestions --- src/traits/slg.md | 117 +++++++++++++++++++++++++++------------------- 1 file changed, 68 insertions(+), 49 deletions(-) diff --git a/src/traits/slg.md b/src/traits/slg.md index 812afd1ec..dcddd01f5 100644 --- a/src/traits/slg.md +++ b/src/traits/slg.md @@ -16,10 +16,10 @@ a set of region constraints, which we'll ignore in this introduction. There are often many, or even infinitely many, solutions to a query. For example, say we want to prove that `exists { Vec: Debug }` for _some_ -type `?T`. Our solver should be capable of iterating over each answer one at -a time, say `?T = u32`, then `?T = i32`, and so on, rather than iterating -over every type in the type system. If we need more answers, we can request -more until we are done. This is similar to how Prolog works. +type `?T`. Our solver should be capable of yielding one answer at a time, say +`?T = u32`, then `?T = i32`, and so on, rather than iterating over every type +in the type system. If we need more answers, we can request more until we are +done. This is similar to how Prolog works. *See also: [The traditional, interactive Prolog query][pq]* @@ -27,32 +27,34 @@ more until we are done. This is similar to how Prolog works. ### Breadth-first -`Vec: Debug` is true if `?T: Debug`. This leads to a cycle: `[Vec , +`Vec: Debug` is true if `?T: Debug`. This leads to a cycle: `[Vec, Vec>, Vec>>]`, and so on all implement `Debug`. Our solver ought to be breadth first and consider answers like `[Vec: Debug, Vec: Debug, ...]` before it recurses, or we may never find the answer we're looking for. -### Cache friendly +### Cachable To speed up compilation, we need to cache results, including partial results left over from past solver queries. ## Description of how it works -The basis of the solver is the `Forest` type. A *forest* stores a +The basis of the solver is the [`Forest`] type. A *forest* stores a collection of *tables* as well as a *stack*. Each *table* represents the stored results of a particular query that is being performed, as well as the various *strands*, which are basically suspended computations that may be used to find more answers. Tables are interdependent: solving one query may require solving others. +[`Forest`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/forest/struct.Forest.html + ### Walkthrough Perhaps the easiest way to explain how the solver works is to walk through an example. Let's imagine that we have the following program: -```rust +```rust,ignore trait Debug { } struct u32 { } @@ -65,19 +67,22 @@ struct Vec { } impl Debug for Vec { } ``` -Now imagine that we want to find answers for the query `exists { -Rc: Debug }`. The first step would be to u-canonicalize this query; this -is the act of giving canonical names to all the unbound inference variables based on the -order of their left-most appearance, as well as canonicalizing the universes of any -universally bound names (e.g., the `T` in `forall { ... }`). In this case, there are no -universally bound names, but the canonical form Q of the query might look something like: +Now imagine that we want to find answers for the query `exists { Rc: +Debug }`. The first step would be to u-canonicalize this query; this is the +act of giving canonical names to all the unbound inference variables based on +the order of their left-most appearance, as well as canonicalizing the +universes of any universally bound names (e.g., the `T` in `forall { ... +}`). In this case, there are no universally bound names, but the canonical +form Q of the query might look something like: + +```text +Rc: Debug +``` - Rc: Debug - where `?0` is a variable in the root universe U0. We would then go and -look for a table with this as the key: since the forest is empty, this -lookup will fail, and we will create a new table T0, corresponding to -the u-canonical goal Q. +look for a table with this canonical query as the key: since the forest is +empty, this lookup will fail, and we will create a new table T0, +corresponding to the u-canonical goal Q. **Ignoring negative reasoning and regions.** To start, we'll ignore the possibility of negative goals like `not { Foo }`. We'll phase them @@ -93,7 +98,7 @@ where-clauses that are in scope. In the case of our example, there would be three clauses, each coming from the program. Using a Prolog-like notation, these look like: -``` +```text (u32: Debug). (Rc: Debug) :- (T: Debug). (Vec: Debug) :- (T: Debug). @@ -105,9 +110,9 @@ clauses are inapplicable because `u32` and `Vec` cannot be unified with `Rc`. The second clause, however, will work. **What is a strand?** Let's talk a bit more about what a strand *is*. In the code, a strand -is the combination of an inference table, an X-clause, and (possibly) +is the combination of an inference table, an _X-clause_, and (possibly) a selected subgoal from that X-clause. But what is an X-clause -(`ExClause`, in the code)? An X-clause pulls together a few things: +([`ExClause`], in the code)? An X-clause pulls together a few things: - The current state of the goal we are trying to prove; - A set of subgoals that have yet to be proven; @@ -118,7 +123,9 @@ The general form of an X-clause is written much like a Prolog clause, but with somewhat different semantics. Since we're ignoring delayed literals and region constraints, an X-clause just looks like this: - G :- L +```text +G :- L +``` where G is a goal and L is a set of subgoals that must be proven. (The L stands for *literal* -- when we address negative reasoning, a @@ -128,7 +135,9 @@ that if we are able to prove L then the goal G can be considered true. In the case of our example, we would wind up creating one strand, with an X-clause like so: - (Rc: Debug) :- (?T: Debug) +```text +(Rc: Debug) :- (?T: Debug) +``` Here, the `?T` refers to one of the inference variables created in the inference table that accompanies the strand. (I'll use named variables @@ -141,37 +150,45 @@ is the subgoal after the turnstile (`:-`) that we are currently trying to prove in this strand. Initally, when a strand is first created, there is no selected subgoal. -**Activating a strand.** Now that we have created the table T0 and -initialized it with strands, we have to actually try and produce an -answer. We do this by invoking the `ensure_answer` operation on the -table: specifically, we say `ensure_answer(T0, A0)`, meaning "ensure -that there is a 0th answer". - -Remember that tables store not only strands, but also a vector of -cached answers. The first thing that `ensure_answer` does is to check -whether answer 0 is in this vector. If so, we can just return -immediately. In this case, the vector will be empty, and hence that -does not apply (this becomes important for cyclic checks later on). +[`ExClause`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/struct.ExClause.html -When there is no cached answer, `ensure_answer` will try to produce -one. It does this by selecting a strand from the set of active -strands -- the strands are stored in a `VecDeque` and hence processed -in a round-robin fashion. Right now, we have only one strand, storing -the following X-clause with no selected subgoal: - - (Rc: Debug) :- (?T: Debug) +**Activating a strand.** Now that we have created the table T0 and +initialized it with strands, we have to actually try and produce an answer. +We do this by invoking the [`ensure_root_answer`] operation on the table: +specifically, we say `ensure_root_answer(T0, A0)`, meaning "ensure that there +is a 0th answer A0 to query T0". + +Remember that tables store not only strands, but also a vector of cached +answers. The first thing that [`ensure_root_answer`] does is to check whether +answer A0 is in this vector. If so, we can just return immediately. In this +case, the vector will be empty, and hence that does not apply (this becomes +important for cyclic checks later on). + +When there is no cached answer, [`ensure_root_answer`] will try to produce one. +It does this by selecting a strand from the set of active strands -- the +strands are stored in a `VecDeque` and hence processed in a round-robin +fashion. Right now, we have only one strand, storing the following X-clause +with no selected subgoal: + +```text +(Rc: Debug) :- (?T: Debug) +``` When we activate the strand, we see that we have no selected subgoal, and so we first pick one of the subgoals to process. Here, there is only one (`?T: Debug`), so that becomes the selected subgoal, changing the state of the strand to: - (Rc: Debug) :- selected(?T: Debug, A0) +```text +(Rc: Debug) :- selected(?T: Debug, A0) +``` Here, we write `selected(L, An)` to indicate that (a) the literal `L` is the selected subgoal and (b) which answer `An` we are looking for. We start out looking for `A0`. +[`ensure_root_answer`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/forest/struct.Forest.html#method.ensure_root_answer + **Processing the selected subgoal.** Next, we have to try and find an answer to this selected goal. To do that, we will u-canonicalize it and try to find an associated table. In this case, the u-canonical @@ -189,7 +206,7 @@ will be: We can thus summarize the state of the whole forest at this point as follows: -``` +```text Table T0 [Rc: Debug] Strands: (Rc: Debug) :- selected(?T: Debug, A0) @@ -215,7 +232,7 @@ answer). The strand is then removed from the list of strands. The state of table T1 is therefore: -``` +```text Table T1 [?0: Debug] Answers: A0 = [?0 = u32] @@ -227,8 +244,10 @@ Table T1 [?0: Debug] Note that I am writing out the answer A0 as a substitution that can be applied to the table goal; actually, in the code, the goals for each X-clause are also represented as substitutions, but in this exposition -I've chosen to write them as full goals, following NFTD. - +I've chosen to write them as full goals, following [NFTD]. + +[NFTD]: ./bibliography.html#slg + Since we now have an answer, `ensure_answer(T1, A0)` will return `Ok` to the table T0, indicating that answer A0 is available. T0 now has the job of incorporating that result into its active strand. It does @@ -236,7 +255,7 @@ this in two ways. First, it creates a new strand that is looking for the next possible answer of T1. Next, it incorpoates the answer from A0 and removes the subgoal. The resulting state of table T0 is: -``` +```text Table T0 [Rc: Debug] Strands: (Rc: Debug) :- selected(?T: Debug, A1) @@ -250,7 +269,7 @@ then be returned up to our caller, and the whole forest goes quiescent at this point (remember, we only do enough work to generate *one* answer). The ending state of the forest at this point will be: -``` +```text Table T0 [Rc: Debug] Answer: A0 = [?0 = u32] From 49c3d4a36eebab51f5c767f4798daed72e780b56 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Mon, 26 Nov 2018 11:06:41 -0600 Subject: [PATCH 440/648] add link to stabilization pr --- src/walkthrough.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/walkthrough.md b/src/walkthrough.md index 4eae1397d..e77a0f690 100644 --- a/src/walkthrough.md +++ b/src/walkthrough.md @@ -254,8 +254,16 @@ The stabilization report for our feature is [here][stabrep]. [stabrep]: https://github.com/rust-lang/rust/issues/48075#issuecomment-433243048 -After this, a PR is made to remove the feature gate, enabling the feature by +After this, [a PR is made][stab] to remove the feature gate, enabling the feature by default (on the 2018 edition). A note is added to the [Release notes][relnotes] about the feature. +[stab]: https://github.com/rust-lang/rust/pull/56245 + +TODO: currently, we have a [forge article][feature-stab] about stabilization, but +we really ought to move that to the guide (in fact, we probably should have a whole +chapter about feature gates and stabilization). + +[feature-stab]: https://forge.rust-lang.org/stabilization-guide.html + [relnotes]: https://github.com/rust-lang/rust/blob/master/RELEASES.md From 3a804956e3c28d7e44e38804207a00013594e1d3 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Mon, 26 Nov 2018 11:13:21 -0600 Subject: [PATCH 441/648] make the line-length script respect 100 char limit XD (#240) :rofl: :rofl: :rofl: --- ci/check_line_lengths.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/ci/check_line_lengths.sh b/ci/check_line_lengths.sh index 5b7b12d3e..32bf739e4 100755 --- a/ci/check_line_lengths.sh +++ b/ci/check_line_lengths.sh @@ -30,7 +30,9 @@ for file in "$@" ; do (( inside_block = !$inside_block )) continue fi - if ! (( $inside_block )) && ! [[ "$line" =~ " | "|"-|-"|"://"|"]:"|\[\^[^\ ]+\]: ]] && (( "${#line}" > $MAX_LINE_LENGTH )) ; then + if ! (( $inside_block )) \ + && ! [[ "$line" =~ " | "|"-|-"|"://"|"]:"|\[\^[^\ ]+\]: ]] \ + && (( "${#line}" > $MAX_LINE_LENGTH )) ; then (( bad_lines++ )) echo -e "\t$line_no : $line" fi From 3940e94a5e24e454f662ba5c13c1dceed50ff6a4 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Mon, 26 Nov 2018 15:24:39 -0600 Subject: [PATCH 442/648] fix relative link (#242) --- src/traits/lowering-module.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/traits/lowering-module.md b/src/traits/lowering-module.md index c9c4accc1..939484051 100644 --- a/src/traits/lowering-module.md +++ b/src/traits/lowering-module.md @@ -63,4 +63,4 @@ error: aborting due to previous error [chalkify]: https://github.com/rust-lang/rust/tree/master/src/test/ui/chalkify [lower_impl]: https://github.com/rust-lang/rust/tree/master/src/test/ui/chalkify/lower_impl.rs [the stderr file]: https://github.com/rust-lang/rust/tree/master/src/test/ui/chalkify/lower_impl.stderr -[ui test]: https://rust-lang-nursery.github.io/rustc-guide/tests/adding.html#guide-to-the-ui-tests +[ui test]: ../tests/adding.html#guide-to-the-ui-tests From 2f4e10ce5bf34bc2832b60f5ae4ff344333c3110 Mon Sep 17 00:00:00 2001 From: Vadim Petrochenkov Date: Sun, 2 Dec 2018 01:10:43 +0400 Subject: [PATCH 443/648] Fix failing tidy (line endings on Windows) (#246) This happens every time a new doc submodule is added to https://github.com/rust-lang/rust. cc https://github.com/rust-lang/book/pull/549 https://github.com/rust-lang-nursery/reference/pull/36 https://github.com/rust-lang/rust-by-example/pull/1018 --- .gitattributes | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 .gitattributes diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 000000000..d56abbf30 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,2 @@ +# Set the default behavior, in case people don't have core.autocrlf set. +* text=auto eol=lf From 344c4e437ba4cfa5c14db643ec4d6b68dcd164c5 Mon Sep 17 00:00:00 2001 From: yui-knk Date: Sat, 1 Dec 2018 22:11:05 +0900 Subject: [PATCH 444/648] Fix the file path where AST is defined --- src/high-level-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/high-level-overview.md b/src/high-level-overview.md index 733c523c9..93b5592c3 100644 --- a/src/high-level-overview.md +++ b/src/high-level-overview.md @@ -96,7 +96,7 @@ take: 1. **Parsing input** - this processes the `.rs` files and produces the AST ("abstract syntax tree") - - the AST is defined in `syntax/ast.rs`. It is intended to match the lexical + - the AST is defined in `src/libsyntax/ast.rs`. It is intended to match the lexical syntax of the Rust language quite closely. 2. **Name resolution, macro expansion, and configuration** - once parsing is complete, we process the AST recursively, resolving From a18f366010ecc30920f75e2c12da70d5d38d4ec6 Mon Sep 17 00:00:00 2001 From: Andy Wortman Date: Sat, 1 Dec 2018 11:30:26 -0800 Subject: [PATCH 445/648] fix some linkrotted links --- README.md | 2 +- src/about-this-guide.md | 2 +- src/diag.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 45c7503fe..af23454ba 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ The guide can be useful today, but it has a lot of work still go. If you'd like to help improve the guide, we'd love to have you! You can find plenty of issues on the [issue -tracker](https://github.com/rust-lang-nursery/rustc-guide/issue). Just post a +tracker](https://github.com/rust-lang/rustc-guide/issue). Just post a comment on the issue you would like to work on to make sure that we don't accidentally duplicate work. If you think something is missing, please open an issue about it! diff --git a/src/about-this-guide.md b/src/about-this-guide.md index 79106736f..56f0b7538 100644 --- a/src/about-this-guide.md +++ b/src/about-this-guide.md @@ -11,4 +11,4 @@ be found at the [GitHub repository]. If you find any mistakes in the guide, please file an issue about it, or even better, open a PR with a correction! -[GitHub repository]: https://github.com/rust-lang-nursery/rustc-guide/ +[GitHub repository]: https://github.com/rust-lang/rustc-guide/ diff --git a/src/diag.md b/src/diag.md index 936420ab6..00bb23b4c 100644 --- a/src/diag.md +++ b/src/diag.md @@ -81,7 +81,7 @@ suggestions pleasingly in the terminal, or (when the `--error-format json` flag is passed) as JSON for consumption by tools, most notably the [Rust Language Server][rls] and [`rustfix`][rustfix]. -[rls]: https://github.com/rust-lang-nursery/rls +[rls]: https://github.com/rust-lang/rls [rustfix]: https://github.com/rust-lang-nursery/rustfix Not all suggestions should be applied mechanically. Use the From 0f5df93b1314e2761ac497614138bae5dbfa8b47 Mon Sep 17 00:00:00 2001 From: Edd Barrett Date: Tue, 27 Nov 2018 13:31:38 +0000 Subject: [PATCH 446/648] Mention that tests need a Python-enabled gdb. --- src/tests/running.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/tests/running.md b/src/tests/running.md index f8889c8a4..ceabe3a69 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -19,6 +19,10 @@ timestamp file for every test can be found under `build/ARCH/test/`. To force-rerun a test (e.g. in case the test runner fails to notice a change) you can simply remove the timestamp file. +Note that some tests require a Python-enabled gdb. If you are building +gdb from source, you need to configure with +`--with-python=`. + ## Running a subset of the test suites When working on a specific PR, you will usually want to run a smaller From 8f0eaf39613172e080bc21bda3f3567da0d03632 Mon Sep 17 00:00:00 2001 From: Edd Barrett Date: Mon, 3 Dec 2018 15:46:55 +0000 Subject: [PATCH 447/648] Explain how to check if gdb supports Python. --- src/tests/running.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/tests/running.md b/src/tests/running.md index ceabe3a69..62f3ea4d7 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -19,8 +19,10 @@ timestamp file for every test can be found under `build/ARCH/test/`. To force-rerun a test (e.g. in case the test runner fails to notice a change) you can simply remove the timestamp file. -Note that some tests require a Python-enabled gdb. If you are building -gdb from source, you need to configure with +Note that some tests require a Python-enabled gdb. You can test if +your gdb install supports Python by using the `python` command from +within gdb (type some Python code followed by `CTRL+D` to execute it). +If you are building gdb from source, you need to configure with `--with-python=`. ## Running a subset of the test suites From 87830ff034d89b7e1e370c949aa8085b37bf68ed Mon Sep 17 00:00:00 2001 From: Edd Barrett Date: Mon, 3 Dec 2018 17:27:14 +0000 Subject: [PATCH 448/648] Provide a Python example. --- src/tests/running.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/tests/running.md b/src/tests/running.md index 62f3ea4d7..1250fa77b 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -21,8 +21,9 @@ a change) you can simply remove the timestamp file. Note that some tests require a Python-enabled gdb. You can test if your gdb install supports Python by using the `python` command from -within gdb (type some Python code followed by `CTRL+D` to execute it). -If you are building gdb from source, you need to configure with +within gdb. Once invoked you can type some Python code (e.g. +`print("hi")`) followed by return and then `CTRL+D` to execute it. +If you are building gdb from source, you will need to configure with `--with-python=`. ## Running a subset of the test suites From 022d3f4b56c7c203a1b4ceab961fe5c3e45a993f Mon Sep 17 00:00:00 2001 From: Avelino Date: Wed, 5 Dec 2018 18:12:35 -0200 Subject: [PATCH 449/648] fixed typo issues link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index af23454ba..ba5b1cde4 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ The guide can be useful today, but it has a lot of work still go. If you'd like to help improve the guide, we'd love to have you! You can find plenty of issues on the [issue -tracker](https://github.com/rust-lang/rustc-guide/issue). Just post a +tracker](https://github.com/rust-lang/rustc-guide/issues). Just post a comment on the issue you would like to work on to make sure that we don't accidentally duplicate work. If you think something is missing, please open an issue about it! From f1caa8dd77bcdc3b31532aeecaf6dc06c049e2ff Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Wed, 5 Dec 2018 07:38:34 -0800 Subject: [PATCH 450/648] Add some documentation about updating LLVM --- src/SUMMARY.md | 1 + src/codegen/updating-llvm.md | 149 +++++++++++++++++++++++++++++++++++ 2 files changed, 150 insertions(+) create mode 100644 src/codegen/updating-llvm.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index efe963c3f..bfb112ce6 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -65,6 +65,7 @@ - [miri const evaluator](./miri.md) - [Parameter Environments](./param_env.md) - [Code Generation](./codegen.md) + - [Updating LLVM](./codegen/updating-llvm.md) - [Emitting Diagnostics](./diag.md) --- diff --git a/src/codegen/updating-llvm.md b/src/codegen/updating-llvm.md new file mode 100644 index 000000000..e29e041b2 --- /dev/null +++ b/src/codegen/updating-llvm.md @@ -0,0 +1,149 @@ +# Updating LLVM + +The Rust compiler uses LLVM as its primary codegen backend today, and naturally +we want to at least occasionally update this dependency! Currently we do not +have a strict policy about when to update LLVM or what it can be updated to, but +a few guidelines are applied: + +* We try to always support the latest released version of LLVM +* We try to support the "last few" versions of LLVM (how many is changing over + time) +* We allow moving to arbitrary commits during development. +* Strongly prefer to upstream all patches to LLVM before including them in + rustc. + +This policy may change over time (or may actually start to exist as a formal +policy!), but for now these are rough guidelines! + +## Why update LLVM? + +There are two primary reasons nowadays that we want to update LLVM in one way or +another: + +* First, a bug could have been fixed! Often we find bugs in the compiler and fix + them upstream in LLVM. We'll want to pull fixes back to the compiler itself as + they're merged upstream. + +* Second, a new feature may be avaiable in LLVM that we want to use in rustc, + but we don't want to wait for a full LLVM release to test it out. + +Each of these reasons has a different strategy for updating LLVM, and we'll go +over both in detail here. + +## Bugfix Updates + +For updates of LLVM that typically just update a bug, we cherry-pick the bugfix +to the branch we're already using. The steps for this are: + +1. Make sure the bugfix is in upstream LLVM. +2. Identify the branch that rustc is currently using. The `src/llvm` submodule + is always pinned to a branch of the + [rust-lang/llvm](https://github.com/rust-lang/llvm) repository. +3. Fork the rust-lang/llvm repository +4. Check out the appropriate branch (typically named `rust-llvm-release-*`) +5. Cherry-pick the upstream commit onto the branch +6. Push this branch to your fork +7. Send a Pull Request to rust-lang/llvm to the same branch as before +8. Wait for the PR to be merged +9. Send a PR to rust-lang/rust updating the `src/llvm` submodule with your bugfix +10. Wait for PR to be merged + +The tl;dr; is that we can cherry-pick bugfixes at any time and pull them back +into the rust-lang/llvm branch that we're using, and getting it into the +compiler is just updating the submodule via a PR! + +Example PRs look like: +[#56313](https://github.com/rust-lang/rust/pull/56313) + +## Feature updates + +> Note that this is all information as applies to the current day in age. This +> process for updating LLVM changes with practically all LLVM updates, so this +> may be out of date! + +Unlike bugfixes, updating to pick up a new feature of LLVM typically requires a +lot more work. This is where we can't reasonably cherry-pick commits backwards +so we need to do a full update. There's a lot of stuff to do here, so let's go +through each in detail. + +1. Create new branches in all repositories for this update. Branches should be + named `rust-llvm-release-X-Y-Z-vA` where `X.Y.Z` is the LLVM version and `A` + is just increasing based on if there's previous branches of this name. All + repositories here should be branched at the same time from the upstream LLVM + projects, we currently use https://github.com/llvm-mirror repositories. The + list of repositories that need a new branch are: + + * rust-lang/llvm + * rust-lang/compiler-rt + * rust-lang/lld + * rust-lang-nursery/lldb + * rust-lang-nursery/clang + +2. Apply Rust-specific patches to LLVM repositories. All features and bugfixes + are upstream, but there's often some weird build-related patches that don't + make sense to upstream which we have on our repositories. These patches are + typically the latest patches on the branch. All repositories, except `clang`, + currently have Rust-specific patches. + +3. Update the `compiler-rt` submodule in the + `rust-lang-nursery/compiler-builtins` repository. Push this update to a + `rust-llvm-release-*` branch of the `compiler-builtins` repository. + +4. Prepare a commit to rust-lang/rust + + * Update `src/llvm` + * Update `src/tools/lld` + * Update `src/tools/lldb` + * Update `src/tools/clang` + * Update `src/libcompiler_builtins + * Edit `src/rustllvm/llvm-rebuild-trigger` to update its contents + +5. Build your commit. Make sure you've committed the previous changes to ensure + submodule updates aren't reverted. Some commands you should execute are: + + * `./x.py build src/llvm` - test that LLVM still builds + * `./x.py build src/tools/lld` - same for LLD + * `./x.py build` - build the rest of rustc + + You'll likely need to update `src/rustllvm/*.cpp` to compile with updated + LLVM bindings. Note that you should use `#ifdef` and such to ensure that the + bindings still compile on older LLVM versions. + +6. Test for regressions across other platforms. LLVM often has at least one bug + for non-tier-1 architectures, so it's good to do some more testing before + sending this to bors! If you're low on resources you can send the PR as-is + now to bors, though, and it'll get tested anyway. + + Ideally, build LLVM and test it on a few platforms: + + * Linux + * OSX + * Windows + + and afterwards run some docker containers that CI also does: + + * `./src/ci/docker/run.sh wasm32-unknown` + * `./src/ci/docker/run.sh arm-android` + * `./src/ci/docker/run.sh dist-various-1` + * `./src/ci/docker/run.sh dist-various-2` + * `./src/ci/docker/run.sh armhf-gnu` + +7. Send a PR! Hopefully it's smooth sailing from here :). + +For prior art, previous LLVM updates look like +[#55835](https://github.com/rust-lang/rust/pull/55835) +[#47828](https://github.com/rust-lang/rust/pull/47828) + +### Caveats and gotchas + +Ideally the above instructions are pretty smooth, but here's some caveats to +keep in mind while going through them: + +* LLVM bugs are hard to find, don't hesitate to ask for help! Bisection is + definitely your friend here (yes LLVM takes forever to build, yet bisection is + still your friend) +* Updating LLDB has some Rust-specific patches currently that aren't upstream. + If you have difficulty @tromey can likely help out. +* If you've got general questions, @alexcrichton can help you out. +* Creating branches is a privileged operation on GitHub, so you'll need someone + with write access to create the branches for you most likely. From 7acbc48d17c9274934c6f5fdae7da203269a9b85 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 11 Dec 2018 14:46:35 -0600 Subject: [PATCH 451/648] a few updates --- src/borrow_check/region_inference.md | 32 ++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index d08e06a80..762a84477 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -48,26 +48,40 @@ The MIR-based region analysis consists of two major functions: ## Universal regions -The [`UnversalRegions`] type represents a collection of _unversal_ regions +The [`UnversalRegions`] type represents a collection of _universal_ regions corresponding to some MIR `DefId`. It is constructed in [`replace_regions_in_mir`] when we replace all regions with fresh inference variables. [`UniversalRegions`] contains indices for all the free regions in the given MIR along with any relationships that are _known_ to hold between them (e.g. implied bounds, where clauses, etc.). -TODO: is there more to write here? +For example, given the MIR for the following function: + +```rust +fn foo<'a>(x: &'a u32) { + // ... +} +``` + +we would create a universal region for `'a` and one for `'static`. There may +also be some complications for handling closures, but we will ignore those for +the moment. + +TODO: write about _how_ these regions are computed. [`UniversalRegions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/universal_regions/struct.UniversalRegions.html ## Region variables -The value of a region can be thought of as a **set** of points in the MIR where -the region is valid; we call the domain of this set a `RegionElement`. In the -code, the value for all regions is maintained in -[the `rustc_mir::borrow_check::nll::region_infer` module][ri]. For -each region we maintain a set storing what elements are present in its -value (to make this efficient, we give each kind of element an index, -the `RegionElementIndex`, and use sparse bitsets). +The value of a region can be thought of as a **set**. This set contains all +points in the MIR where the region is valid along with any regions that are +outlived by this region (e.g. if `'a: 'b`, then `end('b)` is in the set for +`'a`); we call the domain of this set a `RegionElement`. In the code, the value +for all regions is maintained in [the +`rustc_mir::borrow_check::nll::region_infer` module][ri]. For each region we +maintain a set storing what elements are present in its value (to make this +efficient, we give each kind of element an index, the `RegionElementIndex`, and +use sparse bitsets). [ri]: https://github.com/rust-lang/rust/tree/master/src/librustc_mir/borrow_check/nll/region_infer/ From a69982d7f5b74d743158f5170e27e60811cde6cc Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 11 Dec 2018 15:22:17 -0600 Subject: [PATCH 452/648] added example, reworked inference section --- src/borrow_check/region_inference.md | 125 ++++++++++++++++++++------- 1 file changed, 93 insertions(+), 32 deletions(-) diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index 762a84477..7fe50b870 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -120,45 +120,115 @@ So how do we compute the contents of a region? This process is called _region inference_. The high-level idea is pretty simple, but there are some details we need to take care of. +Here is the high-level idea: we start off each region with the MIR locations we +know must be in it from the liveness constraints. From there, we use all of the +outlives constraints computed from the type checker to _propagate_ the +constraints: for each region `'a`, if `'a: 'b`, then we add all elements of +`'b` to `'a`, including `end('b)`. This all happens in +[`propagate_constraints`]. + +Then, we will check for errors. We first check that type tests are satisfied by +calling [`check_type_tests`]. This checks constraints like `T: 'a`. Second, we +check that universal regions are not "too big". This is done by calling +[`check_universal_regions`]. This checks that for each region `'a` if `'a` +contains the element `end('b)`, then we must already know that `'a: 'b` holds +(e.g. from a where clause). If we don't already know this, that is an error... +well, almost. There is some special handling for closures that we will discuss +later. + +### Example + +Consider the following example: + +```rust,ignore +fn foo<'a, 'b>(x: &'a usize) -> &'b usize { + x +} +``` + +Clearly, this should not compile because we don't know if `'a` outlives `'b` +(if it doesn't then the return value could be a dangling reference). + +Let's back up a bit. We need to introduce some free inference variables (as is +done in [`replace_regions_in_mir`]). This example doesn't use the exact regions +produced, but it (hopefully) is enough to get the idea across. + +```rust,ignore +fn foo<'a, 'b>(x: &'a /* '#1 */ usize) -> &'b /* '#3 */ usize { + x // '#2, location L1 +} +``` + +Some notation: `'#1`, `'#3`, and `'#2` represent the universal regions for the +argument, return value, and the expression `x`, respectively. Additionally, I +will call the location of the expression `x` `L1`. + +So now we can use the liveness constraints to get the following starting points: + +Region | Contents +--------|---------- +'#1 | +'#2 | `L1` +'#3 | `L1` + +Now we use the outlives constraints to expand each region. Specifically, we +know that `'#2: '#3` ... + +Region | Contents +--------|---------- +'#1 | `L1` +'#2 | `L1, end('#3) // add contents of '#3 and end('#3)` +'#3 | `L1` + +... and `'#1: '#2`, so ... + +Region | Contents +--------|---------- +'#1 | `L1, end('#2), end('#3) // add contents of '#2 and end('#2)` +'#2 | `L1, end('#3)` +'#3 | `L1` + +Now, we need to check that no regions were too big (we don't have any type +tests to check in this case). Notice that `'#1` now contains `end('#3)`, but +we have no `where` clause or implied bound to say that `'a: 'b`... that's an +error! + +### Some details + The [`RegionInferenceContext`] type contains all of the information needed to -do inference, including the universal regions from `replace_regions_in_mir` and +do inference, including the universal regions from [`replace_regions_in_mir`] and the constraints computed for each region. It is constructed just after we compute the liveness constraints. Here are some of the fields of the struct: -- `constraints`: contains all the outlives constraints. -- `liveness_constraints`: contains all the liveness constraints. -- `universal_regions`: contains the `UniversalRegions` returned by - `replace_regions_in_mir`. -- `universal_region_relations`: contains relations known to be true about +- [`constraints`]: contains all the outlives constraints. +- [`liveness_constraints`]: contains all the liveness constraints. +- [`universal_regions`]: contains the `UniversalRegions` returned by + [`replace_regions_in_mir`]. +- [`universal_region_relations`]: contains relations known to be true about universal regions. For example, if we have a where clause that `'a: 'b`, that relation is assumed to be true while borrow checking the implementation (it is checked at the caller), so `universal_region_relations` would contain `'a: 'b`. -- `type_tests`: contains some constraints on types that we must check after +- [`type_tests`]: contains some constraints on types that we must check after inference (e.g. `T: 'a`). -- `closure_bounds_mapping`: used for propagating region constraints from +- [`closure_bounds_mapping`]: used for propagating region constraints from closures back out to the creater of the closure. +[`constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#structfield.constraints +[`liveness_constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#structfield.liveness_constraints +[`universal_regions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#structfield.universal_regions +[`universal_region_relations`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#structfield.universal_region_relations +[`type_tests`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#structfield.type_tests +[`closure_bounds_mapping`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#structfield.closure_bounds_mapping + TODO: should we discuss any of the others fields? What about the SCCs? Ok, now that we have constructed a `RegionInferenceContext`, we can do -inference. This is done by calling the [`solve`] method on the context. - -We will start off the value of each region with the liveness constraints (the -places we already know must be in the region). We will then use the outlives -constraints to widen each region until all constraints are met. This is done in -[`propagate_constraints`]. For each region, if `'a: 'b`, we add all elements of -`'b` to `'a`. - -Then, we will check for errors. We first check that type tests are satisfied by -calling [`check_type_tests`]. This checks constraints like `T: 'a`. Second, we -check that universal regions are not "too big". This is done by calling -[`check_universal_regions`]. This checks that for each region `'a` if `'a` -contains the element `end('b)`, then we must already know that `'a: 'b` holds -(e.g. from a where clause). If we don't already know this, that is an error... -well, almost. +inference. This is done by calling the [`solve`] method on the context. This +is where we call [`propagate_constraints`] and then check the resulting type +tests and universal regions, as discussed above. [`propagate_constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.propagate_constraints [`check_type_tests`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.check_type_tests @@ -174,15 +244,6 @@ and return them. Later, when we are borrow check the MIR node that created the closure, we can also check that these constraints hold. At that time, if we can't prove they hold, we report an error. -## Causal tracking - -*to be written* – describe how we can extend the values of a variable - with causal tracking etc - -TODO: is this what I described above or something else? - - - ## Placeholders and universes (This section describes ongoing work that hasn't landed yet.) From 2dfa301deb5fe48a2afc2c2c9dc4218cd91b37a7 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 11 Dec 2018 15:25:03 -0600 Subject: [PATCH 453/648] update a couple of links --- src/compiler-team.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/compiler-team.md b/src/compiler-team.md index a6328198c..728325687 100644 --- a/src/compiler-team.md +++ b/src/compiler-team.md @@ -1,11 +1,11 @@ # About the compiler team -rustc is maintained by the -[Rust compiler team](https://www.rust-lang.org/en-US/team.html). The -people who belong to this team collectively work to track regressions -and implement new features. Members of the Rust compiler team are -people who have made significant contributions to rustc and its -design. +rustc is maintained by the [Rust compiler team][team]. The people who belong to +this team collectively work to track regressions and implement new features. +Members of the Rust compiler team are people who have made significant +contributions to rustc and its design. + +[team]: https://www.rust-lang.org/governance/teams/language-and-compiler ## Discussion @@ -89,7 +89,7 @@ The guidelines for reviewers are as follows: Rust project, so it is expected that you will go above and beyond when it comes to the [Code of Conduct]. -[Code of Conduct]: https://www.rust-lang.org/en-US/conduct.html +[Code of Conduct]: https://www.rust-lang.org/policies/code-of-conduct ### high-five From 81b0c84f416930ded5c4817240a779e789964982 Mon Sep 17 00:00:00 2001 From: Dan Robertson Date: Wed, 12 Dec 2018 02:08:40 +0000 Subject: [PATCH 454/648] Fix link to rustc_mir::hair::cx::expr --- src/mir/construction.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/mir/construction.md b/src/mir/construction.md index 03e220cb4..f0e3cf76a 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -24,9 +24,8 @@ being a `hair::ExprKind::Neg(hair::Expr)` it is a `hair::ExprKind::Neg(hir::Expr This shallowness enables the `HAIR` to represent all datatypes that [HIR] has, but without having to create an in-memory copy of the entire [HIR]. [MIR] lowering will first convert the topmost expression from -[HIR] to [HAIR] (in -[https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/hair/cx/expr/index.html]) -and then process the [HAIR] expressions recursively. +[HIR] to [HAIR] (in [rustc_mir::hair::cx::expr]) and then process +the [HAIR] expressions recursively. The lowering creates local variables for every argument as specified in the signature. Next it creates local variables for every binding specified (e.g. `(a, b): (i32, String)`) @@ -147,4 +146,6 @@ case of `enum`s. [MIR]: ./index.html [HIR]: ../hir.html [HAIR]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/hair/index.html + +[rustc_mir::hair::cx::expr]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/hair/cx/expr/index.html [`mir_built`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/transform/fn.mir_built.html From aee77489fed126f2b4317a79c037c8c52f8a98c4 Mon Sep 17 00:00:00 2001 From: Oliver Scherer Date: Fri, 14 Dec 2018 16:58:19 +0100 Subject: [PATCH 455/648] Add graphs that I forgot to push in the original PR --- src/mir/construction.md | 7 + src/mir/mir_detailed.dot | 27 ++++ src/mir/mir_detailed.svg | 300 +++++++++++++++++++++++++++++++++++++++ src/mir/mir_overview.dot | 10 ++ src/mir/mir_overview.svg | 103 ++++++++++++++ 5 files changed, 447 insertions(+) create mode 100644 src/mir/mir_detailed.dot create mode 100644 src/mir/mir_detailed.svg create mode 100644 src/mir/mir_overview.dot create mode 100644 src/mir/mir_overview.svg diff --git a/src/mir/construction.md b/src/mir/construction.md index f0e3cf76a..0e0bf35e7 100644 --- a/src/mir/construction.md +++ b/src/mir/construction.md @@ -92,6 +92,13 @@ There are essentially four kinds of representations one might want of an express * `Operand` is an argument to e.g. a `+` operation or a function call * a temporary variable containing a copy of the value +These following image depicts a general overview of the interactions between the +representations: + + + +[Click here for a more detailed view](mir_detailed.svg) + We start out with lowering the function body to an `Rvalue` so we can create an assignment to `RETURN_PLACE`, This `Rvalue` lowering will in turn trigger lowering to `Operand` for its arguments (if any). `Operand` lowering either produces a `const` diff --git a/src/mir/mir_detailed.dot b/src/mir/mir_detailed.dot new file mode 100644 index 000000000..1494f4f0b --- /dev/null +++ b/src/mir/mir_detailed.dot @@ -0,0 +1,27 @@ +digraph G { + Operand -> Constant + Operand -> Place [taillabel="read"] + Place -> Projection + Projection -> Place + Place -> LocalId + Rvalue -> "Rvalue\nAggregate" + Rvalue -> "Rvalue\nBinaryOp" + Rvalue -> "Rvalue\nUnaryOp" + Rvalue -> "Rvalue\n..." + "Rvalue\nAggregate" -> Operand [headlabel="*"] + "Rvalue\nBinaryOp" -> Operand [headlabel="2"] + "Rvalue\nUnaryOp" -> Operand + "Statement\nAssignment" -> Place [taillabel="write"] + "Statement\nAssignment" -> Rvalue + Statement -> "Statement\nAssignment" + Statement -> "Statement\n..." + Block -> Statement [headlabel="*"] + Block -> Terminator + Terminator -> "Terminator\nSwitchInt" + "Terminator\nSwitchInt" -> Operand + "Terminator\nSwitchInt" -> Constant [headlabel="*"] + "Terminator\nSwitchInt" -> BlockId [headlabel="*"] + Terminator -> "Terminator\n..." + Mir -> Block [headlabel="*"] + Mir -> Local [headlabel="*"] +} \ No newline at end of file diff --git a/src/mir/mir_detailed.svg b/src/mir/mir_detailed.svg new file mode 100644 index 000000000..771e9e5dc --- /dev/null +++ b/src/mir/mir_detailed.svg @@ -0,0 +1,300 @@ + + + + + + +G + + + +Operand + +Operand + + + +Constant + +Constant + + + +Operand->Constant + + + + + +Place + +Place + + + +Operand->Place + + +read + + + +Projection + +Projection + + + +Place->Projection + + + + + +LocalId + +LocalId + + + +Place->LocalId + + + + + +Projection->Place + + + + + +Rvalue + +Rvalue + + + +Rvalue\nAggregate + +Rvalue +Aggregate + + + +Rvalue->Rvalue\nAggregate + + + + + +Rvalue\nBinaryOp + +Rvalue +BinaryOp + + + +Rvalue->Rvalue\nBinaryOp + + + + + +Rvalue\nUnaryOp + +Rvalue +UnaryOp + + + +Rvalue->Rvalue\nUnaryOp + + + + + +Rvalue\n... + +Rvalue +... + + + +Rvalue->Rvalue\n... + + + + + +Rvalue\nAggregate->Operand + + +* + + + +Rvalue\nBinaryOp->Operand + + +2 + + + +Rvalue\nUnaryOp->Operand + + + + + +Statement\nAssignment + +Statement +Assignment + + + +Statement\nAssignment->Place + + +write + + + +Statement\nAssignment->Rvalue + + + + + +Statement + +Statement + + + +Statement->Statement\nAssignment + + + + + +Statement\n... + +Statement +... + + + +Statement->Statement\n... + + + + + +Block + +Block + + + +Block->Statement + + +* + + + +Terminator + +Terminator + + + +Block->Terminator + + + + + +Terminator\nSwitchInt + +Terminator +SwitchInt + + + +Terminator->Terminator\nSwitchInt + + + + + +Terminator\n... + +Terminator +... + + + +Terminator->Terminator\n... + + + + + +Terminator\nSwitchInt->Operand + + + + + +Terminator\nSwitchInt->Constant + + +* + + + +BlockId + +BlockId + + + +Terminator\nSwitchInt->BlockId + + +* + + + +Mir + +Mir + + + +Mir->Block + + +* + + + +Local + +Local + + + +Mir->Local + + +* + + + diff --git a/src/mir/mir_overview.dot b/src/mir/mir_overview.dot new file mode 100644 index 000000000..c0b51674d --- /dev/null +++ b/src/mir/mir_overview.dot @@ -0,0 +1,10 @@ +digraph G { + Operand -> Constant + Operand -> Place + Place -> Projection + Projection -> Place + Place -> Local + Rvalue -> Operand + Assignment -> Place + Assignment -> Operand +} \ No newline at end of file diff --git a/src/mir/mir_overview.svg b/src/mir/mir_overview.svg new file mode 100644 index 000000000..b8f092cd2 --- /dev/null +++ b/src/mir/mir_overview.svg @@ -0,0 +1,103 @@ + + + + + + +G + + + +Operand + +Operand + + + +Constant + +Constant + + + +Operand->Constant + + + + + +Place + +Place + + + +Operand->Place + + + + + +Projection + +Projection + + + +Place->Projection + + + + + +Local + +Local + + + +Place->Local + + + + + +Projection->Place + + + + + +Rvalue + +Rvalue + + + +Rvalue->Operand + + + + + +Assignment + +Assignment + + + +Assignment->Operand + + + + + +Assignment->Place + + + + + From 28f67849e16323a80b3f8efd3f8963179db85f04 Mon Sep 17 00:00:00 2001 From: Roberto Vidal Date: Fri, 14 Dec 2018 10:11:27 +0100 Subject: [PATCH 456/648] Fixes broken links --- src/diag.md | 2 +- src/hir.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/diag.md b/src/diag.md index 00bb23b4c..9a8020314 100644 --- a/src/diag.md +++ b/src/diag.md @@ -305,4 +305,4 @@ are defining a new lint, you will want to add an entry to this enum. Then, add an appropriate mapping to the body of [`Lint::from_parser_lint_id`][fplid]. [`BufferedEarlyLintId`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/early_buffered_lints/enum.BufferedEarlyLintId.html -[fplid]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/lint/struct.Lint.html#from_parser_lint_id +[fplid]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/lint/struct.Lint.html#method.from_parser_lint_id diff --git a/src/hir.md b/src/hir.md index e6bca7f37..34e478ee4 100644 --- a/src/hir.md +++ b/src/hir.md @@ -105,12 +105,12 @@ sorts of identifiers in active use: ### The HIR Map Most of the time when you are working with the HIR, you will do so via -the **HIR Map**, accessible in the tcx via [`tcx.hir`] (and defined in +the **HIR Map**, accessible in the tcx via [`tcx.hir_map`] (and defined in the [`hir::map`] module). The [HIR map] contains a [number of methods] to convert between IDs of various kinds and to lookup data associated with an HIR node. -[`tcx.hir`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/context/struct.GlobalCtxt.html#structfield.hir +[`tcx.hir_map`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/context/struct.GlobalCtxt.html#structfield.hir_map [`hir::map`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/index.html [HIR map]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/struct.Map.html [number of methods]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/map/struct.Map.html#methods From 451f00936a3319ffd94b03cc811d1a4521c507b7 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Sun, 23 Dec 2018 10:11:26 -0600 Subject: [PATCH 457/648] Fix link --- src/appendix/code-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index e1bde6863..546e9f9b7 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -23,7 +23,7 @@ Item | Kind | Short description | Chapter | `SourceMap` | struct | Maps AST nodes to their source code. It is composed of `SourceFile`s. Was previously called CodeMap | [The parser] | [src/libsyntax/source_map.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html) `Span` | struct | A location in the user's source code, used for error reporting primarily | [Emitting Diagnostics] | [src/libsyntax_pos/span_encoding.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax_pos/struct.Span.html) `StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser] | [src/libsyntax/parse/lexer/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/lexer/struct.StringReader.html) -`syntax::token_stream::TokenStream` | struct | An abstract sequence of tokens, organized into `TokenTree`s | [The parser], [Macro expansion] | [src/libsyntax/tokenstream.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/tokenstream/struct.TokenStream.html) +`syntax::token_stream::TokenStream` | struct | An abstract sequence of tokens, organized into `TokenTree`s | [The parser], [Macro expansion] | [src/libsyntax/tokenstream.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/tokenstream/enum.TokenStream.html) `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules] | [src/librustc/ty/trait_def.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/trait_def/struct.TraitDef.html) `TraitRef` | struct | The combination of a trait and its input types (e.g. `P0: Trait`) | [Trait Solving: Goals and Clauses], [Trait Solving: Lowering impls] | [src/librustc/ty/sty.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TraitRef.html) `Ty<'tcx>` | struct | This is the internal representation of a type used for type checking | [Type checking] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/type.Ty.html) From b85fc377539931211544d905da6d454c7f921155 Mon Sep 17 00:00:00 2001 From: ednix Date: Sat, 22 Dec 2018 23:49:31 +0530 Subject: [PATCH 458/648] Clarify line about RLS being unable to handle rustc I based this clarification on this [answer](https://www.reddit.com/r/rust/comments/a8mk33/question_about_a_line_from_the_rust_compiler_book/ecbx1ma/) to my question on Reddit. --- src/how-to-build-and-run.md | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 89ad47199..cc1a5f9c0 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -45,10 +45,10 @@ debuginfo-lines = true ### What is x.py? -x.py is the script used to orchestrate the tooling in the rustc repository. -It is the script that can build docs, run tests, and compile rustc. -It is the now preferred way to build rustc and it replaces the old makefiles -from before. Below are the different ways to utilize x.py in order to +x.py is the script used to orchestrate the tooling in the rustc repository. +It is the script that can build docs, run tests, and compile rustc. +It is the now preferred way to build rustc and it replaces the old makefiles +from before. Below are the different ways to utilize x.py in order to effectively deal with the repo for various common tasks. ### Running x.py and building a stage1 compiler @@ -86,8 +86,8 @@ compiling `rustc` is done in stages: #### Build Flags -There are other flags you can pass to the build portion of x.py that can be -beneficial to cutting down compile times or fitting other things you might +There are other flags you can pass to the build portion of x.py that can be +beneficial to cutting down compile times or fitting other things you might need to change. They are: ```bash @@ -180,9 +180,9 @@ build`) has quite a few more steps: > ./x.py build src/libcore --stage 1 ``` -Sometimes you might just want to test if the part you’re working on can -compile. Using these commands you can test that it compiles before doing -a bigger build to make sure it works with the compiler. As shown before +Sometimes you might just want to test if the part you’re working on can +compile. Using these commands you can test that it compiles before doing +a bigger build to make sure it works with the compiler. As shown before you can also pass flags at the end such as --stage. @@ -260,7 +260,7 @@ The sequence of commands you want is as follows: stage1 compiler - Subsequent builds: `./x.py build -i --stage 1 src/libstd --keep-stage 1` - Note that we added the `--keep-stage 1` flag here - + The effect of `--keep-stage 1` is that we just *assume* that the old standard library can be re-used. If you are editing the compiler, this is almost always true: you haven't changed the standard library, after @@ -300,9 +300,7 @@ in other sections: ### ctags -One of the challenges with rustc is that the RLS can't handle it, making code -navigation difficult. One solution is to use `ctags`. The following script can -be used to set it up: [https://github.com/nikomatsakis/rust-etags][etags]. +One of the challenges with rustc is that the RLS can't handle it, making code navigation difficult. One solution is to use ctags. The following script can be used to set it up: https://github.com/nikomatsakis/rust-etags. CTAGS integrates into emacs and vim quite easily. The following can then be used to build and generate tags: @@ -318,9 +316,9 @@ you last built, which is ridiculously useful. ### Cleaning out build directories -Sometimes you need to start fresh, but this is normally not the case. -If you need to run this then rustbuild is most likely not acting right and -you should file a bug as to what is going wrong. If you do need to clean +Sometimes you need to start fresh, but this is normally not the case. +If you need to run this then rustbuild is most likely not acting right and +you should file a bug as to what is going wrong. If you do need to clean everything up then you only need to run one command! ```bash From 1b760946bd734a2f623b6d3661fe508d965a7e87 Mon Sep 17 00:00:00 2001 From: ednix Date: Sat, 22 Dec 2018 23:54:05 +0530 Subject: [PATCH 459/648] Do what the last commit said it did --- src/how-to-build-and-run.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index cc1a5f9c0..077ea39f9 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -300,7 +300,7 @@ in other sections: ### ctags -One of the challenges with rustc is that the RLS can't handle it, making code navigation difficult. One solution is to use ctags. The following script can be used to set it up: https://github.com/nikomatsakis/rust-etags. +One of the challenges with rustc is that the RLS can't handle it, since it's a bootstrapping compiler. This makes code navigation difficult. One solution is to use ctags. The following script can be used to set it up: https://github.com/nikomatsakis/rust-etags. CTAGS integrates into emacs and vim quite easily. The following can then be used to build and generate tags: From 5daaa543311e6b4993abdf9ae71ed290962c2a12 Mon Sep 17 00:00:00 2001 From: ednix Date: Sun, 23 Dec 2018 13:18:56 +0530 Subject: [PATCH 460/648] Fix formatting, follow 100 char line length limit No idea why the Markdown stuff got dropped last time. Hopefully everything is fine now. --- src/how-to-build-and-run.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 077ea39f9..241852b19 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -300,7 +300,9 @@ in other sections: ### ctags -One of the challenges with rustc is that the RLS can't handle it, since it's a bootstrapping compiler. This makes code navigation difficult. One solution is to use ctags. The following script can be used to set it up: https://github.com/nikomatsakis/rust-etags. +One of the challenges with rustc is that the RLS can't handle it, since it's a bootstrapping +compiler. This makes code navigation difficult. One solution is to use `ctags`. The following +script can be used to set it up: [https://github.com/nikomatsakis/rust-etags][etags]. CTAGS integrates into emacs and vim quite easily. The following can then be used to build and generate tags: From 615cf17b6ffdeec711b09bef523e6dc99140c1ed Mon Sep 17 00:00:00 2001 From: Philipp Hansch Date: Tue, 1 Jan 2019 11:26:10 +0100 Subject: [PATCH 461/648] Add IRLO to glossary Someone was asking on Discord what it stands for. It's sometimes used in GitHub issues, Twitter and other places. --- src/appendix/glossary.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 82d6f9a21..0e697aeea 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -33,6 +33,7 @@ ICH | incremental compilation hash. ICHs are used as finger inference variable | when doing type or region inference, an "inference variable" is a kind of special type/region that represents what you are trying to infer. Think of X in algebra. For example, if we are trying to infer the type of a variable in a program, we create an inference variable to represent that unknown type. infcx | the inference context (see `librustc/infer`) IR | Intermediate Representation. A general term in compilers. During compilation, the code is transformed from raw source (ASCII text) to various IRs. In Rust, these are primarily HIR, MIR, and LLVM IR. Each IR is well-suited for some set of computations. For example, MIR is well-suited for the borrow checker, and LLVM IR is well-suited for codegen because LLVM accepts it. +IRLO | `IRLO` or `irlo` is sometimes used as an abbreviation for [internals.rust-lang.org](https://internals.rust-lang.org). local crate | the crate currently being compiled. LTO | Link-Time Optimizations. A set of optimizations offered by LLVM that occur just before the final binary is linked. These include optimizations like removing functions that are never used in the final program, for example. _ThinLTO_ is a variant of LTO that aims to be a bit more scalable and efficient, but possibly sacrifices some optimizations. You may also read issues in the Rust repo about "FatLTO", which is the loving nickname given to non-Thin LTO. LLVM documentation: [here][lto] and [here][thinlto] [LLVM] | (actually not an acronym :P) an open-source compiler backend. It accepts LLVM IR and outputs native binaries. Various languages (e.g. Rust) can then implement a compiler front-end that output LLVM IR and use LLVM to compile to all the platforms LLVM supports. From 134d2ba510c33da75033baa634279305dd297c40 Mon Sep 17 00:00:00 2001 From: Igor Matuszewski Date: Fri, 4 Jan 2019 15:23:15 +0100 Subject: [PATCH 462/648] Mention old incr. comp. design doc --- src/incremental-compilation.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/incremental-compilation.md b/src/incremental-compilation.md index 0a25e1664..bed21f115 100644 --- a/src/incremental-compilation.md +++ b/src/incremental-compilation.md @@ -136,6 +136,9 @@ algorithm like so: - We can then compare the hash of the result and color Q as green if it did not change. +## Resources +The initial design document can be found at https://github.com/nikomatsakis/rustc-on-demand-incremental-design-doc/blob/master/0000-rustc-on-demand-and-incremental.md, which expands on the memoization details, provides more high-level overview and motivation for this system. + # Footnotes [^salsa]: I have long wanted to rename it to the Salsa algorithm, but it never caught on. -@nikomatsakis From 1590387f778eb58e5969f88134833442e9a7ea92 Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Tue, 15 Jan 2019 17:07:06 -0500 Subject: [PATCH 463/648] issue_130_stabilization_guide --- src/SUMMARY.md | 1 + src/stabilization_guide.md | 181 +++++++++++++++++++++++++++++++++++++ 2 files changed, 182 insertions(+) create mode 100644 src/stabilization_guide.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index bfb112ce6..9bdf89278 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -6,6 +6,7 @@ - [Build and Install distribution artifacts](./build-install-distribution-artifacts.md) - [Documenting Compiler](./compiler-documenting.md) - [Coding conventions](./conventions.md) +- [Stabilizing Features](./stabilization_guide.md) - [Walkthrough: a typical contribution](./walkthrough.md) - [The compiler testing framework](./tests/intro.md) - [Running tests](./tests/running.md) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md new file mode 100644 index 000000000..af86b450e --- /dev/null +++ b/src/stabilization_guide.md @@ -0,0 +1,181 @@ + +# Request for stabilization + + Once an unstable feature has been well-tested with no outstanding + concern, anyone may push for its stabilization. It involves the + following steps. + + - Documentation PRs + - Write a stabilization report + - FCP + - Stabilization PR + +## Documentation PRs + + Prepare PRs to update documentations involing this new feature. + You need to submit PRs for repositories The Reference, The Book + and Rust by Example. + Maintainers of these repositories will keep these PRs open until + the whole stabilization process has completed. Meanwhile, we can + proceed to the next step. + +## Write a stabilization report + + Find the tracking issue of the feature, and create a short + stabilization report. Essentially this would be a brief summary + of the feature plus some links to test cases showing it works + as expected, along with a list of edge cases that came up and + and were considered. This is a minimal "due diligence" that + we do before stabilizing. + + The report should contain: + + - A summary, showing examples (e.g. code snippets) what is + enabled by this feature. + - Links to test cases in our test suite regarding this feature + and describe the feature's behavior on encountering edge cases. + - Links to the documentations (the PRs we have made in the + previous steps). + - Any other relevant information(Examples of such reports can + be found in rust-lang/rust#44494 and rust-lang/rust#28237). + +## FCP + +If any member of the team responsible for tracking this +feature agrees with stabilizing this feature, they will +start the FCP (final-comment-period) process by +commenting + + ```bash + @rfcbot fcp merge + ``` + +The rest of the team members will review the proposal. If the final +decision is to stabilize, we proceed to do the actual code modification. + +## Stabilization PR + +Once we have decided to stabilize a feature, we need to have a PR that +actually makes that stabilization happen. These kinds of PRs are a +great way to get involved in Rust, as they take you on a little tour +through the source code. + +Here is a general guide to how to stabilize a feature -- every feature +is different, of course, so some features may require steps beyond +what this guide talks about. + +Note: Before we stabilize any feature, it is rule that it should appear +in the documentation. + +### Updating the feature-gate listing + +There is a central listing of feature-gates in +`src/libsyntax/feature_gate.rs`. Search for the `declare_features!` +macro. There should be an entry for the feature you are aiming to +stabilize, something like (this example is taken from +[rust-lang/rust#32409]: + + ``` + // pub(restricted) visibilities (RFC 1422) + (active, pub_restricted, "1.9.0", Some(32409)), + ``` +The above line should be moved down to the area for "accepted" +features, declared below in a separate call to `declare_features!`. +When it is done, it should look like: + + ``` + // pub(restricted) visibilities (RFC 1422) + (accepted, pub_restricted, "1.31.0", Some(32409)), + // ^^^^^^ note that we changed this + ``` + +Note that, the version number is updated to be the version number +of the stable release where this feature will appear. This can be +found by consulting https://forge.rust-lang.org/, which will guide +you the next stable release number. You want to add 1 to that, +because the code that lands today will become go into beta on that +date, and then become stable after that. So, at the time of this +writing, the next stable release (what is currently beta, iow) was +1.30.0, hence I wrote 1.31.0 above. + +### Removing existing uses of the feature-gate + +Next search for the feature string (in this case, pub_restricted) +in the codebase to find where it appears. Change uses of +`#![feature(XXX)]` from the stdlib and rustc crates to be +`#![cfg_attr(stage0, feature(XXX))]`. This includes the feature-gate +only for stage0, which is built using the current beta (this is +needed because the feature is still unstable in the current beta). + +Also, remove those strings from any tests. If there are tests +specifically targeting the feature-gate (i.e., testing that the +feature-gate is required to use the feature, but nothing else), +simply remove the test. + +### Do not require the feature-gate to use the feature + +Most importantly, remove the code which flags an error if the +feature-gate is not present (since the feature is now considered +stable). If the feature can be detected because it employs some +new syntax, then a common place for that code to be is in the +same `feature_gate.rs`. For example, you might see code like this: + + ``` + gate_feature_post!(&self, pub_restricted, span, + "`pub(restricted)` syntax is experimental"); + ``` + +This `gate_feature_post!` macro prints an error if the +`pub_restricted` feature is not enabled. It is not needed +now that `#[pub_restricted]` is stable. + +For more subtle features, you may find code like this: + + ``` + if self.tcx.sess.features.borrow().pub_restricted { /* XXX */ } + ``` + +This `pub_restricted` field (obviously named after the feature) +would ordinarily be false if the feature flag is not present +and true if it is. So transform the code to assume that the field +is true. In this case, that would mean removing the `if` and +leaving just the `/* XXX */`. + + ``` + if self.tcx.sess.features.borrow().pub_restricted { /* XXX */ } + = becomes ==> + /* XXX */ + + if self.tcx.sess.features.borrow().pub_restricted && something { /* XXX */ } + = becomes ==> + if something { /* XXX */ } + ``` + +## Updating documentation + +If any documentation for this feature exists, it should be +in the `Unstable Book`, located at `src/doc/unstable-book`. +Regardless of its existence, the page for the feature gate +should be removed. + +If there was documentation there, integrating it into the +existing documentation is needed. + +If there wasn't documentation there, it needs to be added. + +Places that may need updated documentation: + + [The Reference]: this must be updated, in full detail. + [The Book]: this may or may not need updating, depending. + If you're not sure, please open an issue on this repository + and it can be discussed. + standard library documentation: as needed. Language features + often don't need this, but if it's a feature that changes + how good examples are written, such as when `?` was added + to the language, updating examples is important. + [Rust by Example]: as needed. + +[rust-lang/rust#32409]:https://github.com/rust-lang/rust/issues/32409 +[The Reference]: https://github.com/rust-lang-nursery/reference +[The Book]: https://github.com/rust-lang/book +[Rust by Example]: https://github.com/rust-lang/rust-by-example \ No newline at end of file From afc027f7c574f4f8ec403cfa532f38af1fd59579 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Tue, 15 Jan 2019 21:00:36 -0500 Subject: [PATCH 464/648] Update src/stabilization_guide.md Co-Authored-By: rajcspsg --- src/stabilization_guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index af86b450e..db8b5c3f4 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -64,7 +64,7 @@ Here is a general guide to how to stabilize a feature -- every feature is different, of course, so some features may require steps beyond what this guide talks about. -Note: Before we stabilize any feature, it is rule that it should appear +Note: Before we stabilize any feature, it's the rule that it should appear in the documentation. ### Updating the feature-gate listing @@ -178,4 +178,4 @@ Places that may need updated documentation: [rust-lang/rust#32409]:https://github.com/rust-lang/rust/issues/32409 [The Reference]: https://github.com/rust-lang-nursery/reference [The Book]: https://github.com/rust-lang/book -[Rust by Example]: https://github.com/rust-lang/rust-by-example \ No newline at end of file +[Rust by Example]: https://github.com/rust-lang/rust-by-example From dd43338012e2ad8cc872fdd8757737c1e75b6747 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Tue, 15 Jan 2019 21:02:20 -0500 Subject: [PATCH 465/648] Update src/stabilization_guide.md Co-Authored-By: rajcspsg --- src/stabilization_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index db8b5c3f4..22ec49f8a 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -91,7 +91,7 @@ When it is done, it should look like: Note that, the version number is updated to be the version number of the stable release where this feature will appear. This can be -found by consulting https://forge.rust-lang.org/, which will guide +found by consulting [the forge](https://forge.rust-lang.org/), which will guide you the next stable release number. You want to add 1 to that, because the code that lands today will become go into beta on that date, and then become stable after that. So, at the time of this From ea37742d2c670257efbe525338098ac9f641e3cc Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Tue, 15 Jan 2019 21:02:36 -0500 Subject: [PATCH 466/648] Update src/stabilization_guide.md Co-Authored-By: rajcspsg --- src/stabilization_guide.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index 22ec49f8a..3af4aff73 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -147,7 +147,9 @@ leaving just the `/* XXX */`. /* XXX */ if self.tcx.sess.features.borrow().pub_restricted && something { /* XXX */ } - = becomes ==> + ``` +becomes +```rust if something { /* XXX */ } ``` From 8823da540fb2e953140c07e4eaae7c73486c5282 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Tue, 15 Jan 2019 21:02:51 -0500 Subject: [PATCH 467/648] Update src/stabilization_guide.md Co-Authored-By: rajcspsg --- src/stabilization_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index 3af4aff73..d21fbb174 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -157,7 +157,7 @@ becomes If any documentation for this feature exists, it should be in the `Unstable Book`, located at `src/doc/unstable-book`. -Regardless of its existence, the page for the feature gate +If it exists, the page for the feature gate should be removed. If there was documentation there, integrating it into the From 0f82f1c5a175b2229a4da57dad2d2395c58ab421 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Tue, 15 Jan 2019 21:03:13 -0500 Subject: [PATCH 468/648] Update src/stabilization_guide.md Co-Authored-By: rajcspsg --- src/stabilization_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index d21fbb174..a245d170b 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -100,7 +100,7 @@ writing, the next stable release (what is currently beta, iow) was ### Removing existing uses of the feature-gate -Next search for the feature string (in this case, pub_restricted) +Next search for the feature string (in this case, `pub_restricted`) in the codebase to find where it appears. Change uses of `#![feature(XXX)]` from the stdlib and rustc crates to be `#![cfg_attr(stage0, feature(XXX))]`. This includes the feature-gate From e680f28d059b8e3ca0a5404de3dcbd55735aacf1 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Tue, 15 Jan 2019 21:10:56 -0500 Subject: [PATCH 469/648] Update src/stabilization_guide.md Co-Authored-By: rajcspsg --- src/stabilization_guide.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index a245d170b..d73e96d8c 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -143,7 +143,9 @@ leaving just the `/* XXX */`. ``` if self.tcx.sess.features.borrow().pub_restricted { /* XXX */ } - = becomes ==> + ``` +becomes +```rust /* XXX */ if self.tcx.sess.features.borrow().pub_restricted && something { /* XXX */ } From ca9e6be0c6563131fbbfdd7ddb2ef07fff7dc2e4 Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Tue, 15 Jan 2019 22:22:36 -0500 Subject: [PATCH 470/648] issue_180 incorporated the review comments --- src/stabilization_guide.md | 157 +++++++++++++++++-------------------- 1 file changed, 74 insertions(+), 83 deletions(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index d73e96d8c..2974f6314 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -1,43 +1,61 @@ - # Request for stabilization - Once an unstable feature has been well-tested with no outstanding - concern, anyone may push for its stabilization. It involves the - following steps. +Once an unstable feature has been well-tested with no outstanding +concern, anyone may push for its stabilization. It involves the +following steps. - - Documentation PRs - - Write a stabilization report - - FCP - - Stabilization PR +- Documentation PRs +- Write a stabilization report +- FCP +- Stabilization PR ## Documentation PRs - Prepare PRs to update documentations involing this new feature. - You need to submit PRs for repositories The Reference, The Book - and Rust by Example. - Maintainers of these repositories will keep these PRs open until - the whole stabilization process has completed. Meanwhile, we can - proceed to the next step. +If any documentation for this feature exists, it should be +in the `Unstable Book`, located at `src/doc/unstable-book`. +If it exists, the page for the feature gate should be removed. + +If there was documentation there, integrating it into the +existing documentation is needed. + +If there wasn't documentation there, it needs to be added. + +Places that may need updated documentation: + + [The Reference]: This must be updated, in full detail. + [The Book]: This may or may not need updating, depends. + If you're not sure, please open an issue on this repository + and it can be discussed. + standard library documentation: As needed. Language features + often don't need this, but if it's a feature that changes + how good examples are written, such as when `?` was added + to the language, updating examples is important. + [Rust by Example]: As needed. + +Prepare PRs to update documentations invovling this new feature +for repositories mentioned above.Maintainers of these repositories +will keep these PRs open until the whole stabilization process +has completed. Meanwhile, we can proceed to the next step. ## Write a stabilization report - Find the tracking issue of the feature, and create a short - stabilization report. Essentially this would be a brief summary - of the feature plus some links to test cases showing it works - as expected, along with a list of edge cases that came up and - and were considered. This is a minimal "due diligence" that - we do before stabilizing. +Find the tracking issue of the feature, and create a short +stabilization report. Essentially this would be a brief summary +of the feature plus some links to test cases showing it works +as expected, along with a list of edge cases that came up and +and were considered. This is a minimal "due diligence" that +we do before stabilizing. - The report should contain: +The report should contain: - - A summary, showing examples (e.g. code snippets) what is - enabled by this feature. - - Links to test cases in our test suite regarding this feature - and describe the feature's behavior on encountering edge cases. - - Links to the documentations (the PRs we have made in the - previous steps). - - Any other relevant information(Examples of such reports can - be found in rust-lang/rust#44494 and rust-lang/rust#28237). +- A summary, showing examples (e.g. code snippets) what is + enabled by this feature. +- Links to test cases in our test suite regarding this feature + and describe the feature's behavior on encountering edge cases. +- Links to the documentations (the PRs we have made in the + previous steps). +- Any other relevant information(Examples of such reports can + be found in rust-lang/rust#44494 and rust-lang/rust#28237). ## FCP @@ -46,9 +64,9 @@ feature agrees with stabilizing this feature, they will start the FCP (final-comment-period) process by commenting - ```bash - @rfcbot fcp merge - ``` +```bash +@rfcbot fcp merge +``` The rest of the team members will review the proposal. If the final decision is to stabilize, we proceed to do the actual code modification. @@ -75,19 +93,20 @@ macro. There should be an entry for the feature you are aiming to stabilize, something like (this example is taken from [rust-lang/rust#32409]: - ``` - // pub(restricted) visibilities (RFC 1422) - (active, pub_restricted, "1.9.0", Some(32409)), - ``` +```rust,ignore +// pub(restricted) visibilities (RFC 1422) +(active, pub_restricted, "1.9.0", Some(32409)), +``` + The above line should be moved down to the area for "accepted" features, declared below in a separate call to `declare_features!`. When it is done, it should look like: - ``` - // pub(restricted) visibilities (RFC 1422) - (accepted, pub_restricted, "1.31.0", Some(32409)), - // ^^^^^^ note that we changed this - ``` +```rust,ignore +// pub(restricted) visibilities (RFC 1422) +(accepted, pub_restricted, "1.31.0", Some(32409)), +// note that we changed this +``` Note that, the version number is updated to be the version number of the stable release where this feature will appear. This can be @@ -120,10 +139,10 @@ stable). If the feature can be detected because it employs some new syntax, then a common place for that code to be is in the same `feature_gate.rs`. For example, you might see code like this: - ``` - gate_feature_post!(&self, pub_restricted, span, - "`pub(restricted)` syntax is experimental"); - ``` +```rust,ignore +gate_feature_post!(&self, pub_restricted, span, + "`pub(restricted)` syntax is experimental"); +``` This `gate_feature_post!` macro prints an error if the `pub_restricted` feature is not enabled. It is not needed @@ -131,9 +150,9 @@ now that `#[pub_restricted]` is stable. For more subtle features, you may find code like this: - ``` - if self.tcx.sess.features.borrow().pub_restricted { /* XXX */ } - ``` +```rust,ignore +if self.tcx.sess.features.borrow().pub_restricted { /* XXX */ } +``` This `pub_restricted` field (obviously named after the feature) would ordinarily be false if the feature flag is not present @@ -141,43 +160,15 @@ and true if it is. So transform the code to assume that the field is true. In this case, that would mean removing the `if` and leaving just the `/* XXX */`. - ``` - if self.tcx.sess.features.borrow().pub_restricted { /* XXX */ } - ``` +```rust,ignore +if self.tcx.sess.features.borrow().pub_restricted { /* XXX */ } becomes -```rust - /* XXX */ +/* XXX */ - if self.tcx.sess.features.borrow().pub_restricted && something { /* XXX */ } - ``` -becomes -```rust - if something { /* XXX */ } - ``` - -## Updating documentation - -If any documentation for this feature exists, it should be -in the `Unstable Book`, located at `src/doc/unstable-book`. -If it exists, the page for the feature gate -should be removed. - -If there was documentation there, integrating it into the -existing documentation is needed. - -If there wasn't documentation there, it needs to be added. - -Places that may need updated documentation: - - [The Reference]: this must be updated, in full detail. - [The Book]: this may or may not need updating, depending. - If you're not sure, please open an issue on this repository - and it can be discussed. - standard library documentation: as needed. Language features - often don't need this, but if it's a feature that changes - how good examples are written, such as when `?` was added - to the language, updating examples is important. - [Rust by Example]: as needed. +if self.tcx.sess.features.borrow().pub_restricted && something { /* XXX */ } + becomes +if something { /* XXX */ } +``` [rust-lang/rust#32409]:https://github.com/rust-lang/rust/issues/32409 [The Reference]: https://github.com/rust-lang-nursery/reference From d3e8792c2cd2ce7e66e28240ecde4fb20c126f21 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 17 Jan 2019 14:04:50 -0500 Subject: [PATCH 471/648] Update src/stabilization_guide.md Co-Authored-By: rajcspsg --- src/stabilization_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index 2974f6314..a17eff162 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -114,7 +114,7 @@ found by consulting [the forge](https://forge.rust-lang.org/), which will guide you the next stable release number. You want to add 1 to that, because the code that lands today will become go into beta on that date, and then become stable after that. So, at the time of this -writing, the next stable release (what is currently beta, iow) was +writing, the next stable release (i.e. what is currently beta) was 1.30.0, hence I wrote 1.31.0 above. ### Removing existing uses of the feature-gate From 74fd971003d9ea38e55b3fc6e3eeaabcc4d9f1c4 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 17 Jan 2019 14:05:13 -0500 Subject: [PATCH 472/648] Update src/stabilization_guide.md Co-Authored-By: rajcspsg --- src/stabilization_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index a17eff162..89407308d 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -121,7 +121,7 @@ writing, the next stable release (i.e. what is currently beta) was Next search for the feature string (in this case, `pub_restricted`) in the codebase to find where it appears. Change uses of -`#![feature(XXX)]` from the stdlib and rustc crates to be +`#![feature(XXX)]` from the `libstd` and any rustc crates to be `#![cfg_attr(stage0, feature(XXX))]`. This includes the feature-gate only for stage0, which is built using the current beta (this is needed because the feature is still unstable in the current beta). From c36a489d97caa449f926b449d070c9de31dcb1bc Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Thu, 17 Jan 2019 14:09:41 -0500 Subject: [PATCH 473/648] issue 130 stabilization guide --- src/stabilization_guide.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index 89407308d..e56313bb0 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -22,15 +22,15 @@ If there wasn't documentation there, it needs to be added. Places that may need updated documentation: - [The Reference]: This must be updated, in full detail. - [The Book]: This may or may not need updating, depends. +- [The Reference]: This must be updated, in full detail. +- [The Book]: This may or may not need updating, depends. If you're not sure, please open an issue on this repository and it can be discussed. - standard library documentation: As needed. Language features +- standard library documentation: As needed. Language features often don't need this, but if it's a feature that changes how good examples are written, such as when `?` was added to the language, updating examples is important. - [Rust by Example]: As needed. +- [Rust by Example]: As needed. Prepare PRs to update documentations invovling this new feature for repositories mentioned above.Maintainers of these repositories From a230029888ab3b66ef18e4a86aa18c643996849c Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Thu, 17 Jan 2019 14:20:24 -0500 Subject: [PATCH 474/648] issue 130 stabilization guide --- src/stabilization_guide.md | 5 ++++- src/walkthrough.md | 4 ---- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index e56313bb0..7f3c7321a 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -56,6 +56,8 @@ The report should contain: previous steps). - Any other relevant information(Examples of such reports can be found in rust-lang/rust#44494 and rust-lang/rust#28237). +- The resolutions of any unresolved questions if the stabilization + is for an RFC. ## FCP @@ -88,7 +90,7 @@ in the documentation. ### Updating the feature-gate listing There is a central listing of feature-gates in -`src/libsyntax/feature_gate.rs`. Search for the `declare_features!` +`[src/libsyntax/feature_gate.rs]`. Search for the `declare_features!` macro. There should be an entry for the feature you are aiming to stabilize, something like (this example is taken from [rust-lang/rust#32409]: @@ -171,6 +173,7 @@ if something { /* XXX */ } ``` [rust-lang/rust#32409]:https://github.com/rust-lang/rust/issues/32409 +[src/libsyntax/feature_gate.rs]:https://doc.rust-lang.org/nightly/nightly-rustc/syntax/feature_gate/index.html [The Reference]: https://github.com/rust-lang-nursery/reference [The Book]: https://github.com/rust-lang/book [Rust by Example]: https://github.com/rust-lang/rust-by-example diff --git a/src/walkthrough.md b/src/walkthrough.md index e77a0f690..fa259de99 100644 --- a/src/walkthrough.md +++ b/src/walkthrough.md @@ -260,10 +260,6 @@ about the feature. [stab]: https://github.com/rust-lang/rust/pull/56245 -TODO: currently, we have a [forge article][feature-stab] about stabilization, but -we really ought to move that to the guide (in fact, we probably should have a whole -chapter about feature gates and stabilization). - [feature-stab]: https://forge.rust-lang.org/stabilization-guide.html [relnotes]: https://github.com/rust-lang/rust/blob/master/RELEASES.md From 996d9493a23da668f4992be47f764a23d9908c63 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 17 Jan 2019 16:40:07 -0500 Subject: [PATCH 475/648] Update src/stabilization_guide.md Co-Authored-By: rajcspsg --- src/stabilization_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index 7f3c7321a..c217c2251 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -33,7 +33,7 @@ Places that may need updated documentation: - [Rust by Example]: As needed. Prepare PRs to update documentations invovling this new feature -for repositories mentioned above.Maintainers of these repositories +for repositories mentioned above. Maintainers of these repositories will keep these PRs open until the whole stabilization process has completed. Meanwhile, we can proceed to the next step. From fe1c4b73a1a203ff8ad65a47928f23b2e0227468 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 17 Jan 2019 16:40:18 -0500 Subject: [PATCH 476/648] Update src/stabilization_guide.md Co-Authored-By: rajcspsg --- src/stabilization_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index c217c2251..6296aaa01 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -90,7 +90,7 @@ in the documentation. ### Updating the feature-gate listing There is a central listing of feature-gates in -`[src/libsyntax/feature_gate.rs]`. Search for the `declare_features!` +[`src/libsyntax/feature_gate.rs`]. Search for the `declare_features!` macro. There should be an entry for the feature you are aiming to stabilize, something like (this example is taken from [rust-lang/rust#32409]: From 72237edfa4d4f2ce90dcb178671e030079c0f2b6 Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Thu, 17 Jan 2019 17:04:03 -0500 Subject: [PATCH 477/648] issue 130 stabilization guide --- src/stabilization_guide.md | 29 +++++++++++++++-------------- src/walkthrough.md | 4 +--- 2 files changed, 16 insertions(+), 17 deletions(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index 6296aaa01..eb2c64298 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -12,7 +12,7 @@ following steps. ## Documentation PRs If any documentation for this feature exists, it should be -in the `Unstable Book`, located at `src/doc/unstable-book`. +in the [`Unstable Book`], located at [`src/doc/unstable-book`]. If it exists, the page for the feature gate should be removed. If there was documentation there, integrating it into the @@ -63,8 +63,7 @@ The report should contain: If any member of the team responsible for tracking this feature agrees with stabilizing this feature, they will -start the FCP (final-comment-period) process by -commenting +start the FCP (final-comment-period) process by commenting ```bash @rfcbot fcp merge @@ -75,24 +74,24 @@ decision is to stabilize, we proceed to do the actual code modification. ## Stabilization PR -Once we have decided to stabilize a feature, we need to have a PR that -actually makes that stabilization happen. These kinds of PRs are a -great way to get involved in Rust, as they take you on a little tour -through the source code. +Once we have decided to stabilize a feature, we need to have +a PR that actually makes that stabilization happen. These kinds +of PRs are a great way to get involved in Rust, as they take +you on a little tour through the source code. -Here is a general guide to how to stabilize a feature -- every feature -is different, of course, so some features may require steps beyond -what this guide talks about. +Here is a general guide to how to stabilize a feature -- +every feature is different, of course, so some features may +require steps beyond what this guide talks about. -Note: Before we stabilize any feature, it's the rule that it should appear -in the documentation. +Note: Before we stabilize any feature, it's the rule that it +should appear in the documentation. ### Updating the feature-gate listing There is a central listing of feature-gates in [`src/libsyntax/feature_gate.rs`]. Search for the `declare_features!` -macro. There should be an entry for the feature you are aiming to -stabilize, something like (this example is taken from +macro. There should be an entry for the feature you are aiming +to stabilize, something like (this example is taken from [rust-lang/rust#32409]: ```rust,ignore @@ -177,3 +176,5 @@ if something { /* XXX */ } [The Reference]: https://github.com/rust-lang-nursery/reference [The Book]: https://github.com/rust-lang/book [Rust by Example]: https://github.com/rust-lang/rust-by-example +[`Unstable Book`]: https://doc.rust-lang.org/unstable-book/index.html +[`src/doc/unstable-book`]: https://github.com/rust-lang/rust/tree/master/src/doc/unstable-book \ No newline at end of file diff --git a/src/walkthrough.md b/src/walkthrough.md index fa259de99..2823426be 100644 --- a/src/walkthrough.md +++ b/src/walkthrough.md @@ -258,8 +258,6 @@ After this, [a PR is made][stab] to remove the feature gate, enabling the featur default (on the 2018 edition). A note is added to the [Release notes][relnotes] about the feature. -[stab]: https://github.com/rust-lang/rust/pull/56245 - -[feature-stab]: https://forge.rust-lang.org/stabilization-guide.html +Steps to stabilize the feature can be found at [Stabilizing Features](./stabilization_guide.md). [relnotes]: https://github.com/rust-lang/rust/blob/master/RELEASES.md From 1aadda5c5e1b8c1ce8a558891f346362d16edbd7 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Thu, 17 Jan 2019 18:57:05 -0600 Subject: [PATCH 478/648] Update stabilization_guide.md --- src/stabilization_guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index eb2c64298..ab5340c95 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -172,9 +172,9 @@ if something { /* XXX */ } ``` [rust-lang/rust#32409]:https://github.com/rust-lang/rust/issues/32409 -[src/libsyntax/feature_gate.rs]:https://doc.rust-lang.org/nightly/nightly-rustc/syntax/feature_gate/index.html +[`src/libsyntax/feature_gate.rs`]:https://doc.rust-lang.org/nightly/nightly-rustc/syntax/feature_gate/index.html [The Reference]: https://github.com/rust-lang-nursery/reference [The Book]: https://github.com/rust-lang/book [Rust by Example]: https://github.com/rust-lang/rust-by-example [`Unstable Book`]: https://doc.rust-lang.org/unstable-book/index.html -[`src/doc/unstable-book`]: https://github.com/rust-lang/rust/tree/master/src/doc/unstable-book \ No newline at end of file +[`src/doc/unstable-book`]: https://github.com/rust-lang/rust/tree/master/src/doc/unstable-book From be6722764c701acb698ff697ec45dcd9fc1995aa Mon Sep 17 00:00:00 2001 From: king6cong Date: Fri, 18 Jan 2019 00:06:00 +0800 Subject: [PATCH 479/648] Update the doc related to complier logging --- src/compiler-debugging.md | 59 ++++++++++----------------------------- 1 file changed, 15 insertions(+), 44 deletions(-) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index aca7f7424..19c939d6d 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -1,7 +1,3 @@ -**Note: This is copied from the -[rust-forge](https://github.com/rust-lang-nursery/rust-forge). If anything needs - updating, please open an issue or make a PR on the github repo.** - # Debugging the compiler [debugging]: #debugging @@ -134,6 +130,11 @@ $ # Cool, now I have a backtrace for the error ## Getting logging output [getting-logging-output]: #getting-logging-output +These crates are used in compiler for logging: + +* [log] +* [env-logger]: check the link to see the full `RUST_LOG` syntax + The compiler has a lot of `debug!` calls, which print out logging information at many points. These are very useful to at least narrow down the location of a bug if not to find it entirely, or just to orient yourself as to why the @@ -141,10 +142,8 @@ compiler is doing a particular thing. To see the logs, you need to set the `RUST_LOG` environment variable to your log filter, e.g. to get the logs for a specific module, you can run the -compiler as `RUST_LOG=module::path rustc my-file.rs`. The Rust logs are -powered by [env-logger], and you can look at the docs linked there to see -the full `RUST_LOG` syntax. All `debug!` output will then appear in -standard error. +compiler as `RUST_LOG=module::path rustc my-file.rs`. All `debug!` output will +then appear in standard error. Note that unless you use a very strict filter, the logger will emit a *lot* of output - so it's typically a good idea to pipe standard error to a file @@ -176,8 +175,10 @@ $ RUST_LOG=debug rustc +local my-file.rs 2>all-log $ RUST_LOG=rustc_trans=info rustc +local my-file.rs ``` -While calls to `info!` are included in every build of the compiler, -calls to `debug!` are only included in the program if the +### How to keep or remove `debug!` and `trace!` calls from the resulting binary + +While calls to `error!`, `warn!` and `info!` are included in every build of the compiler, +calls to `debug!` and `trace!` are only included in the program if `debug-assertions=yes` is turned on in config.toml (it is turned off by default), so if you don't see `DEBUG` logs, especially if you run the compiler with `RUST_LOG=rustc rustc some.rs` and only see @@ -200,48 +201,18 @@ However, there are still a few concerns that you might care about: ### Expensive operations in logs -A note of caution: the expressions *within* the `debug!` call are run -whenever RUST_LOG is set, even if the filter would exclude the log. This means -that if in the module `rustc::foo` you have a statement +If in the module `rustc::foo` you have a statement ```Rust debug!("{:?}", random_operation(tcx)); ``` Then if someone runs a debug `rustc` with `RUST_LOG=rustc::bar`, then -`random_operation()` will still run - even while it's output will never be -needed! +`random_operation()` will run. This means that you should not put anything too expensive or likely to crash there - that would annoy anyone who wants to use logging for their own -module. Note that if `RUST_LOG` is unset (the default), then the code will not -run - this means that if your logging code panics, then no-one will know it -until someone tries to use logging to find *another* bug. - -If you *need* to do an expensive operation in a log, be aware that while log -expressions are *evaluated* even if logging is not enabled in your module, -they are not *formatted* unless it *is*. This means you can put your -expensive/crashy operations inside an `fmt::Debug` impl, and they will not be -run unless your log is enabled: - -```Rust -use std::fmt; - -struct ExpensiveOperationContainer<'a, 'gcx, 'tcx> - where 'tcx: 'gcx, 'a: 'tcx -{ - tcx: TyCtxt<'a, 'gcx, 'tcx> -} - -impl<'a, 'gcx, 'tcx> fmt::Debug for ExpensiveOperationContainer<'a, 'gcx, 'tcx> { - fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result { - let value = random_operation(tcx); - fmt::Debug::fmt(&value, fmt) - } -} - -debug!("{:?}", ExpensiveOperationContainer { tcx }); -``` +module. No-one will know it until someone tries to use logging to find *another* bug. ## Formatting Graphviz output (.dot files) [formatting-graphviz-output]: #formatting-graphviz-output @@ -382,7 +353,7 @@ create a minimal working example with Godbolt. Go to 5. Once you have a godbolt link demonstrating the issue, it is pretty easy to fill in an LLVM bug. - +[log]: https://docs.rs/log/0.4.6/log/index.html [env-logger]: https://docs.rs/env_logger/0.4.3/env_logger/ ## Narrowing (Bisecting) Regressions From ae6381c6d76ecbfb5aea676f73cba87815464745 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Fri, 18 Jan 2019 14:09:30 -0600 Subject: [PATCH 480/648] Fix relative link --- src/appendix/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 0e697aeea..af099d09a 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -54,7 +54,7 @@ rib | a data structure in the name resolver that keeps trac sess | the compiler session, which stores global data used throughout compilation side tables | because the AST and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. sigil | like a keyword but composed entirely of non-alphanumeric tokens. For example, `&` is a sigil for references. -placeholder | **NOTE: skolemization is deprecated by placeholder** a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)`) as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on placeholder and universes](../borrow_check/region_inference.html#placeholder) for more details. +placeholder | **NOTE: skolemization is deprecated by placeholder** a way of handling subtyping around "for-all" types (e.g., `for<'a> fn(&'a u32)`) as well as solving higher-ranked trait bounds (e.g., `for<'a> T: Trait<'a>`). See [the chapter on placeholder and universes](../borrow_check/region_inference.html#placeholders-and-universes) for more details. soundness | soundness is a technical term in type theory. Roughly, if a type system is sound, then if a program type-checks, it is type-safe; i.e. I can never (in safe rust) force a value into a variable of the wrong type. (see "completeness"). span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. substs | the substitutions for a given generic type or item (e.g. the `i32`, `u32` in `HashMap`) From e900144d759fdf0245c2365e36016281106f252d Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Fri, 18 Jan 2019 14:36:52 -0600 Subject: [PATCH 481/648] Typo + minor edits --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index ba5b1cde4..e6230439e 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -This is a collaborate effort to build a guide that explains how rustc +This is a collaborative effort to build a guide that explains how rustc works. The aim of the guide is to help new contributors get oriented to rustc, as well as to help more experienced folks in figuring out some new part of the compiler that they haven't worked on before. @@ -9,10 +9,10 @@ You may also find the rustdocs [for the compiler itself][rustdocs] useful. [rustdocs]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ -The guide can be useful today, but it has a lot of work still go. - ### Contributing to the guide +The guide is useful today, but it has a lot of work still go. + If you'd like to help improve the guide, we'd love to have you! You can find plenty of issues on the [issue tracker](https://github.com/rust-lang/rustc-guide/issues). Just post a From 8375007aa8b428f3973b208b9c6bbc91c903c35d Mon Sep 17 00:00:00 2001 From: mark Date: Fri, 18 Jan 2019 15:16:13 -0600 Subject: [PATCH 482/648] Reorganize the book into 3 parts --- src/SUMMARY.md | 27 +++++++++++++++++---------- src/about-this-guide.md | 5 +++++ src/part-1-intro.md | 7 +++++++ src/part-2-intro.md | 12 ++++++++++++ 4 files changed, 41 insertions(+), 10 deletions(-) create mode 100644 src/part-1-intro.md create mode 100644 src/part-2-intro.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 9bdf89278..a17a3c6a0 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -1,21 +1,28 @@ # Summary -- [About this guide](./about-this-guide.md) +[About this guide](./about-this-guide.md) + +--- + +- [Part 1 Intro](./part-1-intro.md) - [About the compiler team](./compiler-team.md) - [How to build the compiler and run what you built](./how-to-build-and-run.md) - [Build and Install distribution artifacts](./build-install-distribution-artifacts.md) - [Documenting Compiler](./compiler-documenting.md) -- [Coding conventions](./conventions.md) -- [Stabilizing Features](./stabilization_guide.md) -- [Walkthrough: a typical contribution](./walkthrough.md) - [The compiler testing framework](./tests/intro.md) - [Running tests](./tests/running.md) - [Adding new tests](./tests/adding.md) - - [Using `compiletest` + commands to control test - execution](./compiletest.md) + - [Using `compiletest` + commands to control test execution](./compiletest.md) +- [Walkthrough: a typical contribution](./walkthrough.md) +- [Stabilizing Features](./stabilization_guide.md) - [Debugging the Compiler](./compiler-debugging.md) - [Profiling the compiler](./profiling.md) - [with the linux perf tool](./profiling/with_perf.md) +- [Coding conventions](./conventions.md) + +--- + +- [Part 2 Intro](./part-2-intro.md) - [High-level overview of the compiler source](./high-level-overview.md) - [The Rustc Driver](./rustc-driver.md) - [Rustdoc](./rustdoc.md) @@ -71,7 +78,7 @@ --- -- [Appendix A: Stupid Stats](./appendix/stupid-stats.md) -- [Appendix B: Background material](./appendix/background.md) -- [Appendix C: Glossary](./appendix/glossary.md) -- [Appendix D: Code Index](./appendix/code-index.md) +[Appendix A: Stupid Stats](./appendix/stupid-stats.md) +[Appendix B: Background material](./appendix/background.md) +[Appendix C: Glossary](./appendix/glossary.md) +[Appendix D: Code Index](./appendix/code-index.md) diff --git a/src/about-this-guide.md b/src/about-this-guide.md index 56f0b7538..30a3b8870 100644 --- a/src/about-this-guide.md +++ b/src/about-this-guide.md @@ -6,6 +6,11 @@ development. It is not meant to replace code documentation – each chapter gives only high-level details – the kinds of things that (ideally) don't change frequently. +There are three parts to this guide. Part 1 contains information that should +be useful no matter how you are contributing. Part 2 contains information +about how the compiler works. Finally, there are some appendices at the +end with useful reference information. + The guide itself is of course open-source as well, and the sources can be found at the [GitHub repository]. If you find any mistakes in the guide, please file an issue about it, or even better, open a PR diff --git a/src/part-1-intro.md b/src/part-1-intro.md new file mode 100644 index 000000000..c829f8c3b --- /dev/null +++ b/src/part-1-intro.md @@ -0,0 +1,7 @@ +# Part 1: General Knowledge + +This section of the rustc-guide contains knowledge that should be useful to you +regardless of what part of the compiler you are working on. This includes both +technical info and tips (e.g. how to compile and debug the compiler) and info +about processes in the Rust project (e.g. stabilization and info about the +compiler team). diff --git a/src/part-2-intro.md b/src/part-2-intro.md new file mode 100644 index 000000000..6dd26f44c --- /dev/null +++ b/src/part-2-intro.md @@ -0,0 +1,12 @@ +# Part 2: How rustc works + +This part of the guide describes how the compiler works. It goes through +everything from high-level structure of the compiler to how each stage of +compilation works. + +This section should be friendly to both readers interested in the end-to-end +process of compilation _and_ readers interested in learning about a specific +system they wish to contribute to. If anything is unclear, feel free to file +an issue on the [rustc-guide repo](https://github.com/rust-lang/rustc-guide) +or contact the compiler team, as detailed in [this chapter from Part +1](./compiler-team.md). From 2c5ff5503a620c555a87bc5743e3dbbdbec6a94e Mon Sep 17 00:00:00 2001 From: mark Date: Fri, 18 Jan 2019 15:32:17 -0600 Subject: [PATCH 483/648] better titles a la Niko --- src/SUMMARY.md | 4 ++-- src/part-1-intro.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index a17a3c6a0..4df03dc7a 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -4,7 +4,7 @@ --- -- [Part 1 Intro](./part-1-intro.md) +- [Part 1: Building, debugging, and contributing to Rustc](./part-1-intro.md) - [About the compiler team](./compiler-team.md) - [How to build the compiler and run what you built](./how-to-build-and-run.md) - [Build and Install distribution artifacts](./build-install-distribution-artifacts.md) @@ -22,7 +22,7 @@ --- -- [Part 2 Intro](./part-2-intro.md) +- [Part 2: How rustc works](./part-2-intro.md) - [High-level overview of the compiler source](./high-level-overview.md) - [The Rustc Driver](./rustc-driver.md) - [Rustdoc](./rustdoc.md) diff --git a/src/part-1-intro.md b/src/part-1-intro.md index c829f8c3b..65cb1ae48 100644 --- a/src/part-1-intro.md +++ b/src/part-1-intro.md @@ -1,4 +1,4 @@ -# Part 1: General Knowledge +# Part 1: Building, debugging, and contributing to Rustc This section of the rustc-guide contains knowledge that should be useful to you regardless of what part of the compiler you are working on. This includes both From 3a79ba54320110f88096c747935172f86f8dec00 Mon Sep 17 00:00:00 2001 From: rchaser53 Date: Sat, 19 Jan 2019 23:48:13 +0900 Subject: [PATCH 484/648] fix related miri variables rename variables like below - Value => ConstValue - PrimVal => Scalar - Value::ByVal => ConstValue::Scalar - Value::ByValPair => ConstValue::ScalarPair --- src/const-eval.md | 6 +++--- src/miri.md | 16 ++++++++-------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/src/const-eval.md b/src/const-eval.md index 1f801fb22..aaa2ed2d6 100644 --- a/src/const-eval.md +++ b/src/const-eval.md @@ -30,9 +30,9 @@ used) and a `GlobalId`. The `GlobalId` is made up of an Constant evaluation returns a `Result` with either the error, or the simplest representation of the constant. "simplest" meaning if it is representable as an -integer or fat pointer, it will directly yield the value (via `Value::ByVal` or -`Value::ByValPair`), instead of referring to the [`miri`](./miri.html) virtual -memory allocation (via `Value::ByRef`). This means that the `const_eval` +integer or fat pointer, it will directly yield the value (via `ConstValue::Scalar` or +`ConstValue::ScalarPair`), instead of referring to the [`miri`](./miri.html) virtual +memory allocation (via `ConstValue::ByRef`). This means that the `const_eval` function cannot be used to create miri-pointers to the evaluated constant or static. If you need that, you need to directly work with the functions in [src/librustc_mir/const_eval.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/const_eval/index.html). diff --git a/src/miri.md b/src/miri.md index a3c7b3ff4..cbed690eb 100644 --- a/src/miri.md +++ b/src/miri.md @@ -55,18 +55,18 @@ Before the evaluation, a virtual memory location (in this case essentially a `vec![u8; 4]` or `vec![u8; 8]`) is created for storing the evaluation result. At the start of the evaluation, `_0` and `_1` are -`Value::ByVal(PrimVal::Undef)`. When the initialization of `_1` is invoked, the +`ConstValue::Scalar(Scalar::Undef)`. When the initialization of `_1` is invoked, the value of the `FOO` constant is required, and triggers another call to `tcx.const_eval`, which will not be shown here. If the evaluation of FOO is successful, 42 will be subtracted by its value `4096` and the result stored in -`_1` as `Value::ByValPair(PrimVal::Bytes(4054), PrimVal::Bytes(0))`. The first +`_1` as `ConstValue::ScalarPair(Scalar::Bytes(4054), Scalar::Bytes(0))`. The first part of the pair is the computed value, the second part is a bool that's true if an overflow happened. The next statement asserts that said boolean is `0`. In case the assertion fails, its error message is used for reporting a compile-time error. -Since it does not fail, `Value::ByVal(PrimVal::Bytes(4054))` is stored in the +Since it does not fail, `ConstValue::Scalar(Scalar::Bytes(4054))` is stored in the virtual memory was allocated before the evaluation. `_0` always refers to that location directly. @@ -75,7 +75,7 @@ After the evaluation is done, the virtual memory allocation is interned into the miri, but just extract the value from the interned allocation. The `tcx.const_eval` function has one additional feature: it will not return a -`ByRef(interned_allocation_id)`, but a `ByVal(computed_value)` if possible. This +`ByRef(interned_allocation_id)`, but a `Scalar(computed_value)` if possible. This makes using the result much more convenient, as no further queries need to be executed in order to get at something as simple as a `usize`. @@ -83,15 +83,15 @@ executed in order to get at something as simple as a `usize`. Miri's core datastructures can be found in [librustc/mir/interpret](https://github.com/rust-lang/rust/blob/master/src/librustc/mir/interpret). -This is mainly the error enum and the `Value` and `PrimVal` types. A `Value` can -be either `ByVal` (a single `PrimVal`), `ByValPair` (two `PrimVal`s, usually fat +This is mainly the error enum and the `ConstValue` and `Scalar` types. A `ConstValue` can +be either `Scalar` (a single `Scalar`), `ScalarPair` (two `Scalar`s, usually fat pointers or two element tuples) or `ByRef`, which is used for anything else and refers to a virtual allocation. These allocations can be accessed via the methods on `tcx.interpret_interner`. If you are expecting a numeric result, you can use `unwrap_u64` (panics on anything that can't be representad as a `u64`) or `to_raw_bits` which results -in an `Option` yielding the `ByVal` if possible. +in an `Option` yielding the `Scalar` if possible. ## Allocations @@ -113,7 +113,7 @@ to a pointer to `b`. Although the main entry point to constant evaluation is the `tcx.const_eval` query, there are additional functions in [librustc_mir/const_eval.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/const_eval/index.html) -that allow accessing the fields of a `Value` (`ByRef` or otherwise). You should +that allow accessing the fields of a `ConstValue` (`ByRef` or otherwise). You should never have to access an `Allocation` directly except for translating it to the compilation target (at the moment just LLVM). From 4f0d4744b82a14a443ed793c48cab1d4b0b5fa5f Mon Sep 17 00:00:00 2001 From: rchaser53 Date: Tue, 22 Jan 2019 00:24:36 +0900 Subject: [PATCH 485/648] fix unwrap_u64 and to_raw_bits --- src/miri.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/miri.md b/src/miri.md index cbed690eb..d361103f2 100644 --- a/src/miri.md +++ b/src/miri.md @@ -89,8 +89,8 @@ pointers or two element tuples) or `ByRef`, which is used for anything else and refers to a virtual allocation. These allocations can be accessed via the methods on `tcx.interpret_interner`. -If you are expecting a numeric result, you can use `unwrap_u64` (panics on -anything that can't be representad as a `u64`) or `to_raw_bits` which results +If you are expecting a numeric result, you can use `unwrap_usize` (panics on +anything that can't be representad as a `u64`) or `assert_usize` which results in an `Option` yielding the `Scalar` if possible. ## Allocations From 3a1620143f2b87ca67096431267bca0915705a54 Mon Sep 17 00:00:00 2001 From: Oliver Scherer Date: Mon, 21 Jan 2019 16:45:09 +0100 Subject: [PATCH 486/648] Fix TokenStream link --- src/appendix/code-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index 546e9f9b7..e1bde6863 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -23,7 +23,7 @@ Item | Kind | Short description | Chapter | `SourceMap` | struct | Maps AST nodes to their source code. It is composed of `SourceFile`s. Was previously called CodeMap | [The parser] | [src/libsyntax/source_map.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html) `Span` | struct | A location in the user's source code, used for error reporting primarily | [Emitting Diagnostics] | [src/libsyntax_pos/span_encoding.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax_pos/struct.Span.html) `StringReader` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser] | [src/libsyntax/parse/lexer/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/lexer/struct.StringReader.html) -`syntax::token_stream::TokenStream` | struct | An abstract sequence of tokens, organized into `TokenTree`s | [The parser], [Macro expansion] | [src/libsyntax/tokenstream.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/tokenstream/enum.TokenStream.html) +`syntax::token_stream::TokenStream` | struct | An abstract sequence of tokens, organized into `TokenTree`s | [The parser], [Macro expansion] | [src/libsyntax/tokenstream.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/tokenstream/struct.TokenStream.html) `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules] | [src/librustc/ty/trait_def.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/trait_def/struct.TraitDef.html) `TraitRef` | struct | The combination of a trait and its input types (e.g. `P0: Trait`) | [Trait Solving: Goals and Clauses], [Trait Solving: Lowering impls] | [src/librustc/ty/sty.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TraitRef.html) `Ty<'tcx>` | struct | This is the internal representation of a type used for type checking | [Type checking] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/type.Ty.html) From 54bf20498bbdc7f5b97e076ceb386ff703b0d59c Mon Sep 17 00:00:00 2001 From: Michael Woerister Date: Tue, 22 Jan 2019 15:26:01 +0100 Subject: [PATCH 487/648] Add "The Query Evaluation Model in Detail" Chapter. --- src/SUMMARY.md | 1 + .../query-evaluation-model-in-detail.md | 236 ++++++++++++++++++ src/query.md | 60 +---- 3 files changed, 243 insertions(+), 54 deletions(-) create mode 100644 src/queries/query-evaluation-model-in-detail.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 9bdf89278..4d6136702 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -20,6 +20,7 @@ - [The Rustc Driver](./rustc-driver.md) - [Rustdoc](./rustdoc.md) - [Queries: demand-driven compilation](./query.md) + - [The Query Evaluation Model in Detail](./query-evaluation-model-in-detail.md) - [Incremental compilation](./incremental-compilation.md) - [Debugging and Testing](./incrcomp-debugging.md) - [The parser](./the-parser.md) diff --git a/src/queries/query-evaluation-model-in-detail.md b/src/queries/query-evaluation-model-in-detail.md new file mode 100644 index 000000000..d2dc10479 --- /dev/null +++ b/src/queries/query-evaluation-model-in-detail.md @@ -0,0 +1,236 @@ + + +# The Query Evaluation Model in Detail + +This chapter provides a deeper dive into the abstract model queries are built on. +It does not go into implementation details but tries to explain +the underlying logic. The examples here, therefore, have been stripped down and +simplified and don't directly reflect the compilers internal APIs. + +## What is a query? + +Abstractly we view the compiler's knowledge about a given crate as a "database" +and queries are the way of asking the compiler questions about it, i.e. +we "query" the compiler's "database" for facts. + +However, there's something special to this compiler database: It starts out empty +and is filled on-demand when queries are executed. Consequently, a query must +know how to compute its result if the database does not contain it yet. For +doing so, it can access other queries and certain input values that the database +is pre-filled with on creation. + +A query thus consists of the following things: + + - A name that identifies the query + - A "key" that specifies what we want to look up + - A result type that specifies what kind of result it yields + - A "provider" which is a function that specifies how the result is to be + computed if it isn't already present in the database. + +As an example, the name of the `type_of` query is `type_of`, its query key is a +`DefId` identifying the item we want to know the type of, the result type is +`Ty<'tcx>`, and the provider is a function that, given the query key and access +to the rest of the database, can compute the type of the item identified by the +key. + +So in some sense a query is just a function that maps the query key to the +corresponding result. However, we have to apply some restrictions in order for +this to be sound: + + - The key and result must be immutable values. + - The provider function must be a pure function, that is, for the same key it + must always yield the same result. + - The only parameters a provider function takes are the key and a reference to + the "query context" (which provides access to rest of the "database"). + +The database is built up lazily by invoking queries. The query providers will +invoke other queries, for which the result is either already cached or computed +by calling another query provider. These query provider invocations +conceptually form a directed acyclic graph (DAG) at the leaves of which are +input values that are already known when the query context is created. + + + +## Caching/Memoization + +Results of query invocations are "memoized" which means that the query context +will cache the result in an internal table and, when the query is invoked with +the same query key again, will return the result from the cache instead of +running the provider again. + +This caching is crucial for making the query engine efficient. Without +memoization the system would still be sound (that is, it would yield the same +results) but the same computations would be done over and over again. + +Memoization is one of the main reasons why query providers have to be pure +functions. If calling a provider function could yield different results for +each invocation (because it accesses some global mutable state) then we could +not memoize the result. + + + +## Input data + +When the query context is created, it is still empty: No queries have been +executed, no results are cached. But the context already provides access to +"input" data, i.e. pieces of immutable data that where computed before the +context was created and that queries can access to do their computations. +Currently this input data consists mainly of the HIR map and the command-line +options the compiler was invoked with. In the future, inputs will just consist +of command-line options and a list of source files -- the HIR map will itself +be provided by a query which processes these source files. + +Without inputs, queries would live in a void without anything to compute their +result from (remember, query providers only have access to other queries and +the context but not any other outside state or information). + +For a query provider, input data and results of other queries look exactly the +same: It just tells the context "give me the value of X". Because input data +is immutable, the provider can rely on it being the same across +different query invocations, just as is the case for query results. + + + +## An example execution trace of some queries + +How does this DAG of query invocations come into existence? At some point +the compiler driver will create the, as yet empty, query context. It will then, +from outside of the query system, invoke the queries it needs to perform its +task. This looks something like the following: + +```rust,ignore +fn compile_crate() {} + let cli_options = ...; + let hir_map = ...; + + // Create the query context `tcx` + let tcx = TyCtxt::new(cli_options, hir_map); + + // Do type checking by invoking the type check query + tcx.type_check_crate(); +} +``` + +The `type_check_crate` query provider would look something like the following: + +```rust,ignore +fn type_check_crate_provider(tcx, _key: ()) { + let list_of_items = tcx.hir_map.list_of_items(); + + for item_def_id in list_of_hir_items { + tcx.type_check_item(item_def_id); + } +} +``` + +We see that the `type_check_crate` query accesses input data (`tcx.hir_map`) +and invokes other queries (`type_check_item`). The `type_check_item` +invocations will themselves access input data and/or invoke other queries, +so that in the end the DAG of query invocations will be built up backwards +from the node that was initially executed: + +``` + (1) + hir_map <--------------------------------------------------- type_check_crate() + ^ | + | (4) (3) (2) | + +-- Hir(foo) <--- type_of(foo) <--- type_check_item(foo) <-------+ + | | | + | +-----------------+ | + | | | + | (6) v (5) (7) | + +-- Hir(bar) <--- type_of(bar) <--- type_check_item(bar) <-------+ + +// (x) denotes invocation order +``` + +We also see that often a query result can be read from the cache: +`type_of(bar)` was computed for `type_check_item(foo)` so when +`type_check_item(bar)` needs it, it is already in the cache. + +Query results stay cached in the query context as long as the context lives. +So if the compiler driver invoked another query later on, the above graph +would still exist and already executed queries would not have to be re-done. + + + +## Cycles + +Earlier we stated that query invocations form a DAG. However, it would be easy +form a cyclic graph by, for example, having a query provider like the following: + +```rust,ignore +fn cyclic_query_provider(tcx, key) -> u32 { + // Invoke the same query with the same key again + tcx.cyclic_query(key) +} +``` + +Since query providers are regular functions, this would behave much as expected: +Evaluation would get stuck in an infinite recursion. A query like this would not +be very useful either. However, sometimes certain kinds of invalid user input +can result in queries being called in a cyclic way. The query engine includes +a check for cyclic invocations and, because cycles are an irrecoverable error, +will abort execution with a "cycle error" messages that tries to be human +readable. + +At some point the compiler had a notion of "cycle recovery", that is, one could +"try" to execute a query and if it ended up causing a cycle, proceed in some +other fashion. However, this was later removed because it is not entirely +clear what the theoretical consequences of this are, especially regarding +incremental compilation. + + +## "Steal" Queries + +Some queries have their result wrapped in a `Steal` struct. These queries +behave exactly the same as regular with one exception: Their result is expected +to be "stolen" out of the cache at some point, meaning some other part of the +program is taking ownership of it and the result cannot be accessed anymore. + +This stealing mechanism exists purely as a performance optimization because some +result values are too costly to clone (e.g. the MIR of a function). It seems +like result stealing would violate the condition that query results must be +immutable (after all we are moving the result value out of the cache) but it is +OK as long as the mutation is not observable. This is achieved by two things: + +- Before a result is stolen, we make sure to eagerly run all queries that + might ever need to read that result. This has to be done manually by calling + those queries. +- Whenever a query tries to access a stolen result, we make the compiler ICE so + that such a condition cannot go unnoticed. + +This is not an ideal setup because of the manual intervention needed, so it +should be used sparingly and only when it is well known which queries might +access a given result. In practice, however, stealing has not turned out to be +much of a maintainance burden. + +To summarize: "Steal queries" break some of the rules in a controlled way. +There are checks in place that make sure that nothing can go silently wrong. + + +## Parallel Query Execution + +The query model has some properties that make it actually feasible to evaluate +multiple queries in parallel without too much of an effort: + +- All data a query provider can access is accessed via the query context, so + the query context can take care of synchronizing access. +- Query results are required to be immutable so they can safely be used by + different threads concurrently. + +The nightly compiler already implements parallel query evaluation as follows: + +When a query `foo` is evaluated, the cache table for `foo` is locked. + +- If there already is a result, we can clone it,release the lock and + we are done. +- If there is no cache entry and no other active query invocation computing the + same result, we mark the key as being "in progress", release the lock and + start evaluating. +- If there *is* another query invocation for the same key in progress, we + release the lock, and just block the thread until the other invocation has + computed the result we are waiting for. This cannot deadlock because, as + mentioned before, query invocations form a DAG. Some thread will always make + progress. + diff --git a/src/query.md b/src/query.md index a0c0d6282..ee7d60e4b 100644 --- a/src/query.md +++ b/src/query.md @@ -35,6 +35,12 @@ will in turn demand information about that crate, starting from the However, that vision is not fully realized. Still, big chunks of the compiler (for example, generating MIR) work exactly like this. +### The Query Evaluation Model in Detail + +The [Query Evaluation Model in Detail](query-evaluation-model-in-detail.html) +chapter gives a more in-depth description of what queries are and how they work. +If you intend to write a query of your own, this is a good read. + ### Invoking queries To invoke a query is simple. The tcx ("type context") offers a method @@ -45,60 +51,6 @@ query, you would just do this: let ty = tcx.type_of(some_def_id); ``` -### Cycles between queries - -A cycle is when a query becomes stuck in a loop e.g. query A generates query B -which generates query A again. - -Currently, cycles during query execution should always result in a -compilation error. Typically, they arise because of illegal programs -that contain cyclic references they shouldn't (though sometimes they -arise because of compiler bugs, in which case we need to factor our -queries in a more fine-grained fashion to avoid them). - -However, it is nonetheless often useful to *recover* from a cycle -(after reporting an error, say) and try to soldier on, so as to give a -better user experience. In order to recover from a cycle, you don't -get to use the nice method-call-style syntax. Instead, you invoke -using the `try_get` method, which looks roughly like this: - -```rust,ignore -use ty::queries; -... -match queries::type_of::try_get(tcx, DUMMY_SP, self.did) { - Ok(result) => { - // no cycle occurred! You can use `result` - } - Err(err) => { - // A cycle occurred! The error value `err` is a `DiagnosticBuilder`, - // meaning essentially an "in-progress", not-yet-reported error message. - // See below for more details on what to do here. - } -} -``` - -So, if you get back an `Err` from `try_get`, then a cycle *did* occur. This -means that you must ensure that a compiler error message is reported. You can -do that in two ways: - -The simplest is to invoke `err.emit()`. This will emit the cycle error to the -user. - -However, often cycles happen because of an illegal program, and you -know at that point that an error either already has been reported or -will be reported due to this cycle by some other bit of code. In that -case, you can invoke `err.cancel()` to not emit any error. It is -traditional to then invoke: - -```rust,ignore -tcx.sess.delay_span_bug(some_span, "some message") -``` - -`delay_span_bug()` is a helper that says: we expect a compilation -error to have happened or to happen in the future; so, if compilation -ultimately succeeds, make an ICE with the message `"some -message"`. This is basically just a precaution in case you are wrong. - ### How the compiler executes a query So you may be wondering what happens when you invoke a query From 952f9366b2ab1158953dd3e5f060da3f3eb93a7b Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 23 Jan 2019 12:41:18 -0600 Subject: [PATCH 488/648] update text about copyright headers --- src/conventions.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/conventions.md b/src/conventions.md index d392ebd57..385658dac 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -20,11 +20,12 @@ in isolation with `./x.py test src/tools/tidy`. ### Copyright notice -Some existing files begin with a copyright and license notice. Please omit this -notice for new files licensed under the standard terms (dual MIT/Apache-2.0). -For existing files, the year at the top is not meaningful: copyright -protections are in fact automatic from the moment of authorship. We do not -typically edit the years on existing files. +In the past, files begin with a copyright and license notice. Please **omit** +this notice for new files licensed under the standard terms (dual +MIT/Apache-2.0). + +All of the copyright notices should be gone by now, but if you come across one +in the rust-lang/rust repo, feel free to open a PR to remove it. ## Line length From bb3392e7d70d72a77ed6c324933072a1d34434a5 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 23 Jan 2019 12:56:40 -0600 Subject: [PATCH 489/648] update conventions --- src/SUMMARY.md | 1 + src/conventions.md | 17 ++++++++++++++++- src/crates-io.md | 23 +++++++++++++++++++++++ 3 files changed, 40 insertions(+), 1 deletion(-) create mode 100644 src/crates-io.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 4df03dc7a..f472bfaab 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -19,6 +19,7 @@ - [Profiling the compiler](./profiling.md) - [with the linux perf tool](./profiling/with_perf.md) - [Coding conventions](./conventions.md) +- [crates.io Dependencies](./crates-io.md) --- diff --git a/src/conventions.md b/src/conventions.md index 385658dac..d9f462a05 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -11,7 +11,7 @@ rustc is slowly moving towards the [Rust standard coding style][fmt]; at the moment, however, it follows a rather more *chaotic* style. We do have some mandatory formatting conventions, which are automatically enforced by a script we affectionately call the "tidy" script. The -tidy script runs automatically when you do `./x.py test` and can be run +tidy script runs automatically when you do `./x.py test` and can be run in isolation with `./x.py test src/tools/tidy`. [fmt]: https://github.com/rust-lang-nursery/fmt-rfcs @@ -133,3 +133,18 @@ require that every intermediate commit successfully builds – we only expect to be able to bisect at a PR level. However, if you *can* make individual commits build, that is always helpful. +# Naming conventions + +Apart from normal Rust style/naming conventions, there are also some specific +to the compiler. + +- `cx` tends to be short for "context" and is often used as a suffix. For + example, `tcx` is a common name for the [Typing Context][tcx]. + +- [`'tcx` and `'gcx`][tcx] are used as the lifetime names for the Typing + Context. + +- Because `crate` is a keyword, if you need a variable to represent something + crate-related, often the spelling is changed to `krate`. + +[tcx]: ./ty.md diff --git a/src/crates-io.md b/src/crates-io.md new file mode 100644 index 000000000..21dd00afe --- /dev/null +++ b/src/crates-io.md @@ -0,0 +1,23 @@ +# crates.io Dependencies + +The rust compiler supports building with some dependencies from `crates.io`. +For example, `log` and `env_logger` come from `crates.io`. + +In general, you should avoid adding dependencies to the compiler for several +reasons: + +- The dependency may not be high quality or well-maintained, whereas we want + the compiler to be high-quality. +- The dependency may not be using a compatible license. +- The dependency may have transitive dependencies that have one of the above + problems. + +TODO: what is the vetting process? + +## Whitelist + +The `tidy` tool has a [whitelist] of crates that are allowed. To add a +dependency that is not already in the compiler, you will need to add it to this +whitelist. + +[whitelist]: https://github.com/rust-lang/rust/blob/659994627234ce7d95a1a52ad8756ce661059adf/src/tools/tidy/src/deps.rs#L56 From 658ce409c4994ade842b193d3e498134907812e7 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 23 Jan 2019 21:55:52 -0600 Subject: [PATCH 490/648] fix #182 --- src/appendix/code-index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index e1bde6863..c0a1d03a9 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -15,6 +15,7 @@ Item | Kind | Short description | Chapter | `DocContext` | struct | A state container used by rustdoc when crawling through a crate to gather its documentation | [Rustdoc] | [src/librustdoc/core.rs](https://github.com/rust-lang/rust/blob/master/src/librustdoc/core.rs) `HirId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.HirId.html) `NodeId` | struct | One of four types of HIR node identifiers. Being phased out | [Identifiers in the HIR] | [src/libsyntax/ast.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.NodeId.html) +`P` | struct | An owned immutable smart pointer. By contrast, `&T` is not owned, and `Box` is not immutable. | None | [src/syntax/ptr.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ptr/struct.P.html) `ParamEnv` | struct | Information about generic parameters or `Self`, useful for working with associated or generic items | [Parameter Environment] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.ParamEnv.html) `ParseSess` | struct | This struct contains information about a parsing session | [The parser] | [src/libsyntax/parse/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/struct.ParseSess.html) `Rib` | struct | Represents a single scope of names | [Name resolution] | [src/librustc_resolve/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/struct.Rib.html) @@ -27,7 +28,7 @@ Item | Kind | Short description | Chapter | `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules] | [src/librustc/ty/trait_def.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/trait_def/struct.TraitDef.html) `TraitRef` | struct | The combination of a trait and its input types (e.g. `P0: Trait`) | [Trait Solving: Goals and Clauses], [Trait Solving: Lowering impls] | [src/librustc/ty/sty.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TraitRef.html) `Ty<'tcx>` | struct | This is the internal representation of a type used for type checking | [Type checking] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/type.Ty.html) -`TyCtxt<'cx, 'tcx, 'tcx>` | type | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries | [The `ty` modules] | [src/librustc/ty/context.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html) +`TyCtxt<'cx, 'tcx, 'tcx>` | struct | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries | [The `ty` modules] | [src/librustc/ty/context.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html) [The HIR]: ../hir.html [Identifiers in the HIR]: ../hir.html#hir-id From b83f584379a10064118bbd9fb131ea692b1c2b0d Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 23 Jan 2019 13:15:12 -0600 Subject: [PATCH 491/648] add link to sunjay talk --- src/traits/specialization.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/traits/specialization.md b/src/traits/specialization.md index 671e5e016..13edceca1 100644 --- a/src/traits/specialization.md +++ b/src/traits/specialization.md @@ -40,3 +40,9 @@ will use at trans time. Thus, we take special care to avoid projecting associated types unless either (1) the associated type does not use `default` and thus cannot be overridden or (2) all input types are known concretely. + +## Additional Resources + +[This talk][talk] by @sunjay may be useful. + +[talk]: https://www.youtube.com/watch?v=rZqS4bLPL24 From e7d7e38e95f964f85aa37c0ee9d337875d093e63 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 23 Jan 2019 19:31:49 -0600 Subject: [PATCH 492/648] add comments from sunjay --- src/traits/specialization.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/traits/specialization.md b/src/traits/specialization.md index 13edceca1..7a30314b4 100644 --- a/src/traits/specialization.md +++ b/src/traits/specialization.md @@ -43,6 +43,9 @@ known concretely. ## Additional Resources -[This talk][talk] by @sunjay may be useful. +[This talk][talk] by @sunjay may be useful. Keep in mind that the talk only +gives a broad overview of the problem and the solution (it was presented about +halfway through @sunjay's work). Also, it was given in June 2018, and some +things may have changed by the time you watch it. [talk]: https://www.youtube.com/watch?v=rZqS4bLPL24 From 9600742738cf5b40f549be00be3ae099c9a5f5c1 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 22 Jan 2019 15:02:36 -0600 Subject: [PATCH 493/648] fix updating-docs link, add regression test --- src/SUMMARY.md | 1 + src/important-links.md | 12 ++++++++++++ src/stabilization_guide.md | 2 ++ 3 files changed, 15 insertions(+) create mode 100644 src/important-links.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 4df03dc7a..2d8c17024 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -82,3 +82,4 @@ [Appendix B: Background material](./appendix/background.md) [Appendix C: Glossary](./appendix/glossary.md) [Appendix D: Code Index](./appendix/code-index.md) +[](./important-links.md) diff --git a/src/important-links.md b/src/important-links.md new file mode 100644 index 000000000..2dc759bf6 --- /dev/null +++ b/src/important-links.md @@ -0,0 +1,12 @@ +# Ignore me + +This file is a collection of links that are not link-checked by anyone else, +but we want them to work. For example, the stabilization guide link is often +posted to tracking issues on GitHub, which might break later if the guide is +changed. + +[Skip this file. Or read it; I'm book, not a cop.](https://imgur.com/gallery/mSHi8) + +[1](https://forge.rust-lang.org/stabilization-guide.html) +[2](https://rust-lang.github.io/rustc-guide/stabilization_guide.html#documentation-prs) +[3](https://forge.rust-lang.org/stabilization-guide.html#updating-documentation) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index ab5340c95..56da5cdaf 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -11,6 +11,8 @@ following steps. ## Documentation PRs + + If any documentation for this feature exists, it should be in the [`Unstable Book`], located at [`src/doc/unstable-book`]. If it exists, the page for the feature gate should be removed. From 1a98854e47e1ca490b453535db9e4a39754a9db0 Mon Sep 17 00:00:00 2001 From: Ralf Jung Date: Tue, 22 Jan 2019 16:00:24 -0600 Subject: [PATCH 494/648] Typo Co-Authored-By: mark-i-m --- src/important-links.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/important-links.md b/src/important-links.md index 2dc759bf6..480be1bdb 100644 --- a/src/important-links.md +++ b/src/important-links.md @@ -5,7 +5,7 @@ but we want them to work. For example, the stabilization guide link is often posted to tracking issues on GitHub, which might break later if the guide is changed. -[Skip this file. Or read it; I'm book, not a cop.](https://imgur.com/gallery/mSHi8) +[Skip this file. Or read it; I'm a book, not a cop.](https://imgur.com/gallery/mSHi8) [1](https://forge.rust-lang.org/stabilization-guide.html) [2](https://rust-lang.github.io/rustc-guide/stabilization_guide.html#documentation-prs) From c7cdf9fe3ecdad3ba5465ea79a77d507d6c96fe0 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 22 Jan 2019 16:02:53 -0600 Subject: [PATCH 495/648] better grouping, more links --- src/important-links.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/important-links.md b/src/important-links.md index 480be1bdb..367532519 100644 --- a/src/important-links.md +++ b/src/important-links.md @@ -7,6 +7,10 @@ changed. [Skip this file. Or read it; I'm a book, not a cop.](https://imgur.com/gallery/mSHi8) -[1](https://forge.rust-lang.org/stabilization-guide.html) -[2](https://rust-lang.github.io/rustc-guide/stabilization_guide.html#documentation-prs) -[3](https://forge.rust-lang.org/stabilization-guide.html#updating-documentation) +[link](https://forge.rust-lang.org/stabilization-guide.html) +[link](https://forge.rust-lang.org/stabilization-guide.html#updating-documentation) +[link](https://forge.rust-lang.org/stabilization-guide.html#documentation-prs) + +[link](https://rust-lang.github.io/rustc-guide/stabilization_guide.html) +[link](https://rust-lang.github.io/rustc-guide/stabilization_guide.html#updating-documentation) +[link](https://rust-lang.github.io/rustc-guide/stabilization_guide.html#documentation-prs) From 3dadf436ad960c34445c6617fa919fe5a48a3f43 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 29 Jan 2019 10:55:57 -0600 Subject: [PATCH 496/648] add blurb about downloading artifacts; fix #77x --- src/compiler-debugging.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index 19c939d6d..31f221057 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -358,4 +358,23 @@ create a minimal working example with Godbolt. Go to ## Narrowing (Bisecting) Regressions -The [cargo-bisect-rustc](https://github.com/rust-lang-nursery/cargo-bisect-rustc) tool can be used as a quick and easy way to find exactly which PR caused a change in `rustc` behavior. It automatically downloads `rustc` PR artifacts and tests them against a project you provide until it finds the regression. You can then look at the PR to get more context on *why* it was changed. See [this tutorial](https://github.com/rust-lang-nursery/cargo-bisect-rustc/blob/master/TUTORIAL.md) on how to use it. +The [cargo-bisect-rustc][bisect] tool can be used as a quick and easy way to +find exactly which PR caused a change in `rustc` behavior. It automatically +downloads `rustc` PR artifacts and tests them against a project you provide +until it finds the regression. You can then look at the PR to get more context +on *why* it was changed. See [this tutorial][bisect-tutorial] on how to use +it. + +[bisect]: https://github.com/rust-lang-nursery/cargo-bisect-rustc +[bisect-tutorial]: https://github.com/rust-lang-nursery/cargo-bisect-rustc/blob/master/TUTORIAL.md + +## Downloading Artifacts from Rust's CI + +The [rustup-toolchain-install-master][rtim] tool by kennytm can be used to +download the artifacts produced by Rust's CI for a specific SHA1 -- this +basically corresponds to the successful landing of some PR -- and then sets +them up for your local use. This also works for artifacts produced by `@bors +try`. This is helpful when you want to examine the resulting build of a PR +without doing the build yourself. + +[rtim]: https://github.com/kennytm/rustup-toolchain-install-master From 808a9a17565555682731d702555e43875a1c18e3 Mon Sep 17 00:00:00 2001 From: Michael Woerister Date: Fri, 25 Jan 2019 16:50:22 +0100 Subject: [PATCH 497/648] Add a more detailed description of how incremental compilation works. --- src/SUMMARY.md | 5 +- src/appendix/glossary.md | 2 +- .../incremental-compilation-in-detail.md | 354 ++++++++++++++++++ src/{ => queries}/incremental-compilation.md | 0 .../query-evaluation-model-in-detail.md | 27 +- src/query.md | 5 +- src/variance.md | 2 +- 7 files changed, 376 insertions(+), 19 deletions(-) create mode 100644 src/queries/incremental-compilation-in-detail.md rename src/{ => queries}/incremental-compilation.md (100%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 4d6136702..cd3c9d331 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -20,8 +20,9 @@ - [The Rustc Driver](./rustc-driver.md) - [Rustdoc](./rustdoc.md) - [Queries: demand-driven compilation](./query.md) - - [The Query Evaluation Model in Detail](./query-evaluation-model-in-detail.md) - - [Incremental compilation](./incremental-compilation.md) + - [The Query Evaluation Model in Detail](./queries/query-evaluation-model-in-detail.md) + - [Incremental compilation](./queries/incremental-compilation.md) + - [Incremental compilation In Detail](./queries/incremental-compilation-in-detail.md) - [Debugging and Testing](./incrcomp-debugging.md) - [The parser](./the-parser.md) - [`#[test]` Implementation](./test-implementation.md) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 0e697aeea..decfb4426 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -15,7 +15,7 @@ completeness | completeness is a technical term in type theory. Comp control-flow graph | a representation of the control-flow of a program; see [the background chapter for more](./background.html#cfg) CTFE | Compile-Time Function Evaluation. This is the ability of the compiler to evaluate `const fn`s at compile time. This is part of the compiler's constant evaluation system. ([see more](../const-eval.html)) cx | we tend to use "cx" as an abbreviation for context. See also `tcx`, `infcx`, etc. -DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](../incremental-compilation.html)) +DAG | a directed acyclic graph is used during compilation to keep track of dependencies between queries. ([see more](../queries/incremental-compilation.html)) data-flow analysis | a static analysis that figures out what properties are true at each point in the control-flow of a program; see [the background chapter for more](./background.html#dataflow) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. Double pointer | a pointer with additional metadata. See "fat pointer" for more. diff --git a/src/queries/incremental-compilation-in-detail.md b/src/queries/incremental-compilation-in-detail.md new file mode 100644 index 000000000..fbe226e90 --- /dev/null +++ b/src/queries/incremental-compilation-in-detail.md @@ -0,0 +1,354 @@ +# Incremental Compilation In Detail + +The incremental compilation scheme is, in essence, a surprisingly +simple extension to the overall query system. It relies on the fact that: + + 1. queries are pure functions -- given the same inputs, a query will always + yield the same result, and + 2. the query model structures compilation in an acyclic graph that makes + dependencies between individual computations explicit. + +This chapter will explain how we can use these properties for making things +incremental and then goes on to discuss version implementation issues. + +# A Basic Algorithm For Incremental Query Evaluation + +As explained in the [query evaluation model primer][query-model], query +invocations form a directed-acyclic graph. Here's the example from the +previous chapter again: + +```ignore + list_of_all_hir_items <----------------------------- type_check_crate() + | + | + Hir(foo) <--- type_of(foo) <--- type_check_item(foo) <-------+ + | | + +-----------------+ | + | | + v | + Hir(bar) <--- type_of(bar) <--- type_check_item(bar) <-------+ +``` + +Since every access from one query to another has to go through the query +context, we can record these accesses and thus actually build this dependency +graph in memory. With dependency tracking enabled, when compilation is done, +we know which queries were invoked (the nodes of the graph) and for each +invocation, which other queries or input has gone into computing the query's +result (the edges of the graph). + +Now suppose, we change the source code of our program so that +HIR of `bar` looks different than before. Our goal is to only recompute +those queries that are actually affected by the change while just re-using +the cached results of all the other queries. Given the dependency graph we can +do exactly that. For a given query invocation, the graph tells us exactly +what data has gone into computing its results, we just have to follow the +edges until we reach something that has changed. If we don't encounter +anything that has changed, we know that the query still would evaluate to +the same result we already have in our cache. + +Taking the `type_of(foo)` invocation from above as example, we can check +whether the cached result is still valid by following the edges to its +inputs. The only edge leads to `Hir(foo)`, an input that has not been affected +by the change. So we know that the cached result for `type_of(foo)` is still +valid. + +The story is a bit different for `type_check_item(foo)`: We again walk the +edges and already know that `type_of(foo)` is fine. Then we get to +`type_of(bar)` which we have not checked yet, so we walk the edges of +`type_of(bar)` and encounter `Hir(bar)` which *has* changed. Consequently +the result of `type_of(bar)` might yield a different same result than what we +have in the cache and, transitively, the result of `type_check_item(foo)` +might have changed too. We thus re-run `type_check_item(foo)`, which in +turn will re-run `type_of(bar)`, which will yield an up-to-date result +because it reads the up-to-date version of `Hir(bar)`. + + +# The Problem With The Basic Algorithm: False Positives + +If you read the previous paragraph carefully, you'll notice that it says that +`type_of(bar)` *might* have changed because one of its inputs has changed. +There's also the possibility that it might still yield exactly the same +result *even though* its input has changed. Consider an example with a +simple query that just computes the sign of an integer: + +```ignore + IntValue(x) <---- sign_of(x) <--- some_other_query(x) +``` + +Let's say that `IntValue(x)` starts out as `1000` and then is set to `2000`. +Even though `IntValue(x)` is different in the two cases, `sign_of(x)` yields +the result `+` in both cases. + +If we follow the basic algorithm, however, `some_other_query(x)` would have to +(unnecessarily) be re-evaluated because it transitively depends on a changed +input. Change detection yields a "false positive" in this case because it has +to conservatively assume that `some_other_query(x)` might be affected by that +changed input. + +Unfortunately it turns out that the actual queries in the compiler are full +of examples like this and small changes to the input often potentially affect +very large parts of the output binaries. As a consequence, we had to make the +change detection system smarter and more accurate. + +# Improving Accuracy: The red-green Algorithm + +The "false positives" problem can be solved by interleaving change detection +and query re-evaluation. Instead of walking the graph all the way to the +inputs when trying to find out if some cached result is still valid, we can +check if a result has *actually* changed after we were forced to re-evaluate +it. + +We call this algorithm, for better or worse, the red-green algorithm because nodes +in the dependency graph are assigned the color green if we were able to prove +that its cached result is still valid and the color red if the result has +turned out to be different after re-evaluating it. + +The meat of red-green change tracking is implemented in the try-mark-green +algorithm, that, you've guessed it, tries to mark a given node as green: + +```rust,ignore +fn try_mark_green(tcx, current_node) -> bool { + + // Fetch the inputs to `current_node`, i.e. get the nodes that the direct + // edges from `node` lead to. + let dependencies = tcx.dep_graph.get_dependencies_of(current_node); + + // Now check all the inputs for changes + for dependency in dependencies { + + match tcx.dep_graph.get_node_color(dependency) { + Green => { + // This input has already been checked before and it has not + // changed; so we can go on to check the next one + } + Red => { + // We found an input that has changed. We cannot mark + // `current_node` as green without re-running the + // corresponding query. + return false + } + Unknown => { + // This is the first time we are look at this node. Let's try + // to mark it green by calling try_mark_green() recursively. + if try_mark_green(tcx, dependency) { + // We successfully marked the input as green, on to the + // next. + } else { + // We could *not* mark the input as green. This means we + // don't know if its value has changed. In order to find + // out, we re-run the corresponding query now! + tcx.run_query_for(dependency); + + // Fetch and check the node color again. Running the query + // has forced it to either red (if it yielded a different + // result than we have in the cache) or green (if it + // yielded the same result). + match tcx.dep_graph.get_node_color(dependency) { + Red => { + // The input turned out to be red, so we cannot + // mark `current_node` as green. + return false + } + Green => { + // Re-running the query paid off! The result is the + // same as before, so this particular input does + // not invalidate `current_node`. + } + Unknown => { + // There is no way a node has no color after + // re-running the query. + panic!("unreachable") + } + } + } + } + } + } + + // If we have gotten through the entire loop, it means that all inputs + // have turned out to be green. If all inputs are unchanged, it means + // that the query result corresponding to `current_node` cannot have + // changed either. + tcx.dep_graph.mark_green(current_node); + + true +} + +// Note: The actual implementation can be found in +// src/librustc/dep_graph/graph.rs +``` + +By using red-green marking we can avoid the devastating cumulative effect of +having false positives during change detection. Whenever a query is executed +in incremental mode, we first check if its already green. If not, we run +`try_mark_green()` on it. If it still isn't green after that, then we actually +invoke the query provider to re-compute the result. + + + +# The Real World: How Persistence Makes Everything Complicated + +The sections above described the underlying algorithm for incremental +compilation but because the compiler process exits after being finished and +takes the query context with its result cache with it into oblivion, we have +persist data to disk, so the next compilation session can make use of it. +This comes with a whole new set of implementation challenges: + +- The query results cache is stored to disk, so they are not readily available + for change comparison. +- A subsequent compilation session will start off with new version of the code + that has arbitrary changes applied to it. All kinds of IDs and indices that + are generated from a global, sequential counter (e.g. `NodeId`, `DefId`, etc) + might have shifted, making the persisted results on disk not immediately + usable anymore because the same numeric IDs and indices might refer to + completely new things in the new compilation session. +- Persisting things to disk comes at a cost, so not every tiny piece of + information should be actually cached in between compilation sessions. + Fixed-sized, plain-old-data is preferred to complex things that need to run + branching code during (de-)serialization. + +The following sections describe how the compiler currently solves these issues. + +## A Question Of Stability: Bridging The Gap Between Compilation Sessions + +As noted before, various IDs (like `DefId`) are generated by the compiler in a +way that depends on the contents of the source code being compiled. ID assignment +is usually deterministic, that is, if the exact same code is compiled twice, +the same things will end up with the same IDs. However, if something +changes, e.g. a function is added in the middle of a file, there is no +guarantee that anything will have the same ID as it had before. + +As a consequence we cannot represent the data in our on-disk cache the same +way it is represented in memory. For example, if we just stored a piece +of type information like `TyKind::FnDef(DefId, &'tcx Substs<'tcx>)` (as we do +in memory) and then the contained `DefId` points to a different function in +a new compilation session we'd be in trouble. + +The solution to this problem is to find "stable" forms for IDs which remain +valid in between compilation sessions. For the most important case, `DefId`s, +these are the so-called `DefPath`s. Each `DefId` has a +corresponding `DefPath` but in place of a numeric ID, a `DefPath` is based on +the path to the identified item, e.g. `std::collections::HashMap`. The +advantage of an ID like this is that it is not affected by unrelated changes. +For example, one can add a new function to `std::collections` but +`std::collections::HashMap` would still be `std::collections::HashMap`. A +`DefPath` is "stable" across changes made to the source code while a `DefId` +isn't. + +There is also the `DefPathHash` which is just a 128-bit hash value of the +`DefPath`. The two contain the same information and we mostly use the +`DefPathHash` because it simpler to handle, being `Copy` and self-contained. + +This principle of stable identifiers is used to make the data in the on-disk +cache resilient to source code changes. Instead of storing a `DefId`, we store +the `DefPathHash` and when we deserialize something from the cache, we map the +`DefPathHash` to the corresponding `DefId` in the *current* compilation session +(which is just a simple hash table lookup). + +The `HirId`, used for identifying HIR components that don't have their own +`DefId`, is another such stable ID. It is (conceptually) a pair of a `DefPath` +and a `LocalId`, where the `LocalId` identifies something (e.g. a `hir::Expr`) +locally within its "owner" (e.g. a `hir::Item`). If the owner is moved around, +the `LocalId`s within it are still the same. + + + +## Checking Query Results For Changes: StableHash And Fingerprints + +In order to do red-green-marking we often need to check if the result of a +query has changed compared to the result it had during the previous +compilation session. There are two performance problems with this though: + +- We'd like to avoid having to load the previous result from disk just for + doing the comparison. We already computed the new result and will use that. + Also loading a result from disk will "pollute" the interners with data that + is unlikely to ever be used. +- We don't want to store each and every result in the on-disk cache. For + example, it would be wasted effort to persist things to disk that are + already available in upstream crates. + +The compiler avoids these problems by using so-called `Fingerprint`s. Each time +a new query result is computed, the query engine will compute a 128 bit hash +value of the result. We call this hash value "the `Fingerprint` of the query +result". The hashing is (and has to be) done "in a stable way". This means +that whenever something is hashed that might change in between compilation +sessions (e.g. a `DefId`), we instead hash its stable equivalent +(e.g. the corresponding `DefPath`). That's what the whole `StableHash` +infrastructure is for. This way `Fingerprint`s computed in two +different compilation sessions are still comparable. + +The next step is to store these fingerprints along with the dependency graph. +This is cheap since fingerprints are just bytes to be copied. It's also cheap to +load the entire set of fingerprints together with the dependency graph. + +Now, when red-green-marking reaches the point where it needs to check if a +result has changed, it can just compare the (already loaded) previous +fingerprint to the fingerprint of the new result. + +This approach works rather well but it's not without flaws: + +- There is a small possibility of hash collisions. That is, two different + results could have the same fingerprint and the system would erroneously + assume that the result hasn't changed, leading to a missed update. + + We mitigate this risk by using a high-quality hash function and a 128 bit + wide hash value. Due to these measures the practical risk of a hash + collision is negligible. + +- Computing fingerprints is quite costly. It is the main reason why incremental + compilation can be slower than non-incremental compilation. We are forced to + use a good and thus expensive hash function, and we have to map things to + their stable equivalents while doing the hashing. + +In the future we might want to explore different approaches to this problem. +For now it's `StableHash` and `Fingerprint`. + + + +## A Tale Of Two DepGraphs: The Old And The New + +The initial description of dependency tracking glosses over a few details +that quickly become a head scratcher when actually trying to implement things. +In particular it's easy to overlook that we are actually dealing with *two* +dependency graphs: The one we built during the previous compilation session and +the one that we are building for the current compilation session. + +When a compilation session starts, the compiler loads the previous dependency +graph into memory as an immutable piece of data. Then, when a query is invoked, +it will first try to mark the corresponding node in the graph as green. This +means really that we are trying to mark the node in the *previous* dep-graph +as green that corresponds to the query key in the *current* session. How do we +do this mapping between current query key and previous `DepNode`? The answer +is again `Fingerprint`s: Nodes in the dependency graph are identified by a +fingerprint of the query key. Since fingerprints are stable across compilation +sessions, computing one in the current session allows us to find a node +in the dependency graph from the previous session. If we don't find a node with +the given fingerprint, it means that the query key refers to something that +did not yet exist in the previous session. + +So, having found the dep-node in the previous dependency graph, we can look +up its dependencies (also dep-nodes in the previous graph) and continue with +the rest of the try-mark-green algorithm. The next interesting thing happens +when we successfully marked the node as green. At that point we copy the node +and the edges to its dependencies from the old graph into the new graph. We +have to do this because the new dep-graph cannot not acquire the +node and edges via the regular dependency tracking. The tracking system can +only record edges while actually running a query -- but running the query, +although we have the result already cached, is exactly what we want to avoid. + +Once the compilation session has finished, all the unchanged parts have been +copied over from the old into the new dependency graph, while the changed parts +have been added to the new graph by the tracking system. At this point, the +new graph is serialized out to disk, alongside the query result cache, and can +act as the previous dep-graph in a subsequent compilation session. + + +## Didn't You Forget Something?: Cache Promotion +TODO + + +# The Future: Shortcomings Of The Current System and Possible Solutions +TODO + + +[query-model]: ./query-evaluation-model-in-detail.html diff --git a/src/incremental-compilation.md b/src/queries/incremental-compilation.md similarity index 100% rename from src/incremental-compilation.md rename to src/queries/incremental-compilation.md diff --git a/src/queries/query-evaluation-model-in-detail.md b/src/queries/query-evaluation-model-in-detail.md index d2dc10479..41789637d 100644 --- a/src/queries/query-evaluation-model-in-detail.md +++ b/src/queries/query-evaluation-model-in-detail.md @@ -123,23 +123,24 @@ fn type_check_crate_provider(tcx, _key: ()) { } ``` -We see that the `type_check_crate` query accesses input data (`tcx.hir_map`) -and invokes other queries (`type_check_item`). The `type_check_item` +We see that the `type_check_crate` query accesses input data +(`tcx.hir_map.list_of_items()`) and invokes other queries +(`type_check_item`). The `type_check_item` invocations will themselves access input data and/or invoke other queries, so that in the end the DAG of query invocations will be built up backwards from the node that was initially executed: -``` - (1) - hir_map <--------------------------------------------------- type_check_crate() - ^ | - | (4) (3) (2) | - +-- Hir(foo) <--- type_of(foo) <--- type_check_item(foo) <-------+ - | | | - | +-----------------+ | - | | | - | (6) v (5) (7) | - +-- Hir(bar) <--- type_of(bar) <--- type_check_item(bar) <-------+ +```ignore + (2) (1) + list_of_all_hir_items <----------------------------- type_check_crate() + | + (5) (4) (3) | + Hir(foo) <--- type_of(foo) <--- type_check_item(foo) <-------+ + | | + +-----------------+ | + | | + (7) v (6) (8) | + Hir(bar) <--- type_of(bar) <--- type_check_item(bar) <-------+ // (x) denotes invocation order ``` diff --git a/src/query.md b/src/query.md index ee7d60e4b..703c560e0 100644 --- a/src/query.md +++ b/src/query.md @@ -37,8 +37,8 @@ compiler (for example, generating MIR) work exactly like this. ### The Query Evaluation Model in Detail -The [Query Evaluation Model in Detail](query-evaluation-model-in-detail.html) -chapter gives a more in-depth description of what queries are and how they work. +The [Query Evaluation Model in Detail][query-model] chapter gives a more +in-depth description of what queries are and how they work. If you intend to write a query of your own, this is a good read. ### Invoking queries @@ -267,3 +267,4 @@ impl<'tcx> QueryDescription for queries::type_of<'tcx> { } ``` +[query-model]: queries/query-evaluation-model-in-detail.html diff --git a/src/variance.md b/src/variance.md index 9fe98b4a1..c6a1a320b 100644 --- a/src/variance.md +++ b/src/variance.md @@ -139,7 +139,7 @@ crate (through `crate_variances`), but since most changes will not result in a change to the actual results from variance inference, the `variances_of` query will wind up being considered green after it is re-evaluated. -[rga]: ./incremental-compilation.html +[rga]: ./queries/incremental-compilation.html From 50c84d1a879d92fea22fd483666777da05abc8a2 Mon Sep 17 00:00:00 2001 From: "Zack M. Davis" Date: Tue, 5 Feb 2019 22:46:53 -0800 Subject: [PATCH 498/648] `_with_applicability` methods are gone The simpler `span_suggestion` method name now takes the applicability argument, thanks to Andy Russell (rust-lang/rust@0897ffc28f6). --- src/diag.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/src/diag.md b/src/diag.md index 9a8020314..cfdd82aa2 100644 --- a/src/diag.md +++ b/src/diag.md @@ -85,11 +85,11 @@ Server][rls] and [`rustfix`][rustfix]. [rustfix]: https://github.com/rust-lang-nursery/rustfix Not all suggestions should be applied mechanically. Use the -[`span_suggestion_with_applicability`][sswa] method of `DiagnosticBuilder` to -make a suggestion while providing a hint to tools whether the suggestion is -mechanically applicable or not. +[`span_suggestion`][span_suggestion] method of `DiagnosticBuilder` to +make a suggestion. The last argument provides a hint to tools whether +the suggestion is mechanically applicable or not. -[sswa]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.DiagnosticBuilder.html#method.span_suggestion_with_applicability +[span_suggestion]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.DiagnosticBuilder.html#method.span_suggestion For example, to make our `qux` suggestion machine-applicable, we would do: @@ -97,8 +97,7 @@ For example, to make our `qux` suggestion machine-applicable, we would do: let mut err = sess.struct_span_err(sp, "oh no! this is an error!"); if let Ok(snippet) = sess.source_map().span_to_snippet(sp) { - // Add applicability info! - err.span_suggestion_with_applicability( + err.span_suggestion( suggestion_sp, "try using a qux here", format!("qux {}", snip), @@ -145,7 +144,7 @@ error: aborting due to previous error For more information about this error, try `rustc --explain E0999`. ``` -There are a few other [`Applicability`][appl] possibilities: +The possible values of [`Applicability`][appl] are: - `MachineApplicable`: Can be applied mechanically. - `HasPlaceholders`: Cannot be applied mechanically because it has placeholder From d94a06dd52760b83ec98ff3ba86fa8c47a062d5f Mon Sep 17 00:00:00 2001 From: varkor Date: Sat, 16 Feb 2019 13:56:40 +0000 Subject: [PATCH 499/648] Add "lang item" to glossary --- src/appendix/glossary.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index a35008053..a2effc7fc 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -34,6 +34,7 @@ inference variable | when doing type or region inference, an "inference va infcx | the inference context (see `librustc/infer`) IR | Intermediate Representation. A general term in compilers. During compilation, the code is transformed from raw source (ASCII text) to various IRs. In Rust, these are primarily HIR, MIR, and LLVM IR. Each IR is well-suited for some set of computations. For example, MIR is well-suited for the borrow checker, and LLVM IR is well-suited for codegen because LLVM accepts it. IRLO | `IRLO` or `irlo` is sometimes used as an abbreviation for [internals.rust-lang.org](https://internals.rust-lang.org). +lang item | items that represent concepts intrinsic to the language itself, such as special built-in traits like `Sync` and `Send`; or traits representing operations such as `Add`; or functions that are called by the compiler. ([see more](https://doc.rust-lang.org/1.9.0/book/lang-items.html)) local crate | the crate currently being compiled. LTO | Link-Time Optimizations. A set of optimizations offered by LLVM that occur just before the final binary is linked. These include optimizations like removing functions that are never used in the final program, for example. _ThinLTO_ is a variant of LTO that aims to be a bit more scalable and efficient, but possibly sacrifices some optimizations. You may also read issues in the Rust repo about "FatLTO", which is the loving nickname given to non-Thin LTO. LLVM documentation: [here][lto] and [here][thinlto] [LLVM] | (actually not an acronym :P) an open-source compiler backend. It accepts LLVM IR and outputs native binaries. Various languages (e.g. Rust) can then implement a compiler front-end that output LLVM IR and use LLVM to compile to all the platforms LLVM supports. From 7ef92b9bb658819e50feb79e3d586e3d0f87bcc7 Mon Sep 17 00:00:00 2001 From: varkor Date: Sat, 16 Feb 2019 13:59:59 +0000 Subject: [PATCH 500/648] Add "item" to glossary --- src/appendix/glossary.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index a2effc7fc..3ceef9344 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -34,6 +34,7 @@ inference variable | when doing type or region inference, an "inference va infcx | the inference context (see `librustc/infer`) IR | Intermediate Representation. A general term in compilers. During compilation, the code is transformed from raw source (ASCII text) to various IRs. In Rust, these are primarily HIR, MIR, and LLVM IR. Each IR is well-suited for some set of computations. For example, MIR is well-suited for the borrow checker, and LLVM IR is well-suited for codegen because LLVM accepts it. IRLO | `IRLO` or `irlo` is sometimes used as an abbreviation for [internals.rust-lang.org](https://internals.rust-lang.org). +item | a kind of "definition" in the language, such as a static, const, use statement, module, struct, etc. Concretely, this corresponds to the `Item` type. lang item | items that represent concepts intrinsic to the language itself, such as special built-in traits like `Sync` and `Send`; or traits representing operations such as `Add`; or functions that are called by the compiler. ([see more](https://doc.rust-lang.org/1.9.0/book/lang-items.html)) local crate | the crate currently being compiled. LTO | Link-Time Optimizations. A set of optimizations offered by LLVM that occur just before the final binary is linked. These include optimizations like removing functions that are never used in the final program, for example. _ThinLTO_ is a variant of LTO that aims to be a bit more scalable and efficient, but possibly sacrifices some optimizations. You may also read issues in the Rust repo about "FatLTO", which is the loving nickname given to non-Thin LTO. LLVM documentation: [here][lto] and [here][thinlto] From ee661e8d61dbcc5065ed59eabfab0154c00c1f06 Mon Sep 17 00:00:00 2001 From: varkor Date: Sat, 16 Feb 2019 14:03:39 +0000 Subject: [PATCH 501/648] Add "early-bound lifetime" and "late-bound lifetime" to the glossary --- src/appendix/glossary.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 3ceef9344..3b4df535d 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -20,6 +20,7 @@ data-flow analysis | a static analysis that figures out what properties ar DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. Double pointer | a pointer with additional metadata. See "fat pointer" for more. DST | Dynamically-Sized Type. A type for which the compiler cannot statically know the size in memory (e.g. `str` or `[u8]`). Such types don't implement `Sized` and cannot be allocated on the stack. They can only occur as the last field in a struct. They can only be used behind a pointer (e.g. `&str` or `&[u8]`). +early-bound lifetime | a lifetime region, which is bound in an item's `Generics` and substituted using a `Substs`. Contrast with **late-bound lifetime**. ([see more](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/enum.RegionKind.html#bound-regions)) empty type | see "uninhabited type". Fat pointer | a two word value carrying the address of some value, along with some further information necessary to put the value to use. Rust includes two kinds of "fat pointers": references to slices, and trait objects. A reference to a slice carries the starting address of the slice and its length. A trait object carries a value's address and a pointer to the trait's implementation appropriate to that value. "Fat pointers" are also known as "wide pointers", and "double pointers". free variable | a "free variable" is one that is not bound within an expression or term; see [the background chapter for more](./background.html#free-vs-bound) @@ -36,6 +37,7 @@ IR | Intermediate Representation. A general term in compil IRLO | `IRLO` or `irlo` is sometimes used as an abbreviation for [internals.rust-lang.org](https://internals.rust-lang.org). item | a kind of "definition" in the language, such as a static, const, use statement, module, struct, etc. Concretely, this corresponds to the `Item` type. lang item | items that represent concepts intrinsic to the language itself, such as special built-in traits like `Sync` and `Send`; or traits representing operations such as `Add`; or functions that are called by the compiler. ([see more](https://doc.rust-lang.org/1.9.0/book/lang-items.html)) +late-bound lifetime | a lifetime region, which is bound in a HRTB and substituted with specific functions in the compiler, such as `liberate_late_bound_regions`. Contrast with **early-bound lifetime**. ([see more](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/enum.RegionKind.html#bound-regions)) local crate | the crate currently being compiled. LTO | Link-Time Optimizations. A set of optimizations offered by LLVM that occur just before the final binary is linked. These include optimizations like removing functions that are never used in the final program, for example. _ThinLTO_ is a variant of LTO that aims to be a bit more scalable and efficient, but possibly sacrifices some optimizations. You may also read issues in the Rust repo about "FatLTO", which is the loving nickname given to non-Thin LTO. LLVM documentation: [here][lto] and [here][thinlto] [LLVM] | (actually not an acronym :P) an open-source compiler backend. It accepts LLVM IR and outputs native binaries. Various languages (e.g. Rust) can then implement a compiler front-end that output LLVM IR and use LLVM to compile to all the platforms LLVM supports. From 50bcd366db07b54f20cef36fb9e40654d5033a41 Mon Sep 17 00:00:00 2001 From: varkor Date: Sat, 16 Feb 2019 14:09:49 +0000 Subject: [PATCH 502/648] Add "intern" to the glossary --- src/appendix/glossary.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 3b4df535d..32171a374 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -33,6 +33,7 @@ ICE | internal compiler error. When the compiler crashes. ICH | incremental compilation hash. ICHs are used as fingerprints for things such as HIR and crate metadata, to check if changes have been made. This is useful in incremental compilation to see if part of a crate has changed and should be recompiled. inference variable | when doing type or region inference, an "inference variable" is a kind of special type/region that represents what you are trying to infer. Think of X in algebra. For example, if we are trying to infer the type of a variable in a program, we create an inference variable to represent that unknown type. infcx | the inference context (see `librustc/infer`) +intern | interning refers to storing certain frequently-used constant data, such as strings, and then referring to the data by an identifier (e.g. a `Symbol`) rather than the data itself, to reduce memory usage. IR | Intermediate Representation. A general term in compilers. During compilation, the code is transformed from raw source (ASCII text) to various IRs. In Rust, these are primarily HIR, MIR, and LLVM IR. Each IR is well-suited for some set of computations. For example, MIR is well-suited for the borrow checker, and LLVM IR is well-suited for codegen because LLVM accepts it. IRLO | `IRLO` or `irlo` is sometimes used as an abbreviation for [internals.rust-lang.org](https://internals.rust-lang.org). item | a kind of "definition" in the language, such as a static, const, use statement, module, struct, etc. Concretely, this corresponds to the `Item` type. From f420e427c7b31364b01226fa768e12d8b741bd53 Mon Sep 17 00:00:00 2001 From: varkor Date: Sat, 16 Feb 2019 14:11:56 +0000 Subject: [PATCH 503/648] Add "memoise" to the glossary --- src/appendix/glossary.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 32171a374..3f10d731d 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -42,6 +42,7 @@ late-bound lifetime | a lifetime region, which is bound in a HRTB and subst local crate | the crate currently being compiled. LTO | Link-Time Optimizations. A set of optimizations offered by LLVM that occur just before the final binary is linked. These include optimizations like removing functions that are never used in the final program, for example. _ThinLTO_ is a variant of LTO that aims to be a bit more scalable and efficient, but possibly sacrifices some optimizations. You may also read issues in the Rust repo about "FatLTO", which is the loving nickname given to non-Thin LTO. LLVM documentation: [here][lto] and [here][thinlto] [LLVM] | (actually not an acronym :P) an open-source compiler backend. It accepts LLVM IR and outputs native binaries. Various languages (e.g. Rust) can then implement a compiler front-end that output LLVM IR and use LLVM to compile to all the platforms LLVM supports. +memoise | memoisation is the process of storing the results of (pure) computations (such as pure function calls) to avoid having to repeat them in the future. This is generally a trade-off between execution speed and memory usage. MIR | the Mid-level IR that is created after type-checking for use by borrowck and codegen ([see more](../mir/index.html)) miri | an interpreter for MIR used for constant evaluation ([see more](../miri.html)) normalize | a general term for converting to a more canonical form, but in the case of rustc typically refers to [associated type normalization](../traits/associated-types.html#normalize) From de2ab38ddbf6572befebc22ce240cc72216698c9 Mon Sep 17 00:00:00 2001 From: varkor Date: Sat, 16 Feb 2019 14:15:41 +0000 Subject: [PATCH 504/648] Add "drop glue" to the glossary --- src/appendix/glossary.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 3f10d731d..86a128efb 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -19,6 +19,7 @@ DAG | a directed acyclic graph is used during compilation t data-flow analysis | a static analysis that figures out what properties are true at each point in the control-flow of a program; see [the background chapter for more](./background.html#dataflow) DefId | an index identifying a definition (see `librustc/hir/def_id.rs`). Uniquely identifies a `DefPath`. Double pointer | a pointer with additional metadata. See "fat pointer" for more. +drop glue | (internal) compiler-generated instructions that handle calling the destructors (`Drop`) for data types. DST | Dynamically-Sized Type. A type for which the compiler cannot statically know the size in memory (e.g. `str` or `[u8]`). Such types don't implement `Sized` and cannot be allocated on the stack. They can only occur as the last field in a struct. They can only be used behind a pointer (e.g. `&str` or `&[u8]`). early-bound lifetime | a lifetime region, which is bound in an item's `Generics` and substituted using a `Substs`. Contrast with **late-bound lifetime**. ([see more](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/enum.RegionKind.html#bound-regions)) empty type | see "uninhabited type". From 7cc2afab20d1d729fa95cb62eb585b99153ff59b Mon Sep 17 00:00:00 2001 From: varkor Date: Thu, 5 Apr 2018 11:49:38 +0100 Subject: [PATCH 505/648] Add preliminary chapter on kinds --- src/SUMMARY.md | 1 + src/kinds.md | 31 +++++++++++++++++++++++++++++++ src/ty.md | 6 ++++-- 3 files changed, 36 insertions(+), 2 deletions(-) create mode 100644 src/kinds.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 091860e9a..b27114f1e 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -38,6 +38,7 @@ - [The HIR (High-level IR)](./hir.md) - [Lowering AST to HIR](./lowering.md) - [The `ty` module: representing types](./ty.md) +- [Kinds](./kinds.md) - [Type inference](./type-inference.md) - [Trait solving (old-style)](./traits/resolution.md) - [Higher-ranked trait bounds](./traits/hrtb.md) diff --git a/src/kinds.md b/src/kinds.md new file mode 100644 index 000000000..d811423a8 --- /dev/null +++ b/src/kinds.md @@ -0,0 +1,31 @@ +# Kinds +A `ty::subst::Kind<'tcx>` represents some entity in the type system: currently +either a type (`Ty<'tcx>`) or a lifetime (`ty::Region<'tcx>`), though in the +future this will also include constants (`ty::Const<'tcx>`) to facilitate the +use of const generics. `Kind` is used for type and lifetime substitution (from +abstract type and lifetime parameters to concrete types and lifetimes). + +## `UnpackedKind` +As `Kind` itself is not type-safe (see [`Kind`](#kind)), the `UnpackedKind` enum +provides a more convenient and safe interface for dealing with kinds. To +convert from an `UnpackedKind` to a `Kind`, you can call `Kind::from` (or +`.into`). It should not be necessary to convert a `Kind` to an `UnpackedKind`: +instead, you should prefer to deal with `UnpackedKind`, converting it only when +passing it to `Subst` methods. + +## `Kind` +The actual `Kind` struct is optimised for space, storing the type or lifetime +as an interned pointer containing a mask identifying its kind (in the lowest +2 bits). + +## `Subst` +`ty::subst::Subst<'tcx>` is simply defined as a slice of `Kind<'tcx>`s +and acts as an ordered list of substitutions from kind parameters (i.e. +type and lifetime parameters) to kinds. + +For example, given a `HashMap` with two type parameters, `K` and `V`, an +instantiation of the parameters, for example `HashMap`, would be +represented by the substitution `&'tcx [tcx.types.i32, tcx.types.u32]`. + +`Subst` provides various convenience methods to instantiant substitutions +given item definitions. diff --git a/src/ty.md b/src/ty.md index fea9afbeb..d9979bc0b 100644 --- a/src/ty.md +++ b/src/ty.md @@ -141,8 +141,8 @@ In addition to types, there are a number of other arena-allocated data structures that you can allocate, and which are found in this module. Here are a few examples: -- `Substs`, allocated with `mk_substs` – this will intern a slice of types, - often used to specify the values to be substituted for generics +- [`Substs`][subst], allocated with `mk_substs` – this will intern a slice of + types, often used to specify the values to be substituted for generics (e.g. `HashMap` would be represented as a slice `&'tcx [tcx.types.i32, tcx.types.u32]`). - `TraitRef`, typically passed by value – a **trait reference** @@ -153,6 +153,8 @@ module. Here are a few examples: - `Predicate` defines something the trait system has to prove (see `traits` module). +[subst]: ./kinds.html#subst + ### Import conventions Although there is no hard and fast rule, the `ty` module tends to be used like From 99d47b84c6841c43559f4d47d485eeceabe0ebdd Mon Sep 17 00:00:00 2001 From: varkor Date: Sat, 16 Feb 2019 14:38:15 +0000 Subject: [PATCH 506/648] Add more information and an example to Kind chapter --- src/kinds.md | 62 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 40 insertions(+), 22 deletions(-) diff --git a/src/kinds.md b/src/kinds.md index d811423a8..bd483c868 100644 --- a/src/kinds.md +++ b/src/kinds.md @@ -1,31 +1,49 @@ # Kinds -A `ty::subst::Kind<'tcx>` represents some entity in the type system: currently -either a type (`Ty<'tcx>`) or a lifetime (`ty::Region<'tcx>`), though in the -future this will also include constants (`ty::Const<'tcx>`) to facilitate the -use of const generics. `Kind` is used for type and lifetime substitution (from -abstract type and lifetime parameters to concrete types and lifetimes). - -## `UnpackedKind` -As `Kind` itself is not type-safe (see [`Kind`](#kind)), the `UnpackedKind` enum -provides a more convenient and safe interface for dealing with kinds. To -convert from an `UnpackedKind` to a `Kind`, you can call `Kind::from` (or -`.into`). It should not be necessary to convert a `Kind` to an `UnpackedKind`: -instead, you should prefer to deal with `UnpackedKind`, converting it only when -passing it to `Subst` methods. - -## `Kind` -The actual `Kind` struct is optimised for space, storing the type or lifetime -as an interned pointer containing a mask identifying its kind (in the lowest -2 bits). +A `ty::subst::Kind<'tcx>` represents some entity in the type system: a type +(`Ty<'tcx>`), lifetime (`ty::Region<'tcx>`) or constant (`ty::Const<'tcx>`). +`Kind` is used to perform substitutions of generic parameters for concrete +arguments, such as when calling a function with generic parameters explicitly +with type arguments. Substitutions are represented using the +[`Subst` type](#subst) as described below. ## `Subst` -`ty::subst::Subst<'tcx>` is simply defined as a slice of `Kind<'tcx>`s -and acts as an ordered list of substitutions from kind parameters (i.e. -type and lifetime parameters) to kinds. +`ty::subst::Subst<'tcx>` is intuitively simply a slice of `Kind<'tcx>`s, +acting as an ordered list of substitutions from generic parameters to +concrete arguments (such as types, lifetimes and consts). For example, given a `HashMap` with two type parameters, `K` and `V`, an instantiation of the parameters, for example `HashMap`, would be represented by the substitution `&'tcx [tcx.types.i32, tcx.types.u32]`. `Subst` provides various convenience methods to instantiant substitutions -given item definitions. +given item definitions, which should generally be used rather than explicitly +constructing such substitution slices. + +## `Kind` +The actual `Kind` struct is optimised for space, storing the type, lifetime or +const as an interned pointer containing a mask identifying its kind (in the +lowest 2 bits). Unless you are working with the `Subst` implementation +specifically, you should generally not have to deal with `Kind` and instead +make use of the safe [`UnpackedKind`](#unpackedkind) abstraction. + +## `UnpackedKind` +As `Kind` itself is not type-safe, the `UnpackedKind` enum provides a more +convenient and safe interface for dealing with kinds. An `UnpackedKind` can +be converted to a raw `Kind` using `Kind::from()` (or simply `.into()` when +the context is clear). As mentioned earlier, substition lists store raw +`Kind`s, so before dealing with them, it is preferable to convert them to +`UnpackedKind`s first. This is done by calling the `.unpack()` method. + +```rust +// An example of unpacking and packing a kind. +fn deal_with_kind<'tcx>(kind: Kind<'tcx>) -> Kind<'tcx> { + // Unpack a raw `Kind` to deal with it safely. + let new_kind: UnpackedKind<'tcx> = match kind.unpack() { + UnpackedKind::Type(ty) => { /* ... */ } + UnpackedKind::Lifetime(lt) => { /* ... */ } + UnpackedKind::Const(ct) => { /* ... */ } + }; + // Pack the `UnpackedKind` to store it in a substitution list. + new_kind.into() +} +``` From d911ba15a245da5dbe855b84707a40f4326af949 Mon Sep 17 00:00:00 2001 From: varkor Date: Mon, 18 Feb 2019 22:57:57 +0000 Subject: [PATCH 507/648] Ignore Rust snippet --- src/kinds.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/kinds.md b/src/kinds.md index bd483c868..d5e218c31 100644 --- a/src/kinds.md +++ b/src/kinds.md @@ -34,7 +34,7 @@ the context is clear). As mentioned earlier, substition lists store raw `Kind`s, so before dealing with them, it is preferable to convert them to `UnpackedKind`s first. This is done by calling the `.unpack()` method. -```rust +```rust,ignore // An example of unpacking and packing a kind. fn deal_with_kind<'tcx>(kind: Kind<'tcx>) -> Kind<'tcx> { // Unpack a raw `Kind` to deal with it safely. From 10fb45ee27437b326d04019398fa4add01f4d15d Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Tue, 19 Feb 2019 00:15:07 +0000 Subject: [PATCH 508/648] Small clarifications to glossary definitions --- src/appendix/glossary.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 86a128efb..a5d5bf56a 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -21,7 +21,7 @@ DefId | an index identifying a definition (see `librustc/hir/ Double pointer | a pointer with additional metadata. See "fat pointer" for more. drop glue | (internal) compiler-generated instructions that handle calling the destructors (`Drop`) for data types. DST | Dynamically-Sized Type. A type for which the compiler cannot statically know the size in memory (e.g. `str` or `[u8]`). Such types don't implement `Sized` and cannot be allocated on the stack. They can only occur as the last field in a struct. They can only be used behind a pointer (e.g. `&str` or `&[u8]`). -early-bound lifetime | a lifetime region, which is bound in an item's `Generics` and substituted using a `Substs`. Contrast with **late-bound lifetime**. ([see more](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/enum.RegionKind.html#bound-regions)) +early-bound lifetime | a lifetime region that is substituted at its definition site. Bound in an item's `Generics` and substituted using a `Substs`. Contrast with **late-bound lifetime**. ([see more](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/enum.RegionKind.html#bound-regions)) empty type | see "uninhabited type". Fat pointer | a two word value carrying the address of some value, along with some further information necessary to put the value to use. Rust includes two kinds of "fat pointers": references to slices, and trait objects. A reference to a slice carries the starting address of the slice and its length. A trait object carries a value's address and a pointer to the trait's implementation appropriate to that value. "Fat pointers" are also known as "wide pointers", and "double pointers". free variable | a "free variable" is one that is not bound within an expression or term; see [the background chapter for more](./background.html#free-vs-bound) @@ -39,11 +39,11 @@ IR | Intermediate Representation. A general term in compil IRLO | `IRLO` or `irlo` is sometimes used as an abbreviation for [internals.rust-lang.org](https://internals.rust-lang.org). item | a kind of "definition" in the language, such as a static, const, use statement, module, struct, etc. Concretely, this corresponds to the `Item` type. lang item | items that represent concepts intrinsic to the language itself, such as special built-in traits like `Sync` and `Send`; or traits representing operations such as `Add`; or functions that are called by the compiler. ([see more](https://doc.rust-lang.org/1.9.0/book/lang-items.html)) -late-bound lifetime | a lifetime region, which is bound in a HRTB and substituted with specific functions in the compiler, such as `liberate_late_bound_regions`. Contrast with **early-bound lifetime**. ([see more](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/enum.RegionKind.html#bound-regions)) +late-bound lifetime | a lifetime region that is substituted at its call site. Bound in a HRTB and substituted by specific functions in the compiler, such as `liberate_late_bound_regions`. Contrast with **early-bound lifetime**. ([see more](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/enum.RegionKind.html#bound-regions)) local crate | the crate currently being compiled. LTO | Link-Time Optimizations. A set of optimizations offered by LLVM that occur just before the final binary is linked. These include optimizations like removing functions that are never used in the final program, for example. _ThinLTO_ is a variant of LTO that aims to be a bit more scalable and efficient, but possibly sacrifices some optimizations. You may also read issues in the Rust repo about "FatLTO", which is the loving nickname given to non-Thin LTO. LLVM documentation: [here][lto] and [here][thinlto] [LLVM] | (actually not an acronym :P) an open-source compiler backend. It accepts LLVM IR and outputs native binaries. Various languages (e.g. Rust) can then implement a compiler front-end that output LLVM IR and use LLVM to compile to all the platforms LLVM supports. -memoise | memoisation is the process of storing the results of (pure) computations (such as pure function calls) to avoid having to repeat them in the future. This is generally a trade-off between execution speed and memory usage. +memoize | memoization is the process of storing the results of (pure) computations (such as pure function calls) to avoid having to repeat them in the future. This is typically a trade-off between execution speed and memory usage. MIR | the Mid-level IR that is created after type-checking for use by borrowck and codegen ([see more](../mir/index.html)) miri | an interpreter for MIR used for constant evaluation ([see more](../miri.html)) normalize | a general term for converting to a more canonical form, but in the case of rustc typically refers to [associated type normalization](../traits/associated-types.html#normalize) From f59ed9dbc52f118199704f320c98fd53271d2e04 Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Thu, 7 Feb 2019 22:45:01 -0500 Subject: [PATCH 509/648] issue_130_7 implementing new features --- src/SUMMARY.md | 1 + src/implementing_new_feature.md | 135 ++++++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+) create mode 100644 src/implementing_new_feature.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index b27114f1e..723608da9 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -14,6 +14,7 @@ - [Adding new tests](./tests/adding.md) - [Using `compiletest` + commands to control test execution](./compiletest.md) - [Walkthrough: a typical contribution](./walkthrough.md) +- [Implementing new feature](./implementing_new_feature.md) - [Stabilizing Features](./stabilization_guide.md) - [Debugging the Compiler](./compiler-debugging.md) - [Profiling the compiler](./profiling.md) diff --git a/src/implementing_new_feature.md b/src/implementing_new_feature.md new file mode 100644 index 000000000..2bc65e145 --- /dev/null +++ b/src/implementing_new_feature.md @@ -0,0 +1,135 @@ +# Implement New Feature + +When you want to implement a new significant feature in the compiler, you need to go through this process to make sure everything goes smoothly. + +## The @rfcbot (p)FCP process + +When the change is small and uncontroversial, then it can be done +with just writing a PR and getting r+ from someone who knows that part of the code. However, if the change is potentially controversial, it would be a bad idea to push it without onsensus from the rest of the team (both in the "distributed system" sense to make sure you don't break anything you don't know about, and in the social sense to avoid PR fights). + +If such a change seems to small to require a full formal RFC process (e.g. a big refactoring of the code, or a "technically-breaking" change, or a "big bugfix" that basically amounts to a small feature) but is still too controversial or big to get by with a single r+, you can start a pFCP (or, if you don't have r+ rights, ask someone who has them to start one - and unless they have a concern themselves, they should). + +Again, the pFCP process is only needed if you need consensus - if you don't think anyone would have a problem with your change, it's ok to get by with only an r+. For example, it is OK to add or modify unstable command-line flags or attributes without an pFCP for compiler development or standard library use, as long as you don't expect them to be in wide use in the nightly ecosystem. + +Rust community is trying to find a better process for iterating on features on nightly, but if you are trying to write a feature that is not too bikesheddy, you could try to ask for an pFCP to land it on nightly under a feature gate. That might work out. However, some team-member might still tell you to go write an actual RFC (however, someone still needs to write an RFC before that feature is stabilized). + +You don't have to have the implementation fully ready for r+ to ask for a pFCP, but it is generally a good idea to have at least a proof of concept so that people can see what you are talking about. + +That starts a "proposed final comment period" (pFCP), which requires all members of the team to sign off the FCP. After they all do so, there's a week long "final comment period" where everybody can comment, and if no new concerns are raised, the PR/issue gets FCP approval. + +After an PR/issue had got FCP approval, team-members should help along attempts to implement it, and not block r+ on the controversy that prompted the FCP. If significant new issues arise after FCP approval, then it is ok to re-block the PR until these are resolved. + +For teams and team members: Even if someone is trying to paint the bikeshed your least favorite color, teams and team members should help them move along. Don't be a slow reviewer. + +## A few etiquette notes about (p)FCPs + +FCPs are a place where people unfamiliar with the Rust teams and process interact with the design process. This means that it is extra important for communication to be clear there. Don't assume the PR's writer knows when the next lang team meeting is. + +pFCP checkbox ticking is not supposed to be a place where PRs stall due to low priority - it is supposed to quickly either reach consensus or find a concern. If there is no real obstacle to consensus on the issue, the relevant team should sign off the PFCP reasonably quickly (a week or two). + +Teams: please review pFCP items during the team meetings to make sure this happens. If you find a real concern, please post a summary on the issue, even if you already discussed it on the team IRC channel, or worse, at a late-hours in-person meeting. + +If you think a pFCP item requires further design work which you are too busy to do right now, please say so on the issue/PR thread. This might appear rude, but it's less rude than leaving a PR accumulating bitrot for 2 months because you have no idea what to do with it. + +## How to technically run a pFCP + +pFCPs are run using the @rfcbot bot. For more details about how to correctly use rfcbot, see [the rfcbot guide], but the gist of it is that someone with privileges first tags your issue with a list of teams, and then someone writes this command in a comment in the issue thread: + +```rust, ignore + @rfcbot fcp merge +``` + +or + +```rust, ignore + @rfcbot fcp close +``` + +That will cause rfcbot to comment its ticky-box comment. + +## The logistics of writing features + +There are a few "logistic" hoops you might need to go through in order to implement a feature in a working way. +The more boring details are listed in the Rust repository's [CONTRIBUTING.md], so I'll just link to it here. You should try to at least skim it fully - it contains a bunch of nice tips. + +### Warning Cycles + +In some cases, a feature or bugfix might break some existing programs in some edge cases. In that case, you might want to do a crater run to assess the impact and possibly add a warning cycle, following the [rustc bug-fix procedure]. + +### Stability + +We [value the stability of Rust]. Code that works and runs on stable should (mostly) not break. Because of that, we don't want to release a feature to the world with only team consensus and code review - we want to gain real-world experience on using that feature on nightly, and we might want to change the feature based on that experience. + +To allow for that, we must make sure users don't accidentally depend on that new feature - otherwise, especially if experimentation takes time or is delayed and the feature takes the trains to stable, it would end up de facto stable and we'll not be able to make changes in it without breaking people's code. + +The way we do that is that we make sure all new features are feature gated - they can't be used without a enabling a feature gate (#[feature(foo)]), which can't be done in a stable/beta compiler. See the [stability in code] section for the technical details. + +Eventually, after we gain enough experience using the feature, make the necessary changes, and are satisfied, we expose it to the world using the stabilization process described [here]. Until then, the feature is not set in stone: every part of the feature can be changed, or the feature might be completely rewritten or removed. Features are not supposed to gain tenure by being unstable and unchanged for a year. + +### Tracking Issues + +To keep track of the status of an unstable feature, the experience we get while using it on nightly, and of the concerns that block its stabilization, every feature-gate needs a tracking issue. + +General discussions about the feature should be done on the tracking issue. + +For features that have an RFC, you should use the RFC's tracking issue for the feature. + +For other features, you'll have to make a tracking issue for that feature. The issue title should be "Tracking issue for YOUR FEATURE" and it should have the `B-unstable` & `C-tracking-issue` tags, along with the tag for your subteam (e.g. `T-compiler` if this is a compiler feature). + +For tracking issues for features (as opposed to future-compat warnings), I don't think the description has to contain anything specific. Generally we put the list of items required for stabilization using a github list, e.g. + +```bash, ignore + **Steps:** + + - [ ] Implement the RFC (cc @rust-lang/compiler -- can anyone write up mentoring instructions?) + - [ ] Adjust documentation ([see instructions on forge][doc-guide]) + - Note: no stabilization step here. +``` + +## Stability in code + +The below steps needs to be followed in order to implement a new unstable feature: + +1. Open a [tracking issue] - if you have an RFC, you can use the tracking issue for the RFC. + +2. Pick a name for the feature gate (for RFCs, use the name in the RFC). + +3. Add a feature gate declaration to `libsyntax/feature_gate.rs` In the active `declare_features` block: + +```rust,ignore + // description of feature + (active, $feature_name, "$current_nightly_version", Some($tracking_issue_number), $edition) +``` + +where `$edition` has the type `Option`, and is typically just `None`. +For example: + +```rust,ignore + // allow '|' at beginning of match arms (RFC 1925) +( active, match_beginning_vert, "1.21.0", Some(44101), None), +``` + +The current version is not actually important – the important version is when you are stabilizing a feature. + +4. Prevent usage of the new feature unless the feature gate is set. + +You can check it in most places in the compiler using the expression `tcx.features().$feature_name` (or `sess.features_untracked().borrow().$feature_name` if the tcx is unavailable) + +If the feature gate is not set, you should either maintain the pre-feature behavior or raise an error, depending on what makes sense. + +5. Add a test to ensure the feature cannot be used without a feature gate, by creating `feature-gate-$feature_name.rs` and `feature-gate-$feature_name.stderr` files under the `src/test/ui/feature-gates` directory. + +6. Add a section to the unstable book, in `src/doc/unstable-book/src/language-features/$feature_name.md`. + +7. Write a lots of tests for the new feature. PRs without tests will not be accepted! + +8. Get your PR reviewed and land it. You have now successfully implemented a feature in Rust! + +[the rfcbot guide]: https://github.com/anp/rfcbot-rs/blob/master/CONDUCT.md +[CONTRIBUTING.md]: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md +[rustc bug-fix procedure]: +https://github.com/rust-lang/rust-forge/blob/master/rustc-bug-fix-procedure.md +[value the stability of Rust]: https://github.com/rust-lang/rfcs/blob/master/text/1122-language-semver.md +[stability in code]: +[here]: https://rust-lang.github.io/rustc-guide/stabilization_guide.html +[tracking issue]: \ No newline at end of file From 08522e27ec3a6395968f4172872d628e3444ff88 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Wed, 20 Feb 2019 12:55:34 -0500 Subject: [PATCH 510/648] Update src/SUMMARY.md Co-Authored-By: rajcspsg --- src/SUMMARY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 723608da9..0761e734d 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -14,7 +14,7 @@ - [Adding new tests](./tests/adding.md) - [Using `compiletest` + commands to control test execution](./compiletest.md) - [Walkthrough: a typical contribution](./walkthrough.md) -- [Implementing new feature](./implementing_new_feature.md) +- [Implementing new features](./implementing_new_feature.md) - [Stabilizing Features](./stabilization_guide.md) - [Debugging the Compiler](./compiler-debugging.md) - [Profiling the compiler](./profiling.md) From c9df17bd536699ea10d20321e647bceff8ddf4f2 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Wed, 20 Feb 2019 12:57:01 -0500 Subject: [PATCH 511/648] Update src/implementing_new_feature.md Co-Authored-By: rajcspsg --- src/implementing_new_feature.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/implementing_new_feature.md b/src/implementing_new_feature.md index 2bc65e145..d787954da 100644 --- a/src/implementing_new_feature.md +++ b/src/implementing_new_feature.md @@ -78,7 +78,7 @@ For other features, you'll have to make a tracking issue for that feature. The i For tracking issues for features (as opposed to future-compat warnings), I don't think the description has to contain anything specific. Generally we put the list of items required for stabilization using a github list, e.g. -```bash, ignore +```txt **Steps:** - [ ] Implement the RFC (cc @rust-lang/compiler -- can anyone write up mentoring instructions?) @@ -132,4 +132,4 @@ https://github.com/rust-lang/rust-forge/blob/master/rustc-bug-fix-procedure.md [value the stability of Rust]: https://github.com/rust-lang/rfcs/blob/master/text/1122-language-semver.md [stability in code]: [here]: https://rust-lang.github.io/rustc-guide/stabilization_guide.html -[tracking issue]: \ No newline at end of file +[tracking issue]: From 04995a5ad0888b69df998308ba61c4b5e5b99b26 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Wed, 20 Feb 2019 12:57:26 -0500 Subject: [PATCH 512/648] Update src/implementing_new_feature.md Co-Authored-By: rajcspsg --- src/implementing_new_feature.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/implementing_new_feature.md b/src/implementing_new_feature.md index d787954da..20c250527 100644 --- a/src/implementing_new_feature.md +++ b/src/implementing_new_feature.md @@ -113,7 +113,7 @@ The current version is not actually important – the important version is when 4. Prevent usage of the new feature unless the feature gate is set. -You can check it in most places in the compiler using the expression `tcx.features().$feature_name` (or `sess.features_untracked().borrow().$feature_name` if the tcx is unavailable) + You can check it in most places in the compiler using the expression `tcx.features().$feature_name` (or `sess.features_untracked().borrow().$feature_name` if the tcx is unavailable) If the feature gate is not set, you should either maintain the pre-feature behavior or raise an error, depending on what makes sense. From 4a53240fb4d18231486fa409fd089367b8a6d242 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Wed, 20 Feb 2019 12:58:01 -0500 Subject: [PATCH 513/648] Update src/implementing_new_feature.md Co-Authored-By: rajcspsg --- src/implementing_new_feature.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/implementing_new_feature.md b/src/implementing_new_feature.md index 20c250527..2344ab6f9 100644 --- a/src/implementing_new_feature.md +++ b/src/implementing_new_feature.md @@ -115,7 +115,7 @@ The current version is not actually important – the important version is when You can check it in most places in the compiler using the expression `tcx.features().$feature_name` (or `sess.features_untracked().borrow().$feature_name` if the tcx is unavailable) -If the feature gate is not set, you should either maintain the pre-feature behavior or raise an error, depending on what makes sense. + If the feature gate is not set, you should either maintain the pre-feature behavior or raise an error, depending on what makes sense. 5. Add a test to ensure the feature cannot be used without a feature gate, by creating `feature-gate-$feature_name.rs` and `feature-gate-$feature_name.stderr` files under the `src/test/ui/feature-gates` directory. From e15b26b349f436c5390ee0616caa85cf1f44c9f7 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Wed, 20 Feb 2019 12:58:38 -0500 Subject: [PATCH 514/648] Update src/implementing_new_feature.md Co-Authored-By: rajcspsg --- src/implementing_new_feature.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/implementing_new_feature.md b/src/implementing_new_feature.md index 2344ab6f9..ff26b8550 100644 --- a/src/implementing_new_feature.md +++ b/src/implementing_new_feature.md @@ -54,7 +54,7 @@ The more boring details are listed in the Rust repository's [CONTRIBUTING.md], s ### Warning Cycles -In some cases, a feature or bugfix might break some existing programs in some edge cases. In that case, you might want to do a crater run to assess the impact and possibly add a warning cycle, following the [rustc bug-fix procedure]. +In some cases, a feature or bugfix might break some existing programs in some edge cases. In that case, you might want to do a crater run to assess the impact and possibly add a future-compatibility lint, similar to those used for [edition-gated lints](./diag.md#edition-gated-lints). ### Stability From c0a6e0b49b0027a7f6148ff9fc72f09d45dcba0f Mon Sep 17 00:00:00 2001 From: Rajkumar Natarajan Date: Wed, 20 Feb 2019 15:04:32 -0500 Subject: [PATCH 515/648] issue_130_7 updated with review comments --- src/SUMMARY.md | 2 +- src/implementing_new_feature.md | 135 ------------------------ src/implementing_new_features.md | 171 +++++++++++++++++++++++++++++++ 3 files changed, 172 insertions(+), 136 deletions(-) delete mode 100644 src/implementing_new_feature.md create mode 100644 src/implementing_new_features.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 0761e734d..3ce74a4e5 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -14,7 +14,7 @@ - [Adding new tests](./tests/adding.md) - [Using `compiletest` + commands to control test execution](./compiletest.md) - [Walkthrough: a typical contribution](./walkthrough.md) -- [Implementing new features](./implementing_new_feature.md) +- [Implementing new features](./implementing_new_features.md) - [Stabilizing Features](./stabilization_guide.md) - [Debugging the Compiler](./compiler-debugging.md) - [Profiling the compiler](./profiling.md) diff --git a/src/implementing_new_feature.md b/src/implementing_new_feature.md deleted file mode 100644 index ff26b8550..000000000 --- a/src/implementing_new_feature.md +++ /dev/null @@ -1,135 +0,0 @@ -# Implement New Feature - -When you want to implement a new significant feature in the compiler, you need to go through this process to make sure everything goes smoothly. - -## The @rfcbot (p)FCP process - -When the change is small and uncontroversial, then it can be done -with just writing a PR and getting r+ from someone who knows that part of the code. However, if the change is potentially controversial, it would be a bad idea to push it without onsensus from the rest of the team (both in the "distributed system" sense to make sure you don't break anything you don't know about, and in the social sense to avoid PR fights). - -If such a change seems to small to require a full formal RFC process (e.g. a big refactoring of the code, or a "technically-breaking" change, or a "big bugfix" that basically amounts to a small feature) but is still too controversial or big to get by with a single r+, you can start a pFCP (or, if you don't have r+ rights, ask someone who has them to start one - and unless they have a concern themselves, they should). - -Again, the pFCP process is only needed if you need consensus - if you don't think anyone would have a problem with your change, it's ok to get by with only an r+. For example, it is OK to add or modify unstable command-line flags or attributes without an pFCP for compiler development or standard library use, as long as you don't expect them to be in wide use in the nightly ecosystem. - -Rust community is trying to find a better process for iterating on features on nightly, but if you are trying to write a feature that is not too bikesheddy, you could try to ask for an pFCP to land it on nightly under a feature gate. That might work out. However, some team-member might still tell you to go write an actual RFC (however, someone still needs to write an RFC before that feature is stabilized). - -You don't have to have the implementation fully ready for r+ to ask for a pFCP, but it is generally a good idea to have at least a proof of concept so that people can see what you are talking about. - -That starts a "proposed final comment period" (pFCP), which requires all members of the team to sign off the FCP. After they all do so, there's a week long "final comment period" where everybody can comment, and if no new concerns are raised, the PR/issue gets FCP approval. - -After an PR/issue had got FCP approval, team-members should help along attempts to implement it, and not block r+ on the controversy that prompted the FCP. If significant new issues arise after FCP approval, then it is ok to re-block the PR until these are resolved. - -For teams and team members: Even if someone is trying to paint the bikeshed your least favorite color, teams and team members should help them move along. Don't be a slow reviewer. - -## A few etiquette notes about (p)FCPs - -FCPs are a place where people unfamiliar with the Rust teams and process interact with the design process. This means that it is extra important for communication to be clear there. Don't assume the PR's writer knows when the next lang team meeting is. - -pFCP checkbox ticking is not supposed to be a place where PRs stall due to low priority - it is supposed to quickly either reach consensus or find a concern. If there is no real obstacle to consensus on the issue, the relevant team should sign off the PFCP reasonably quickly (a week or two). - -Teams: please review pFCP items during the team meetings to make sure this happens. If you find a real concern, please post a summary on the issue, even if you already discussed it on the team IRC channel, or worse, at a late-hours in-person meeting. - -If you think a pFCP item requires further design work which you are too busy to do right now, please say so on the issue/PR thread. This might appear rude, but it's less rude than leaving a PR accumulating bitrot for 2 months because you have no idea what to do with it. - -## How to technically run a pFCP - -pFCPs are run using the @rfcbot bot. For more details about how to correctly use rfcbot, see [the rfcbot guide], but the gist of it is that someone with privileges first tags your issue with a list of teams, and then someone writes this command in a comment in the issue thread: - -```rust, ignore - @rfcbot fcp merge -``` - -or - -```rust, ignore - @rfcbot fcp close -``` - -That will cause rfcbot to comment its ticky-box comment. - -## The logistics of writing features - -There are a few "logistic" hoops you might need to go through in order to implement a feature in a working way. -The more boring details are listed in the Rust repository's [CONTRIBUTING.md], so I'll just link to it here. You should try to at least skim it fully - it contains a bunch of nice tips. - -### Warning Cycles - -In some cases, a feature or bugfix might break some existing programs in some edge cases. In that case, you might want to do a crater run to assess the impact and possibly add a future-compatibility lint, similar to those used for [edition-gated lints](./diag.md#edition-gated-lints). - -### Stability - -We [value the stability of Rust]. Code that works and runs on stable should (mostly) not break. Because of that, we don't want to release a feature to the world with only team consensus and code review - we want to gain real-world experience on using that feature on nightly, and we might want to change the feature based on that experience. - -To allow for that, we must make sure users don't accidentally depend on that new feature - otherwise, especially if experimentation takes time or is delayed and the feature takes the trains to stable, it would end up de facto stable and we'll not be able to make changes in it without breaking people's code. - -The way we do that is that we make sure all new features are feature gated - they can't be used without a enabling a feature gate (#[feature(foo)]), which can't be done in a stable/beta compiler. See the [stability in code] section for the technical details. - -Eventually, after we gain enough experience using the feature, make the necessary changes, and are satisfied, we expose it to the world using the stabilization process described [here]. Until then, the feature is not set in stone: every part of the feature can be changed, or the feature might be completely rewritten or removed. Features are not supposed to gain tenure by being unstable and unchanged for a year. - -### Tracking Issues - -To keep track of the status of an unstable feature, the experience we get while using it on nightly, and of the concerns that block its stabilization, every feature-gate needs a tracking issue. - -General discussions about the feature should be done on the tracking issue. - -For features that have an RFC, you should use the RFC's tracking issue for the feature. - -For other features, you'll have to make a tracking issue for that feature. The issue title should be "Tracking issue for YOUR FEATURE" and it should have the `B-unstable` & `C-tracking-issue` tags, along with the tag for your subteam (e.g. `T-compiler` if this is a compiler feature). - -For tracking issues for features (as opposed to future-compat warnings), I don't think the description has to contain anything specific. Generally we put the list of items required for stabilization using a github list, e.g. - -```txt - **Steps:** - - - [ ] Implement the RFC (cc @rust-lang/compiler -- can anyone write up mentoring instructions?) - - [ ] Adjust documentation ([see instructions on forge][doc-guide]) - - Note: no stabilization step here. -``` - -## Stability in code - -The below steps needs to be followed in order to implement a new unstable feature: - -1. Open a [tracking issue] - if you have an RFC, you can use the tracking issue for the RFC. - -2. Pick a name for the feature gate (for RFCs, use the name in the RFC). - -3. Add a feature gate declaration to `libsyntax/feature_gate.rs` In the active `declare_features` block: - -```rust,ignore - // description of feature - (active, $feature_name, "$current_nightly_version", Some($tracking_issue_number), $edition) -``` - -where `$edition` has the type `Option`, and is typically just `None`. -For example: - -```rust,ignore - // allow '|' at beginning of match arms (RFC 1925) -( active, match_beginning_vert, "1.21.0", Some(44101), None), -``` - -The current version is not actually important – the important version is when you are stabilizing a feature. - -4. Prevent usage of the new feature unless the feature gate is set. - - You can check it in most places in the compiler using the expression `tcx.features().$feature_name` (or `sess.features_untracked().borrow().$feature_name` if the tcx is unavailable) - - If the feature gate is not set, you should either maintain the pre-feature behavior or raise an error, depending on what makes sense. - -5. Add a test to ensure the feature cannot be used without a feature gate, by creating `feature-gate-$feature_name.rs` and `feature-gate-$feature_name.stderr` files under the `src/test/ui/feature-gates` directory. - -6. Add a section to the unstable book, in `src/doc/unstable-book/src/language-features/$feature_name.md`. - -7. Write a lots of tests for the new feature. PRs without tests will not be accepted! - -8. Get your PR reviewed and land it. You have now successfully implemented a feature in Rust! - -[the rfcbot guide]: https://github.com/anp/rfcbot-rs/blob/master/CONDUCT.md -[CONTRIBUTING.md]: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md -[rustc bug-fix procedure]: -https://github.com/rust-lang/rust-forge/blob/master/rustc-bug-fix-procedure.md -[value the stability of Rust]: https://github.com/rust-lang/rfcs/blob/master/text/1122-language-semver.md -[stability in code]: -[here]: https://rust-lang.github.io/rustc-guide/stabilization_guide.html -[tracking issue]: diff --git a/src/implementing_new_features.md b/src/implementing_new_features.md new file mode 100644 index 000000000..f1d501ce2 --- /dev/null +++ b/src/implementing_new_features.md @@ -0,0 +1,171 @@ +# Implement New Feature + +When you want to implement a new significant feature in the compiler, +you need to go through this process to make sure everything goes +smoothly. + +## The @rfcbot (p)FCP process + +When the change is small and uncontroversial, then it can be done +with just writing a PR and getting r+ from someone who knows that +part of the code. However, if the change is potentially controversial, +it would be a bad idea to push it without consensus from the rest +of the team (both in the "distributed system" sense to make sure +you don't break anything you don't know about, and in the social +sense to avoid PR fights). + +If such a change seems to be too small to require a full formal RFC +process (e.g. a big refactoring of the code, or a +"technically-breaking" change, or a "big bugfix" that basically +amounts to a small feature) but is still too controversial or +big to get by with a single r+, you can start a pFCP (or, if you +don't have r+ rights, ask someone who has them to start one - and +unless they have a concern themselves, they should). + +Again, the pFCP process is only needed if you need consensus - if you +don't think anyone would have a problem with your change, it's ok to +get by with only an r+. For example, it is OK to add or modify +unstable command-line flags or attributes without an pFCP for +compiler development or standard library use, as long as you don't +expect them to be in wide use in the nightly ecosystem. + +You don't need to have the implementation fully ready for r+ to ask +for a pFCP, but it is generally a good idea to have at least a proof +of concept so that people can see what you are talking about. + +That starts a "proposed final comment period" (pFCP), which requires +all members of the team to sign off the FCP. After they all do so, +there's a week long "final comment period" where everybody can comment, +and if no new concerns are raised, the PR/issue gets FCP approval. + +## The logistics of writing features + +There are a few "logistic" hoops you might need to go through in +order to implement a feature in a working way. + +### Warning Cycles + +In some cases, a feature or bugfix might break some existing programs +in some edge cases. In that case, you might want to do a crater run +to assess the impact and possibly add a future-compatibility lint, +similar to those used for +[edition-gated lints](./diag.md#edition-gated-lints). + +### Stability + +We [value the stability of Rust]. Code that works and runs on stable +should (mostly) not break. Because of that, we don't want to release +a feature to the world with only team consensus and code review - +we want to gain real-world experience on using that feature on nightly, +and we might want to change the feature based on that experience. + +To allow for that, we must make sure users don't accidentally depend +on that new feature - otherwise, especially if experimentation takes +time or is delayed and the feature takes the trains to stable, +it would end up de facto stable and we'll not be able to make changes +in it without breaking people's code. + +The way we do that is that we make sure all new features are feature +gated - they can't be used without a enabling a feature gate +`(#[feature(foo)])`, which can't be done in a stable/beta compiler. +See the [stability in code] section for the technical details. + +Eventually, after we gain enough experience using the feature, +make the necessary changes, and are satisfied, we expose it to +the world using the stabilization process described [here]. +Until then, the feature is not set in stone: every part of the +feature can be changed, or the feature might be completely +rewritten or removed. Features are not supposed to gain tenure +by being unstable and unchanged for a year. + +### Tracking Issues + +To keep track of the status of an unstable feature, the +experience we get while using it on nightly, and of the +concerns that block its stabilization, every feature-gate +needs a tracking issue. + +General discussions about the feature should be done on +the tracking issue. + +For features that have an RFC, you should use the RFC's +tracking issue for the feature. + +For other features, you'll have to make a tracking issue +for that feature. The issue title should be "Tracking issue +for YOUR FEATURE". + +For tracking issues for features (as opposed to future-compat +warnings), I don't think the description has to contain +anything specific. Generally we put the list of items required +for stabilization using a github list, e.g. + +```txt + **Steps:** + + - [ ] Implement the RFC (cc @rust-lang/compiler -- can anyone write + up mentoring instructions?) + - [ ] Adjust documentation ([see instructions on forge][doc-guide]) + - Note: no stabilization step here. +``` + +## Stability in code + +The below steps needs to be followed in order to implement +a new unstable feature: + +1. Open a [tracking issue] - + if you have an RFC, you can use the tracking issue for the RFC. + +2. Pick a name for the feature gate (for RFCs, use the name + in the RFC). + +3. Add a feature gate declaration to `libsyntax/feature_gate.rs` + in the active `declare_features` block: + +```rust,ignore + // description of feature + (active, $feature_name, "$current_nightly_version", Some($tracking_issue_number), $edition) +``` + +where `$edition` has the type `Option`, and is typically +just `None`. + +For example: + +```rust,ignore + // allow '|' at beginning of match arms (RFC 1925) +( active, match_beginning_vert, "1.21.0", Some(44101), None), +``` + +The current version is not actually important – the important +version is when you are stabilizing a feature. + +4. Prevent usage of the new feature unless the feature gate is set. + You can check it in most places in the compiler using the + expression `tcx.features().$feature_name` (or + `sess.features_untracked().borrow().$feature_name` if the + tcx is unavailable) + + If the feature gate is not set, you should either maintain + the pre-feature behavior or raise an error, depending on + what makes sense. + +5. Add a test to ensure the feature cannot be used without + a feature gate, by creating `feature-gate-$feature_name.rs` + and `feature-gate-$feature_name.stderr` files under the + `src/test/ui/feature-gates` directory. + +6. Add a section to the unstable book, in + `src/doc/unstable-book/src/language-features/$feature_name.md`. + +7. Write a lots of tests for the new feature. + PRs without tests will not be accepted! + +8. Get your PR reviewed and land it. You have now successfully + implemented a feature in Rust! + +[value the stability of Rust]: https://github.com/rust-lang/rfcs/blob/master/text/1122-language-semver.md +[stability in code]: #stability-in-code +[here]: https://rust-lang.github.io/rustc-guide/stabilization_guide.html +[tracking issue]: #tracking-issue From 46d99ee25ae6fe1ca6a6866ac9fa262ca5e94873 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Wed, 20 Feb 2019 18:02:35 -0500 Subject: [PATCH 516/648] Update src/implementing_new_features.md Co-Authored-By: rajcspsg --- src/implementing_new_features.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/implementing_new_features.md b/src/implementing_new_features.md index f1d501ce2..975d10ef8 100644 --- a/src/implementing_new_features.md +++ b/src/implementing_new_features.md @@ -78,7 +78,8 @@ feature can be changed, or the feature might be completely rewritten or removed. Features are not supposed to gain tenure by being unstable and unchanged for a year. -### Tracking Issues + +### Tracking Issues To keep track of the status of an unstable feature, the experience we get while using it on nightly, and of the From 41dee19fd3c42e11224928e8db18da6f24893cf9 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Wed, 20 Feb 2019 18:02:51 -0500 Subject: [PATCH 517/648] Update src/implementing_new_features.md Co-Authored-By: rajcspsg --- src/implementing_new_features.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/implementing_new_features.md b/src/implementing_new_features.md index 975d10ef8..44c1ce510 100644 --- a/src/implementing_new_features.md +++ b/src/implementing_new_features.md @@ -110,7 +110,8 @@ for stabilization using a github list, e.g. - Note: no stabilization step here. ``` -## Stability in code + +## Stability in code The below steps needs to be followed in order to implement a new unstable feature: From ed6f17eb43a8552526f23d01fd2a489d6943f1b3 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Wed, 20 Feb 2019 18:03:01 -0500 Subject: [PATCH 518/648] Update src/implementing_new_features.md Co-Authored-By: rajcspsg --- src/implementing_new_features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/implementing_new_features.md b/src/implementing_new_features.md index 44c1ce510..650d9915c 100644 --- a/src/implementing_new_features.md +++ b/src/implementing_new_features.md @@ -67,7 +67,7 @@ in it without breaking people's code. The way we do that is that we make sure all new features are feature gated - they can't be used without a enabling a feature gate -`(#[feature(foo)])`, which can't be done in a stable/beta compiler. +(`#[feature(foo)]`), which can't be done in a stable/beta compiler. See the [stability in code] section for the technical details. Eventually, after we gain enough experience using the feature, From 2ebab6532b9f5c3522d907a853672b2442d23502 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 23 Jan 2019 12:35:21 -0600 Subject: [PATCH 519/648] Organize and finish debugging chapters --- src/SUMMARY.md | 3 + src/codegen/debugging.md | 122 +++++++++++++++++++++++++++++ src/compiler-debugging.md | 161 +++++++------------------------------- src/hir-debugging.md | 8 ++ src/mir/debugging.md | 81 +++++++++++++++++++ src/mir/passes.md | 76 ------------------ 6 files changed, 241 insertions(+), 210 deletions(-) create mode 100644 src/codegen/debugging.md create mode 100644 src/hir-debugging.md create mode 100644 src/mir/debugging.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 3ce74a4e5..dc2f46bd9 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -38,6 +38,7 @@ - [Name resolution](./name-resolution.md) - [The HIR (High-level IR)](./hir.md) - [Lowering AST to HIR](./lowering.md) + - [Debugging](./hir-debugging.md) - [The `ty` module: representing types](./ty.md) - [Kinds](./kinds.md) - [Type inference](./type-inference.md) @@ -68,6 +69,7 @@ - [MIR visitor and traversal](./mir/visitor.md) - [MIR passes: getting the MIR for a function](./mir/passes.md) - [MIR optimizations](./mir/optimizations.md) + - [Debugging](./mir/debugging.md) - [The borrow checker](./borrow_check.md) - [Tracking moves and initialization](./borrow_check/moves_and_initialization.md) - [Move paths](./borrow_check/moves_and_initialization/move_paths.md) @@ -78,6 +80,7 @@ - [Parameter Environments](./param_env.md) - [Code Generation](./codegen.md) - [Updating LLVM](./codegen/updating-llvm.md) + - [Debugging LLVM](./codegen/debugging.md) - [Emitting Diagnostics](./diag.md) --- diff --git a/src/codegen/debugging.md b/src/codegen/debugging.md new file mode 100644 index 000000000..acf917bd0 --- /dev/null +++ b/src/codegen/debugging.md @@ -0,0 +1,122 @@ +## Debugging LLVM + +> NOTE: If you are looking for info about code generation, please see [this +> chapter][codegen] instead. + +[codegen]: codegen.html + +This section is about debugging compiler bugs in code generation (e.g. why the +compiler generated some piece of code or crashed in LLVM). LLVM is a big +project on its own that probably needs to have its own debugging document (not +that I could find one). But here are some tips that are important in a rustc +context: + +As a general rule, compilers generate lots of information from analyzing code. +Thus, a useful first step is usually to find a minimal example. One way to do +this is to + +1. create a new crate that reproduces the issue (e.g. adding whatever crate is +at fault as a dependency, and using it from there) + +2. minimize the crate by removing external dependencies; that is, moving +everything relevant to the new crate + +3. further minimize the issue by making the code shorter (there are tools that +help with this like `creduce`) + +The official compilers (including nightlies) have LLVM assertions disabled, +which means that LLVM assertion failures can show up as compiler crashes (not +ICEs but "real" crashes) and other sorts of weird behavior. If you are +encountering these, it is a good idea to try using a compiler with LLVM +assertions enabled - either an "alt" nightly or a compiler you build yourself +by setting `[llvm] assertions=true` in your config.toml - and see whether +anything turns up. + +The rustc build process builds the LLVM tools into +`./build//llvm/bin`. They can be called directly. + +The default rustc compilation pipeline has multiple codegen units, which is +hard to replicate manually and means that LLVM is called multiple times in +parallel. If you can get away with it (i.e. if it doesn't make your bug +disappear), passing `-C codegen-units=1` to rustc will make debugging easier. + +To rustc to generate LLVM IR, you need to pass the `--emit=llvm-ir` flag. If +you are building via cargo, use the `RUSTFLAGS` environment variable (e.g. +`RUSTFLAGS='--emit=llvm-ir'`). This causes rustc to spit out LLVM IR into the +target directory. + +`cargo llvm-ir [options] path` spits out the LLVM IR for a particular function +at `path`. (`cargo install cargo-asm` installs `cargo asm` and `cargo +llvm-ir`). `--build-type=debug` emits code for debug builds. There are also +other useful options. Also, debug info in LLVM IR can clutter the output a lot: +`RUSTFLAGS="-C debuginfo=0"` is really useful. + +`RUSTFLAGS="-C save-temps"` outputs LLVM bitcode (not the same as IR) at +different stages during compilation, which is sometimes useful. One just needs +to convert the bitcode files to `.ll` files using `llvm-dis` which should be in +the target local compilation of rustc. + +If you want to play with the optimization pipeline, you can use the `opt` tool +from `./build//llvm/bin/` with the LLVM IR emitted by rustc. Note +that rustc emits different IR depending on whether `-O` is enabled, even +without LLVM's optimizations, so if you want to play with the IR rustc emits, +you should: + +```bash +$ rustc +local my-file.rs --emit=llvm-ir -O -C no-prepopulate-passes \ + -C codegen-units=1 +$ OPT=./build/$TRIPLE/llvm/bin/opt +$ $OPT -S -O2 < my-file.ll > my +``` + +If you just want to get the LLVM IR during the LLVM pipeline, to e.g. see which +IR causes an optimization-time assertion to fail, or to see when LLVM performs +a particular optimization, you can pass the rustc flag `-C +llvm-args=-print-after-all`, and possibly add `-C +llvm-args='-filter-print-funcs=EXACT_FUNCTION_NAME` (e.g. `-C +llvm-args='-filter-print-funcs=_ZN11collections3str21_$LT$impl$u20$str$GT$\ +7replace17hbe10ea2e7c809b0bE'`). + +That produces a lot of output into standard error, so you'll want to pipe that +to some file. Also, if you are using neither `-filter-print-funcs` nor `-C +codegen-units=1`, then, because the multiple codegen units run in parallel, the +printouts will mix together and you won't be able to read anything. + +If you want just the IR for a specific function (say, you want to see why it +causes an assertion or doesn't optimize correctly), you can use `llvm-extract`, +e.g. + +```bash +$ ./build/$TRIPLE/llvm/bin/llvm-extract \ + -func='_ZN11collections3str21_$LT$impl$u20$str$GT$7replace17hbe10ea2e7c809b0bE' \ + -S \ + < unextracted.ll \ + > extracted.ll +``` + +### Filing LLVM bug reports + +When filing an LLVM bug report, you will probably want some sort of minimal +working example that demonstrates the problem. The Godbolt compiler explorer is +really helpful for this. + +1. Once you have some LLVM IR for the problematic code (see above), you can +create a minimal working example with Godbolt. Go to +[gcc.godbolt.org](https://gcc.godbolt.org). + +2. Choose `LLVM-IR` as programming language. + +3. Use `llc` to compile the IR to a particular target as is: + - There are some useful flags: `-mattr` enables target features, `-march=` + selects the target, `-mcpu=` selects the CPU, etc. + - Commands like `llc -march=help` output all architectures available, which + is useful because sometimes the Rust arch names and the LLVM names do not + match. + - If you have compiled rustc yourself somewhere, in the target directory + you have binaries for `llc`, `opt`, etc. + +4. If you want to optimize the LLVM-IR, you can use `opt` to see how the LLVM + optimizations transform it. + +5. Once you have a godbolt link demonstrating the issue, it is pretty easy to + fill in an LLVM bug. diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index 31f221057..714385ca0 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -1,7 +1,21 @@ # Debugging the compiler [debugging]: #debugging -Here are a few tips to debug the compiler: +This chapter contains a few tips to debug the compiler. These tips aim to be +useful no matter what you are working on. Some of the other chapters have +advice about specific parts of the compiler (e.g. the [Queries Debugging and +Testing +chapter](./incrcomp-debugging.html) or +the [LLVM Debugging chapter](./codegen/debugging.md)). + +## `-Z` flags + +The compiler has a bunch of `-Z` flags. These are unstable flags that are only +enabled on nightly. Many of them are useful for debugging. To get a full listing +of `-Z` flags, use `-Z help`. + +One useful flag is `-Z verbose`, which generally enables printing more info that +could be useful for debugging. ## Getting a backtrace [getting-a-backtrace]: #getting-a-backtrace @@ -135,6 +149,9 @@ These crates are used in compiler for logging: * [log] * [env-logger]: check the link to see the full `RUST_LOG` syntax +[log]: https://docs.rs/log/0.4.6/log/index.html +[env-logger]: https://docs.rs/env_logger/0.4.3/env_logger/ + The compiler has a lot of `debug!` calls, which print out logging information at many points. These are very useful to at least narrow down the location of a bug if not to find it entirely, or just to orient yourself as to why the @@ -189,7 +206,7 @@ I also think that in some cases just setting it will not trigger a rebuild, so if you changed it and you already have a compiler built, you might want to call `x.py clean` to force one. -### Logging etiquette +### Logging etiquette and conventions Because calls to `debug!` are removed by default, in most cases, don't worry about adding "unnecessary" calls to `debug!` and leaving them in code you @@ -197,9 +214,12 @@ commit - they won't slow down the performance of what we ship, and if they helped you pinning down a bug, they will probably help someone else with a different one. -However, there are still a few concerns that you might care about: +A loosely followed convention is to use `debug!("foo(...)")` at the _start_ of +a function `foo` and `debug!("foo: ...")` _within_ the function. Another +loosely followed convention is to use the `{:?}` format specifier for debug +logs. -### Expensive operations in logs +One thing to be **careful** of is **expensive** operations in logs. If in the module `rustc::foo` you have a statement @@ -210,9 +230,9 @@ debug!("{:?}", random_operation(tcx)); Then if someone runs a debug `rustc` with `RUST_LOG=rustc::bar`, then `random_operation()` will run. -This means that you should not put anything too expensive or likely -to crash there - that would annoy anyone who wants to use logging for their own -module. No-one will know it until someone tries to use logging to find *another* bug. +This means that you should not put anything too expensive or likely to crash +there - that would annoy anyone who wants to use logging for their own module. +No-one will know it until someone tries to use logging to find *another* bug. ## Formatting Graphviz output (.dot files) [formatting-graphviz-output]: #formatting-graphviz-output @@ -229,133 +249,6 @@ $ dot -T pdf maybe_init_suffix.dot > maybe_init_suffix.pdf $ firefox maybe_init_suffix.pdf # Or your favorite pdf viewer ``` -## Debugging LLVM -[debugging-llvm]: #debugging-llvm - -> NOTE: If you are looking for info about code generation, please see [this -> chapter][codegen] instead. - -[codegen]: codegen.html - -This section is about debugging compiler bugs in code generation (e.g. why the -compiler generated some piece of code or crashed in LLVM). LLVM is a big -project on its own that probably needs to have its own debugging document (not -that I could find one). But here are some tips that are important in a rustc -context: - -As a general rule, compilers generate lots of information from analyzing code. -Thus, a useful first step is usually to find a minimal example. One way to do -this is to - -1. create a new crate that reproduces the issue (e.g. adding whatever crate is -at fault as a dependency, and using it from there) - -2. minimize the crate by removing external dependencies; that is, moving -everything relevant to the new crate - -3. further minimize the issue by making the code shorter (there are tools that -help with this like `creduce`) - -The official compilers (including nightlies) have LLVM assertions disabled, -which means that LLVM assertion failures can show up as compiler crashes (not -ICEs but "real" crashes) and other sorts of weird behavior. If you are -encountering these, it is a good idea to try using a compiler with LLVM -assertions enabled - either an "alt" nightly or a compiler you build yourself -by setting `[llvm] assertions=true` in your config.toml - and see whether -anything turns up. - -The rustc build process builds the LLVM tools into -`./build//llvm/bin`. They can be called directly. - -The default rustc compilation pipeline has multiple codegen units, which is -hard to replicate manually and means that LLVM is called multiple times in -parallel. If you can get away with it (i.e. if it doesn't make your bug -disappear), passing `-C codegen-units=1` to rustc will make debugging easier. - -To rustc to generate LLVM IR, you need to pass the `--emit=llvm-ir` flag. If -you are building via cargo, use the `RUSTFLAGS` environment variable (e.g. -`RUSTFLAGS='--emit=llvm-ir'`). This causes rustc to spit out LLVM IR into the -target directory. - -`cargo llvm-ir [options] path` spits out the LLVM IR for a particular function -at `path`. (`cargo install cargo-asm` installs `cargo asm` and `cargo -llvm-ir`). `--build-type=debug` emits code for debug builds. There are also -other useful options. Also, debug info in LLVM IR can clutter the output a lot: -`RUSTFLAGS="-C debuginfo=0"` is really useful. - -`RUSTFLAGS="-C save-temps"` outputs LLVM bitcode (not the same as IR) at -different stages during compilation, which is sometimes useful. One just needs -to convert the bitcode files to `.ll` files using `llvm-dis` which should be in -the target local compilation of rustc. - -If you want to play with the optimization pipeline, you can use the `opt` tool -from `./build//llvm/bin/` with the LLVM IR emitted by rustc. Note -that rustc emits different IR depending on whether `-O` is enabled, even -without LLVM's optimizations, so if you want to play with the IR rustc emits, -you should: - -```bash -$ rustc +local my-file.rs --emit=llvm-ir -O -C no-prepopulate-passes \ - -C codegen-units=1 -$ OPT=./build/$TRIPLE/llvm/bin/opt -$ $OPT -S -O2 < my-file.ll > my -``` - -If you just want to get the LLVM IR during the LLVM pipeline, to e.g. see which -IR causes an optimization-time assertion to fail, or to see when LLVM performs -a particular optimization, you can pass the rustc flag `-C -llvm-args=-print-after-all`, and possibly add `-C -llvm-args='-filter-print-funcs=EXACT_FUNCTION_NAME` (e.g. `-C -llvm-args='-filter-print-funcs=_ZN11collections3str21_$LT$impl$u20$str$GT$\ -7replace17hbe10ea2e7c809b0bE'`). - -That produces a lot of output into standard error, so you'll want to pipe that -to some file. Also, if you are using neither `-filter-print-funcs` nor `-C -codegen-units=1`, then, because the multiple codegen units run in parallel, the -printouts will mix together and you won't be able to read anything. - -If you want just the IR for a specific function (say, you want to see why it -causes an assertion or doesn't optimize correctly), you can use `llvm-extract`, -e.g. - -```bash -$ ./build/$TRIPLE/llvm/bin/llvm-extract \ - -func='_ZN11collections3str21_$LT$impl$u20$str$GT$7replace17hbe10ea2e7c809b0bE' \ - -S \ - < unextracted.ll \ - > extracted.ll -``` - -### Filing LLVM bug reports - -When filing an LLVM bug report, you will probably want some sort of minimal -working example that demonstrates the problem. The Godbolt compiler explorer is -really helpful for this. - -1. Once you have some LLVM IR for the problematic code (see above), you can -create a minimal working example with Godbolt. Go to -[gcc.godbolt.org](https://gcc.godbolt.org). - -2. Choose `LLVM-IR` as programming language. - -3. Use `llc` to compile the IR to a particular target as is: - - There are some useful flags: `-mattr` enables target features, `-march=` - selects the target, `-mcpu=` selects the CPU, etc. - - Commands like `llc -march=help` output all architectures available, which - is useful because sometimes the Rust arch names and the LLVM names do not - match. - - If you have compiled rustc yourself somewhere, in the target directory - you have binaries for `llc`, `opt`, etc. - -4. If you want to optimize the LLVM-IR, you can use `opt` to see how the LLVM - optimizations transform it. - -5. Once you have a godbolt link demonstrating the issue, it is pretty easy to - fill in an LLVM bug. - -[log]: https://docs.rs/log/0.4.6/log/index.html -[env-logger]: https://docs.rs/env_logger/0.4.3/env_logger/ - ## Narrowing (Bisecting) Regressions The [cargo-bisect-rustc][bisect] tool can be used as a quick and easy way to diff --git a/src/hir-debugging.md b/src/hir-debugging.md new file mode 100644 index 000000000..bb7f541fd --- /dev/null +++ b/src/hir-debugging.md @@ -0,0 +1,8 @@ +# HIR Debugging + +The `-Zunpretty=hir-tree` flag will dump out the HIR. + +If you are trying to correlate `NodeId`s or `DefId`s with source code, the +`--pretty expanded,identified` flag may be useful. + +TODO: anything else? diff --git a/src/mir/debugging.md b/src/mir/debugging.md new file mode 100644 index 000000000..93a79076c --- /dev/null +++ b/src/mir/debugging.md @@ -0,0 +1,81 @@ +# MIR Debugging + +The `-Zdump-mir` flag can be used to dump a text representation of the MIR. The +`-Zdump-mir-graphviz` flag can be used to dump a `.dot` file that represents +MIR as a control-flow graph. + +`-Zdump-mir=F` is a handy compiler options that will let you view the MIR for +each function at each stage of compilation. `-Zdump-mir` takes a **filter** `F` +which allows you to control which functions and which passes you are +interesting in. For example: + +```bash +> rustc -Zdump-mir=foo ... +``` + +This will dump the MIR for any function whose name contains `foo`; it +will dump the MIR both before and after every pass. Those files will +be created in the `mir_dump` directory. There will likely be quite a +lot of them! + +```bash +> cat > foo.rs +fn main() { + println!("Hello, world!"); +} +^D +> rustc -Zdump-mir=main foo.rs +> ls mir_dump/* | wc -l + 161 +``` + +The files have names like `rustc.main.000-000.CleanEndRegions.after.mir`. These +names have a number of parts: + +```text +rustc.main.000-000.CleanEndRegions.after.mir + ---- --- --- --------------- ----- either before or after + | | | name of the pass + | | index of dump within the pass (usually 0, but some passes dump intermediate states) + | index of the pass + def-path to the function etc being dumped +``` + +You can also make more selective filters. For example, `main & CleanEndRegions` +will select for things that reference *both* `main` and the pass +`CleanEndRegions`: + +```bash +> rustc -Zdump-mir='main & CleanEndRegions' foo.rs +> ls mir_dump +rustc.main.000-000.CleanEndRegions.after.mir rustc.main.000-000.CleanEndRegions.before.mir +``` + +Filters can also have `|` parts to combine multiple sets of +`&`-filters. For example `main & CleanEndRegions | main & +NoLandingPads` will select *either* `main` and `CleanEndRegions` *or* +`main` and `NoLandingPads`: + +```bash +> rustc -Zdump-mir='main & CleanEndRegions | main & NoLandingPads' foo.rs +> ls mir_dump +rustc.main-promoted[0].002-000.NoLandingPads.after.mir +rustc.main-promoted[0].002-000.NoLandingPads.before.mir +rustc.main-promoted[0].002-006.NoLandingPads.after.mir +rustc.main-promoted[0].002-006.NoLandingPads.before.mir +rustc.main-promoted[1].002-000.NoLandingPads.after.mir +rustc.main-promoted[1].002-000.NoLandingPads.before.mir +rustc.main-promoted[1].002-006.NoLandingPads.after.mir +rustc.main-promoted[1].002-006.NoLandingPads.before.mir +rustc.main.000-000.CleanEndRegions.after.mir +rustc.main.000-000.CleanEndRegions.before.mir +rustc.main.002-000.NoLandingPads.after.mir +rustc.main.002-000.NoLandingPads.before.mir +rustc.main.002-006.NoLandingPads.after.mir +rustc.main.002-006.NoLandingPads.before.mir +``` + +(Here, the `main-promoted[0]` files refer to the MIR for "promoted constants" +that appeared within the `main` function.) + +TODO: anything else? diff --git a/src/mir/passes.md b/src/mir/passes.md index 7dc1249a0..a0de3ae0e 100644 --- a/src/mir/passes.md +++ b/src/mir/passes.md @@ -22,82 +22,6 @@ where we want to access the MIR for type checking or other purposes: - `optimized_mir(D)` – the final state, after all optimizations have been performed. -### Seeing how the MIR changes as the compiler executes - -`-Zdump-mir=F` is a handy compiler options that will let you view the MIR for -each function at each stage of compilation. `-Zdump-mir` takes a **filter** `F` -which allows you to control which functions and which passes you are -interesting in. For example: - -```bash -> rustc -Zdump-mir=foo ... -``` - -This will dump the MIR for any function whose name contains `foo`; it -will dump the MIR both before and after every pass. Those files will -be created in the `mir_dump` directory. There will likely be quite a -lot of them! - -```bash -> cat > foo.rs -fn main() { - println!("Hello, world!"); -} -^D -> rustc -Zdump-mir=main foo.rs -> ls mir_dump/* | wc -l - 161 -``` - -The files have names like `rustc.main.000-000.CleanEndRegions.after.mir`. These -names have a number of parts: - -```text -rustc.main.000-000.CleanEndRegions.after.mir - ---- --- --- --------------- ----- either before or after - | | | name of the pass - | | index of dump within the pass (usually 0, but some passes dump intermediate states) - | index of the pass - def-path to the function etc being dumped -``` - -You can also make more selective filters. For example, `main & CleanEndRegions` -will select for things that reference *both* `main` and the pass -`CleanEndRegions`: - -```bash -> rustc -Zdump-mir='main & CleanEndRegions' foo.rs -> ls mir_dump -rustc.main.000-000.CleanEndRegions.after.mir rustc.main.000-000.CleanEndRegions.before.mir -``` - -Filters can also have `|` parts to combine multiple sets of -`&`-filters. For example `main & CleanEndRegions | main & -NoLandingPads` will select *either* `main` and `CleanEndRegions` *or* -`main` and `NoLandingPads`: - -```bash -> rustc -Zdump-mir='main & CleanEndRegions | main & NoLandingPads' foo.rs -> ls mir_dump -rustc.main-promoted[0].002-000.NoLandingPads.after.mir -rustc.main-promoted[0].002-000.NoLandingPads.before.mir -rustc.main-promoted[0].002-006.NoLandingPads.after.mir -rustc.main-promoted[0].002-006.NoLandingPads.before.mir -rustc.main-promoted[1].002-000.NoLandingPads.after.mir -rustc.main-promoted[1].002-000.NoLandingPads.before.mir -rustc.main-promoted[1].002-006.NoLandingPads.after.mir -rustc.main-promoted[1].002-006.NoLandingPads.before.mir -rustc.main.000-000.CleanEndRegions.after.mir -rustc.main.000-000.CleanEndRegions.before.mir -rustc.main.002-000.NoLandingPads.after.mir -rustc.main.002-000.NoLandingPads.before.mir -rustc.main.002-006.NoLandingPads.after.mir -rustc.main.002-006.NoLandingPads.before.mir -``` - -(Here, the `main-promoted[0]` files refer to the MIR for "promoted constants" -that appeared within the `main` function.) - ### Implementing and registering a pass A `MirPass` is some bit of code that processes the MIR, typically – From fea9c2361fa7b25a9da9cb5030539e7ecf5abf66 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 23 Jan 2019 12:59:34 -0600 Subject: [PATCH 520/648] fix link --- src/codegen/debugging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/codegen/debugging.md b/src/codegen/debugging.md index acf917bd0..51d5a3b2c 100644 --- a/src/codegen/debugging.md +++ b/src/codegen/debugging.md @@ -3,7 +3,7 @@ > NOTE: If you are looking for info about code generation, please see [this > chapter][codegen] instead. -[codegen]: codegen.html +[codegen]: ../codegen.md This section is about debugging compiler bugs in code generation (e.g. why the compiler generated some piece of code or crashed in LLVM). LLVM is a big From 0456aaa9e197e6d3f8349bca6299becb836e4070 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 1 Mar 2019 17:02:54 -0600 Subject: [PATCH 521/648] update mdbook --- ci/install.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install.sh b/ci/install.sh index 947d751b4..835f23192 100755 --- a/ci/install.sh +++ b/ci/install.sh @@ -20,5 +20,5 @@ function cargo_install() { fi } -cargo_install mdbook 0.2.2 +cargo_install mdbook 0.2.3 cargo_install mdbook-linkcheck 0.2.3 From 4013f453e51a6cf4bf83531f583f048c9cc7e6dc Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 15 Mar 2019 17:24:46 -0500 Subject: [PATCH 522/648] chalk has moved to rust-lang --- src/traits/chalk-overview.md | 64 ++++++++++++++++++------------------ src/traits/slg.md | 8 ++--- 2 files changed, 36 insertions(+), 36 deletions(-) diff --git a/src/traits/chalk-overview.md b/src/traits/chalk-overview.md index 59572479d..54046b22e 100644 --- a/src/traits/chalk-overview.md +++ b/src/traits/chalk-overview.md @@ -172,7 +172,7 @@ Chalk's functionality is broken up into the following crates: - `coherence`, which implements coherence rules - Also includes [chalki][chalki], chalk's REPL. -[Browse source code on GitHub](https://github.com/rust-lang-nursery/chalk) +[Browse source code on GitHub](https://github.com/rust-lang/chalk) ## Testing @@ -202,8 +202,8 @@ Likewise, lowering tests use the [`lowering_success!` and ## More Resources -* [Chalk Source Code](https://github.com/rust-lang-nursery/chalk) -* [Chalk Glossary](https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md) +* [Chalk Source Code](https://github.com/rust-lang/chalk) +* [Chalk Glossary](https://github.com/rust-lang/chalk/blob/master/GLOSSARY.md) ### Blog Posts @@ -224,34 +224,34 @@ Likewise, lowering tests use the [`lowering_success!` and [wf-checking]: ./wf.html [ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree -[chalk]: https://github.com/rust-lang-nursery/chalk -[rustc-issues]: https://github.com/rust-lang-nursery/rustc-guide/issues +[chalk]: https://github.com/rust-lang/chalk +[rustc-issues]: https://github.com/rust-lang/rustc-guide/issues [universal quantification]: https://en.wikipedia.org/wiki/Universal_quantification -[`ProgramClause`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/enum.ProgramClause.html -[`ProgramEnvironment`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/struct.ProgramEnvironment.html -[chalk_engine]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/index.html -[chalk_ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk_ir/index.html -[chalk_parse]: https://rust-lang-nursery.github.io/chalk/doc/chalk_parse/index.html -[chalk_solve]: https://rust-lang-nursery.github.io/chalk/doc/chalk_solve/index.html -[doc-chalk]: https://rust-lang-nursery.github.io/chalk/doc/chalk/index.html -[engine-context]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/context/index.html -[rust_ir-program]: https://rust-lang-nursery.github.io/chalk/doc/chalk/rust_ir/struct.Program.html -[rust_ir]: https://rust-lang-nursery.github.io/chalk/doc/chalk/rust_ir/index.html - -[binders-struct]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L661 -[chalk-ast]: https://github.com/rust-lang-nursery/chalk/blob/master/chalk-parse/src/ast.rs -[chalk-test-example]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L115 -[chalk-test-lowering-example]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rust_ir/lowering/test.rs#L8-L31 -[chalk-test-lowering]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rust_ir/lowering/test.rs -[chalk-test-wf]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules/wf/test.rs#L1 -[chalki]: https://rust-lang-nursery.github.io/chalk/doc/chalki/index.html -[clause]: https://github.com/rust-lang-nursery/chalk/blob/master/GLOSSARY.md#clause -[coherence-src]: https://github.com/rust-lang-nursery/chalk/blob/master/src/coherence.rs -[ir-code]: https://github.com/rust-lang-nursery/chalk/blob/master/src/rust_ir.rs -[rules-environment]: https://github.com/rust-lang-nursery/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/rules.rs#L9 -[rules-src]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules.rs -[rules-wf-src]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules/wf.rs -[solve_goal]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L85 -[test-lowering-macros]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test_util.rs#L21-L54 -[test-macro]: https://github.com/rust-lang-nursery/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L33 +[`ProgramClause`]: https://rust-lang.github.io/chalk/doc/chalk_ir/enum.ProgramClause.html +[`ProgramEnvironment`]: https://rust-lang.github.io/chalk/doc/chalk_ir/struct.ProgramEnvironment.html +[chalk_engine]: https://rust-lang.github.io/chalk/doc/chalk_engine/index.html +[chalk_ir]: https://rust-lang.github.io/chalk/doc/chalk_ir/index.html +[chalk_parse]: https://rust-lang.github.io/chalk/doc/chalk_parse/index.html +[chalk_solve]: https://rust-lang.github.io/chalk/doc/chalk_solve/index.html +[doc-chalk]: https://rust-lang.github.io/chalk/doc/chalk/index.html +[engine-context]: https://rust-lang.github.io/chalk/doc/chalk_engine/context/index.html +[rust_ir-program]: https://rust-lang.github.io/chalk/doc/chalk/rust_ir/struct.Program.html +[rust_ir]: https://rust-lang.github.io/chalk/doc/chalk/rust_ir/index.html + +[binders-struct]: https://github.com/rust-lang/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L661 +[chalk-ast]: https://github.com/rust-lang/chalk/blob/master/chalk-parse/src/ast.rs +[chalk-test-example]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L115 +[chalk-test-lowering-example]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rust_ir/lowering/test.rs#L8-L31 +[chalk-test-lowering]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rust_ir/lowering/test.rs +[chalk-test-wf]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules/wf/test.rs#L1 +[chalki]: https://rust-lang.github.io/chalk/doc/chalki/index.html +[clause]: https://github.com/rust-lang/chalk/blob/master/GLOSSARY.md#clause +[coherence-src]: https://github.com/rust-lang/chalk/blob/master/src/coherence.rs +[ir-code]: https://github.com/rust-lang/chalk/blob/master/src/rust_ir.rs +[rules-environment]: https://github.com/rust-lang/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/rules.rs#L9 +[rules-src]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules.rs +[rules-wf-src]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules/wf.rs +[solve_goal]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L85 +[test-lowering-macros]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test_util.rs#L21-L54 +[test-macro]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L33 diff --git a/src/traits/slg.md b/src/traits/slg.md index dcddd01f5..98547575d 100644 --- a/src/traits/slg.md +++ b/src/traits/slg.md @@ -47,7 +47,7 @@ well as the various *strands*, which are basically suspended computations that may be used to find more answers. Tables are interdependent: solving one query may require solving others. -[`Forest`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/forest/struct.Forest.html +[`Forest`]: https://rust-lang.github.io/chalk/doc/chalk_engine/forest/struct.Forest.html ### Walkthrough @@ -150,7 +150,7 @@ is the subgoal after the turnstile (`:-`) that we are currently trying to prove in this strand. Initally, when a strand is first created, there is no selected subgoal. -[`ExClause`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/struct.ExClause.html +[`ExClause`]: https://rust-lang.github.io/chalk/doc/chalk_engine/struct.ExClause.html **Activating a strand.** Now that we have created the table T0 and initialized it with strands, we have to actually try and produce an answer. @@ -187,7 +187,7 @@ Here, we write `selected(L, An)` to indicate that (a) the literal `L` is the selected subgoal and (b) which answer `An` we are looking for. We start out looking for `A0`. -[`ensure_root_answer`]: https://rust-lang-nursery.github.io/chalk/doc/chalk_engine/forest/struct.Forest.html#method.ensure_root_answer +[`ensure_root_answer`]: https://rust-lang.github.io/chalk/doc/chalk_engine/forest/struct.Forest.html#method.ensure_root_answer **Processing the selected subgoal.** Next, we have to try and find an answer to this selected goal. To do that, we will u-canonicalize it @@ -297,6 +297,6 @@ more answers later on. - [Negative Reasoning in Chalk][negative-reasoning-blog] explains the need for negative reasoning, but not how the SLG solver does it -[readme]: https://github.com/rust-lang-nursery/chalk/blob/239e4ae4e69b2785b5f99e0f2b41fc16b0b4e65e/chalk-engine/src/README.md +[readme]: https://github.com/rust-lang/chalk/blob/239e4ae4e69b2785b5f99e0f2b41fc16b0b4e65e/chalk-engine/src/README.md [slg-blog]: http://smallcultfollowing.com/babysteps/blog/2018/01/31/an-on-demand-slg-solver-for-chalk/ [negative-reasoning-blog]: http://aturon.github.io/blog/2017/04/24/negative-chalk/ From e8370fda838fc29feed244d44e887b0fb613db85 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Fri, 15 Mar 2019 17:25:12 -0500 Subject: [PATCH 523/648] update rustc driver chapters --- src/SUMMARY.md | 2 +- src/appendix/code-index.md | 7 ++-- src/appendix/stupid-stats.md | 16 +++++---- src/rustc-driver.md | 63 +++++++++++------------------------- 4 files changed, 32 insertions(+), 56 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 6cd75c588..d3cef1a25 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -26,7 +26,7 @@ - [Part 2: How rustc works](./part-2-intro.md) - [High-level overview of the compiler source](./high-level-overview.md) -- [The Rustc Driver](./rustc-driver.md) +- [The Rustc Driver and Interface](./rustc-driver.md) - [Rustdoc](./rustdoc.md) - [Queries: demand-driven compilation](./query.md) - [The Query Evaluation Model in Detail](./queries/query-evaluation-model-in-detail.md) diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index c0a1d03a9..3c71d6bba 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -7,7 +7,7 @@ compiler. Item | Kind | Short description | Chapter | Declaration ----------------|----------|-----------------------------|--------------------|------------------- `BodyId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.BodyId.html) -`CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver] | [src/librustc_driver/driver.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/struct.CompileState.html) +`Compiler` | struct | Represents a compiler session and can be used to drive a compilation. | [The Rustc Driver and Interface] | [src/librustc_interface/interface.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/interface/struct.Compiler.html) `ast::Crate` | struct | A syntax-level representation of a parsed crate | [The parser] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Crate.html) `hir::Crate` | struct | A more abstract, compiler-friendly form of a crate's AST | [The Hir] | [src/librustc/hir/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/struct.Crate.html) `DefId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [src/librustc/hir/def_id.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/hir/def_id/struct.DefId.html) @@ -18,8 +18,9 @@ Item | Kind | Short description | Chapter | `P` | struct | An owned immutable smart pointer. By contrast, `&T` is not owned, and `Box` is not immutable. | None | [src/syntax/ptr.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ptr/struct.P.html) `ParamEnv` | struct | Information about generic parameters or `Self`, useful for working with associated or generic items | [Parameter Environment] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.ParamEnv.html) `ParseSess` | struct | This struct contains information about a parsing session | [The parser] | [src/libsyntax/parse/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/struct.ParseSess.html) +`Query` | struct | Represents the result of query to the `Compiler` interface and allows stealing, borrowing, and returning the results of compiler passes. | [The Rustc Driver and Interface] | [src/librustc_interface/queries.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/queries/struct.Query.html) `Rib` | struct | Represents a single scope of names | [Name resolution] | [src/librustc_resolve/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/struct.Rib.html) -`Session` | struct | The data associated with a compilation session | [The parser], [The Rustc Driver] | [src/librustc/session/mod.html](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html) +`Session` | struct | The data associated with a compilation session | [The parser], [The Rustc Driver and Interface] | [src/librustc/session/mod.html](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html) `SourceFile` | struct | Part of the `SourceMap`. Maps AST nodes to their source code for a single source file. Was previously called FileMap | [The parser] | [src/libsyntax_pos/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceFile.html) `SourceMap` | struct | Maps AST nodes to their source code. It is composed of `SourceFile`s. Was previously called CodeMap | [The parser] | [src/libsyntax/source_map.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html) `Span` | struct | A location in the user's source code, used for error reporting primarily | [Emitting Diagnostics] | [src/libsyntax_pos/span_encoding.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax_pos/struct.Span.html) @@ -33,7 +34,7 @@ Item | Kind | Short description | Chapter | [The HIR]: ../hir.html [Identifiers in the HIR]: ../hir.html#hir-id [The parser]: ../the-parser.html -[The Rustc Driver]: ../rustc-driver.html +[The Rustc Driver and Interface]: ../rustc-driver.html [Type checking]: ../type-checking.html [The `ty` modules]: ../ty.html [Rustdoc]: ../rustdoc.html diff --git a/src/appendix/stupid-stats.md b/src/appendix/stupid-stats.md index a36cac42b..5066973a9 100644 --- a/src/appendix/stupid-stats.md +++ b/src/appendix/stupid-stats.md @@ -1,9 +1,12 @@ # Appendix A: A tutorial on creating a drop-in replacement for rustc > **Note:** This is a copy of `@nrc`'s amazing [stupid-stats]. You should find -> a copy of the code on the GitHub repository although due to the compiler's -> constantly evolving nature, there is no guarantee it'll compile on the first -> go. +> a copy of the code on the GitHub repository. +> +> Due to the compiler's constantly evolving nature, the `rustc_driver` +> mechanisms described in this chapter have been replaced by a new +> [`rustc_interface`] crate. See [The Rustc Driver and Interface] for more +> information. Many tools benefit from being a drop-in replacement for a compiler. By this, I mean that any user of the tool can use `mytool` in all the ways they would @@ -92,7 +95,7 @@ translation). > and [`librustc_codegen_utils`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_utils/index.html). All these phases are coordinated by the driver. To see the exact sequence, look -at [the `compile_input` function in `librustc_driver`][compile-input]. +at the `compile_input` function in `librustc_driver`. The driver handles all the highest level coordination of compilation - 1. handling command-line arguments 2. maintaining compilation state (primarily in the `Session`) @@ -101,9 +104,6 @@ The driver handles all the highest level coordination of compilation - To create a drop-in compiler replacement or a compiler replacement, we leave most of compilation alone and customise the driver using its APIs. -[compile-input]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/fn.compile_input.html - - ## The driver customisation APIs There are two primary ways to customise compilation - high level control of the @@ -410,3 +410,5 @@ internally (I already changed save-analysis to use `CompilerController`). I've been experimenting with a prototype rustfmt which also uses these APIs. [stupid-stats]: https://github.com/nrc/stupid-stats +[`rustc_interface`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/index.html +[The Rustc Driver and Interface]: ../rustc-driver.html diff --git a/src/rustc-driver.md b/src/rustc-driver.md index 397949836..92754f675 100644 --- a/src/rustc-driver.md +++ b/src/rustc-driver.md @@ -1,46 +1,25 @@ -# The Rustc Driver +# The Rustc Driver and Interface The [`rustc_driver`] is essentially `rustc`'s `main()` function. It acts as the glue for running the various phases of the compiler in the correct order, -managing state such as the [`SourceMap`] \(maps AST nodes to source code), -[`Session`] \(general build context and error messaging) and the [`TyCtxt`] -\(the "typing context", allowing you to query the type system and other cool -stuff). The `rustc_driver` crate also provides external users with a method +using the interface defined in the [`rustc_interface`] crate. + +The `rustc_interface` crate provides external users with an (unstable) API for running code at particular times during the compilation process, allowing third parties to effectively use `rustc`'s internals as a library for -analysing a crate or emulating the compiler in-process (e.g. the RLS). - -For those using `rustc` as a library, the `run_compiler()` function is the main -entrypoint to the compiler. Its main parameters are a list of command-line -arguments and a reference to something which implements the `CompilerCalls` -trait. A `CompilerCalls` creates the overall `CompileController`, letting it -govern which compiler passes are run and attach callbacks to be fired at the end -of each phase. - -From `rustc_driver`'s perspective, the main phases of the compiler are: - -1. *Parse Input:* Initial crate parsing -2. *Configure and Expand:* Resolve `#[cfg]` attributes, name resolution, and - expand macros -3. *Run Analysis Passes:* Run trait resolution, typechecking, region checking - and other miscellaneous analysis passes on the crate -4. *Translate to LLVM:* Translate to the in-memory form of LLVM IR and turn it - into an executable/object files +analysing a crate or emulating the compiler in-process (e.g. the RLS or rustdoc). -The `CompileController` then gives users the ability to inspect the ongoing -compilation process +For those using `rustc` as a library, the `interface::run_compiler()` function is the main +entrypoint to the compiler. It takes a configuration for the compiler and a closure that +takes a [`Compiler`]. `run_compiler` creates a `Compiler` from the configuration and passes +it to the closure. Inside the closure, you can use the `Compiler` to drive queries to compile +a crate and get the results. This is what the `rustc_driver` does too. -- after parsing -- after AST expansion -- after HIR lowering -- after analysis, and -- when compilation is done - -The `CompileState`'s various `state_after_*()` constructors can be inspected to -determine what bits of information are available to which callback. - -For a more detailed explanation on using `rustc_driver`, check out the -[stupid-stats] guide by `@nrc` (attached as [Appendix A]). +You can see what queries are currently available through the rustdocs for [`Compiler`]. +You can see an example of how to use them by looking at the `rustc_driver` implementation, +specifically the [`rustc_driver::run_compiler` function][rd_rc] (not to be confused with +`interface::run_compiler`). The `rustc_driver::run_compiler` function takes a bunch of +command-line args and some other configurations and drives the compilation to completion. > **Warning:** By its very nature, the internal compiler APIs are always going > to be unstable. That said, we do try not to break things unnecessarily. @@ -54,21 +33,15 @@ manifests itself in the way people can plug into the compiler, preferring a "push"-style API (callbacks) instead of the more Rust-ic "pull" style (think the `Iterator` trait). -For example the [`CompileState`], the state passed to callbacks after each -phase, is essentially just a box of optional references to pieces inside the -compiler. The lifetime bound on the `CompilerCalls` trait then helps to ensure -compiler internals don't "escape" the compiler (e.g. if you tried to keep a -reference to the AST after the compiler is finished), while still letting users -record *some* state for use after the `run_compiler()` function finishes. - Thread-local storage and interning are used a lot through the compiler to reduce duplication while also preventing a lot of the ergonomic issues due to many pervasive lifetimes. The `rustc::ty::tls` module is used to access these thread-locals, although you should rarely need to touch it. - +[rd_rc]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/fn.run_compiler.html +[`rustc_interface`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/index.html [`rustc_driver`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/ -[`CompileState`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/driver/struct.CompileState.html +[`Compiler`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/interface/struct.Compiler.html [`Session`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html [`TyCtxt`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html [`SourceMap`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html From 960ddd62214bc2dc2a6a3b77b3d8c8e9c3ad1a35 Mon Sep 17 00:00:00 2001 From: varkor Date: Thu, 14 Mar 2019 13:17:20 +0000 Subject: [PATCH 524/648] Fix mask -> tag terminology --- src/kinds.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/kinds.md b/src/kinds.md index d5e218c31..ac6a2d35b 100644 --- a/src/kinds.md +++ b/src/kinds.md @@ -21,7 +21,7 @@ constructing such substitution slices. ## `Kind` The actual `Kind` struct is optimised for space, storing the type, lifetime or -const as an interned pointer containing a mask identifying its kind (in the +const as an interned pointer containing a tag identifying its kind (in the lowest 2 bits). Unless you are working with the `Subst` implementation specifically, you should generally not have to deal with `Kind` and instead make use of the safe [`UnpackedKind`](#unpackedkind) abstraction. From e627f5031a4b82e52d341a07b40a4d6360c48b9b Mon Sep 17 00:00:00 2001 From: Santiago Pastorino Date: Tue, 12 Mar 2019 13:54:50 -0300 Subject: [PATCH 525/648] Add experts map section to compiler team page --- src/compiler-team.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/compiler-team.md b/src/compiler-team.md index 728325687..b837c038c 100644 --- a/src/compiler-team.md +++ b/src/compiler-team.md @@ -17,6 +17,14 @@ home. In any case, you can find people in one of three places at the moment: - The `t-compiler` stream on [the Zulip instance](https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler) - The `compiler` channel on the [rust-lang discord](https://discord.gg/rust-lang) +## Expert map + +If you're interested in figuring out who can answer questions about a +particular part of the compiler, or you'd just like to know who works on what, +check out our [experts directory](https://github.com/rust-lang/compiler-team/blob/master/experts/MAP.md). +It contains a listing of the various parts of the compiler and a list of people +who are experts on each one. + ## Rust compiler meeting The compiler team has a weekly meeting where we do triage and try to From 5c827ef97b91f3bd90cfae9438abf1cc261bb188 Mon Sep 17 00:00:00 2001 From: mark Date: Tue, 19 Mar 2019 19:02:59 -0500 Subject: [PATCH 526/648] address Zoxc's comments --- src/appendix/stupid-stats.md | 5 ++++- src/rustc-driver.md | 8 ++++++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/src/appendix/stupid-stats.md b/src/appendix/stupid-stats.md index 5066973a9..6ed7d1cbd 100644 --- a/src/appendix/stupid-stats.md +++ b/src/appendix/stupid-stats.md @@ -4,7 +4,9 @@ > a copy of the code on the GitHub repository. > > Due to the compiler's constantly evolving nature, the `rustc_driver` -> mechanisms described in this chapter have been replaced by a new +> mechanisms described in this chapter have changed. In particular, the +> `CompilerCalls` and `CompileController` types have been replaced by +> [`Callbacks`][cb]. Also, there is a new query-based interface in the > [`rustc_interface`] crate. See [The Rustc Driver and Interface] for more > information. @@ -409,6 +411,7 @@ analysis, rather than doing its own analysis). Other parts of the compiler internally (I already changed save-analysis to use `CompilerController`). I've been experimenting with a prototype rustfmt which also uses these APIs. +[cb]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/trait.Callbacks.html [stupid-stats]: https://github.com/nrc/stupid-stats [`rustc_interface`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/index.html [The Rustc Driver and Interface]: ../rustc-driver.html diff --git a/src/rustc-driver.md b/src/rustc-driver.md index 92754f675..715e6295d 100644 --- a/src/rustc-driver.md +++ b/src/rustc-driver.md @@ -21,6 +21,13 @@ specifically the [`rustc_driver::run_compiler` function][rd_rc] (not to be confu `interface::run_compiler`). The `rustc_driver::run_compiler` function takes a bunch of command-line args and some other configurations and drives the compilation to completion. +`rustc_driver::run_compiler` also takes a [`Callbacks`][cb]. In the past, when +the `rustc_driver::run_compiler` was the primary way to use the compiler as a +library, these callbacks were used to have some custom code run after different +phases of the compilation. If you read [Appendix A], you may notice the use of the +types `CompilerCalls` and `CompileController`, which no longer exist. `Callbacks` +replaces this functionality. + > **Warning:** By its very nature, the internal compiler APIs are always going > to be unstable. That said, we do try not to break things unnecessarily. @@ -38,6 +45,7 @@ duplication while also preventing a lot of the ergonomic issues due to many pervasive lifetimes. The `rustc::ty::tls` module is used to access these thread-locals, although you should rarely need to touch it. +[cb]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/trait.Callbacks.html [rd_rc]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/fn.run_compiler.html [`rustc_interface`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/index.html [`rustc_driver`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/ From c7d4e2fde216ead53fb88b6cd70376359455517f Mon Sep 17 00:00:00 2001 From: Alexander Regueiro Date: Thu, 21 Mar 2019 23:29:17 +0000 Subject: [PATCH 527/648] Small fix to code for checking feature gate --- src/implementing_new_features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/implementing_new_features.md b/src/implementing_new_features.md index 650d9915c..8105e1474 100644 --- a/src/implementing_new_features.md +++ b/src/implementing_new_features.md @@ -146,7 +146,7 @@ version is when you are stabilizing a feature. 4. Prevent usage of the new feature unless the feature gate is set. You can check it in most places in the compiler using the expression `tcx.features().$feature_name` (or - `sess.features_untracked().borrow().$feature_name` if the + `sess.features_untracked().$feature_name` if the tcx is unavailable) If the feature gate is not set, you should either maintain From 67ec74d6296e54424c4efa142e3346df0036decc Mon Sep 17 00:00:00 2001 From: Dan Robertson Date: Wed, 20 Mar 2019 14:03:54 +0000 Subject: [PATCH 528/648] Update link to debugging codegen --- src/codegen.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/codegen.md b/src/codegen.md index 766dfff59..c8d51f318 100644 --- a/src/codegen.md +++ b/src/codegen.md @@ -4,9 +4,9 @@ Code generation or "codegen" is the part of the compiler that actually generates an executable binary. rustc uses LLVM for code generation. > NOTE: If you are looking for hints on how to debug code generation bugs, -> please see [this section of the debugging chapter][debug]. +> please see [this section of the debugging chapter][debugging]. -[debug]: compiler-debugging.html#debugging-llvm +[debugging]: codegen/debugging.html ## What is LLVM? From 9df878f4b860c4c03d1659a46b4100445dd9d085 Mon Sep 17 00:00:00 2001 From: Denys Zariaiev Date: Sat, 16 Mar 2019 23:33:43 +0100 Subject: [PATCH 529/648] Add a note about `assembly` test suite. --- src/tests/intro.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/tests/intro.md b/src/tests/intro.md index 4d509f3a8..c043da429 100644 --- a/src/tests/intro.md +++ b/src/tests/intro.md @@ -38,6 +38,8 @@ that give more details. - `debuginfo` – tests that run in gdb or lldb and query the debug info - `codegen` – tests that compile and then test the generated LLVM code to make sure that the optimizations we want are taking effect. +- `assembly` – similar to `codegen` tests, but verifies assembly output + to make sure LLVM target backend can handle provided code. - `mir-opt` – tests that check parts of the generated MIR to make sure we are building things correctly or doing the optimizations we expect. From 9b06a122e4ea11d5830402c9f3c752a16eb6ef68 Mon Sep 17 00:00:00 2001 From: mark Date: Tue, 19 Mar 2019 18:50:01 -0500 Subject: [PATCH 530/648] Use eddyb's suggestion from #11 about logging --- src/compiler-debugging.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index 714385ca0..6747fd31c 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -162,9 +162,10 @@ your log filter, e.g. to get the logs for a specific module, you can run the compiler as `RUST_LOG=module::path rustc my-file.rs`. All `debug!` output will then appear in standard error. -Note that unless you use a very strict filter, the logger will emit a *lot* -of output - so it's typically a good idea to pipe standard error to a file -and look at the log output with a text editor. +**Note that unless you use a very strict filter, the logger will emit a lot of +output, so use the most specific module(s) you can (comma-separated if +multiple)**. It's typically a good idea to pipe standard error to a file and +look at the log output with a text editor. You should also . So to put it together. From 464cb5b166378dff64619081dd4c42533a1eb989 Mon Sep 17 00:00:00 2001 From: mark Date: Sat, 23 Mar 2019 18:12:37 -0500 Subject: [PATCH 531/648] oops --- src/compiler-debugging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index 6747fd31c..bce5959dc 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -165,7 +165,7 @@ then appear in standard error. **Note that unless you use a very strict filter, the logger will emit a lot of output, so use the most specific module(s) you can (comma-separated if multiple)**. It's typically a good idea to pipe standard error to a file and -look at the log output with a text editor. You should also . +look at the log output with a text editor. So to put it together. From d78594637392eb26bf747be9befc94ee7e6eb475 Mon Sep 17 00:00:00 2001 From: Krishna Sannasi Date: Thu, 11 Apr 2019 17:04:02 -0700 Subject: [PATCH 532/648] Update query-evaluation-model-in-detail.md Fix typo in the example --- src/queries/query-evaluation-model-in-detail.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/queries/query-evaluation-model-in-detail.md b/src/queries/query-evaluation-model-in-detail.md index 41789637d..47412acef 100644 --- a/src/queries/query-evaluation-model-in-detail.md +++ b/src/queries/query-evaluation-model-in-detail.md @@ -99,7 +99,7 @@ from outside of the query system, invoke the queries it needs to perform its task. This looks something like the following: ```rust,ignore -fn compile_crate() {} +fn compile_crate() { let cli_options = ...; let hir_map = ...; From 07d1d3c83b200208d8bac28af65271d628c3a3ee Mon Sep 17 00:00:00 2001 From: "Zack M. Davis" Date: Tue, 5 Feb 2019 22:36:35 -0800 Subject: [PATCH 533/648] subchapter with information about `--error-format json` --- src/SUMMARY.md | 1 + src/diag/json-format.md | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100644 src/diag/json-format.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index d3cef1a25..099c65a73 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -83,6 +83,7 @@ - [Updating LLVM](./codegen/updating-llvm.md) - [Debugging LLVM](./codegen/debugging.md) - [Emitting Diagnostics](./diag.md) + - [JSON diagnostic format](./diag/json-format.md) --- diff --git a/src/diag/json-format.md b/src/diag/json-format.md new file mode 100644 index 000000000..6ba555b1d --- /dev/null +++ b/src/diag/json-format.md @@ -0,0 +1,36 @@ +# JSON diagnostic output + +The compiler accepts an `--error-format json` flag to output +diagnostics as JSON objects (for the benefit of tools such as `cargo +fix` or the RLS). It looks like this— + +```ignore +$ rustc json_error_demo.rs --error-format json +{"message":"cannot add `&str` to `{integer}`","code":{"code":"E0277","explanation":"\nYou tried to use a type which doesn't implement some trait in a place which\nexpected that trait. Erroneous code example:\n\n```compile_fail,E0277\n// here we declare the Foo trait with a bar method\ntrait Foo {\n fn bar(&self);\n}\n\n// we now declare a function which takes an object implementing the Foo trait\nfn some_func(foo: T) {\n foo.bar();\n}\n\nfn main() {\n // we now call the method with the i32 type, which doesn't implement\n // the Foo trait\n some_func(5i32); // error: the trait bound `i32 : Foo` is not satisfied\n}\n```\n\nIn order to fix this error, verify that the type you're using does implement\nthe trait. Example:\n\n```\ntrait Foo {\n fn bar(&self);\n}\n\nfn some_func(foo: T) {\n foo.bar(); // we can now use this method since i32 implements the\n // Foo trait\n}\n\n// we implement the trait on the i32 type\nimpl Foo for i32 {\n fn bar(&self) {}\n}\n\nfn main() {\n some_func(5i32); // ok!\n}\n```\n\nOr in a generic context, an erroneous code example would look like:\n\n```compile_fail,E0277\nfn some_func(foo: T) {\n println!(\"{:?}\", foo); // error: the trait `core::fmt::Debug` is not\n // implemented for the type `T`\n}\n\nfn main() {\n // We now call the method with the i32 type,\n // which *does* implement the Debug trait.\n some_func(5i32);\n}\n```\n\nNote that the error here is in the definition of the generic function: Although\nwe only call it with a parameter that does implement `Debug`, the compiler\nstill rejects the function: It must work with all possible input types. In\norder to make this example compile, we need to restrict the generic type we're\naccepting:\n\n```\nuse std::fmt;\n\n// Restrict the input type to types that implement Debug.\nfn some_func(foo: T) {\n println!(\"{:?}\", foo);\n}\n\nfn main() {\n // Calling the method is still fine, as i32 implements Debug.\n some_func(5i32);\n\n // This would fail to compile now:\n // struct WithoutDebug;\n // some_func(WithoutDebug);\n}\n```\n\nRust only looks at the signature of the called function, as such it must\nalready specify all requirements that will be used for every type parameter.\n"},"level":"error","spans":[{"file_name":"json_error_demo.rs","byte_start":50,"byte_end":51,"line_start":4,"line_end":4,"column_start":7,"column_end":8,"is_primary":true,"text":[{"text":" a + b","highlight_start":7,"highlight_end":8}],"label":"no implementation for `{integer} + &str`","suggested_replacement":null,"suggestion_applicability":null,"expansion":null}],"children":[{"message":"the trait `std::ops::Add<&str>` is not implemented for `{integer}`","code":null,"level":"help","spans":[],"children":[],"rendered":null}],"rendered":"error[E0277]: cannot add `&str` to `{integer}`\n --> json_error_demo.rs:4:7\n |\n4 | a + b\n | ^ no implementation for `{integer} + &str`\n |\n = help: the trait `std::ops::Add<&str>` is not implemented for `{integer}`\n\n"} +{"message":"aborting due to previous error","code":null,"level":"error","spans":[],"children":[],"rendered":"error: aborting due to previous error\n\n"} +{"message":"For more information about this error, try `rustc --explain E0277`.","code":null,"level":"","spans":[],"children":[],"rendered":"For more information about this error, try `rustc --explain E0277`.\n"} +``` + +Note that the output is a series of lines, each of which is a JSON +object, but the series of lines taken together is, unfortunately, not +valid JSON, thwarting tools and tricks (such as [piping to `python3 -m +json.tool`](https://docs.python.org/3/library/json.html#module-json.tool)) +that require such. (One speculates that this was intentional for LSP +performance purposes, so that each line/object can be sent to RLS as +it is flushed?) + +Also note the "rendered" field, which contains the "human" output as a +string; this was introduced so that UI tests could both make use of +the structured JSON and see the "human" output (well, _sans_ colors) +without having to compile everything twice. + +The JSON emitter currently lives in libsyntax/json.rs. (But arguably +it should live in librustc_errors along with the "human" emitter? It's +not obvious to the present author why it wasn't moved from libsyntax +to librustc_errors at the same [time the "human" emitter was +moved](https://github.com/rust-lang/rust/commit/6ae3502134).) + +The JSON emitter defines [its own `Diagnostic` +struct](https://github.com/rust-lang/rust/blob/b2c6b8c29f13f8d1f242da89e587960b95337819/src/libsyntax/json.rs#L85-L99) +(and sub-structs) for the JSON serialization. Don't confuse this with +[`errors::Diagnostic`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.Diagnostic.html)! From 7503057194de61550ba8387b2c7ff04526e5f56c Mon Sep 17 00:00:00 2001 From: Alexey Shmalko Date: Tue, 16 Apr 2019 00:52:08 +0300 Subject: [PATCH 534/648] Update query chapter for the query macro rewrite There was a big macro rewrite in these pull requests: https://github.com/rust-lang/rust/pull/56462 https://github.com/rust-lang/rust/pull/59517 Update the query chapter to describe the new macro usage. --- src/query.md | 76 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 44 insertions(+), 32 deletions(-) diff --git a/src/query.md b/src/query.md index 703c560e0..4d80187aa 100644 --- a/src/query.md +++ b/src/query.md @@ -167,50 +167,45 @@ Well, defining a query takes place in two steps: To specify the query name and arguments, you simply add an entry to the big macro invocation in -[`src/librustc/ty/query/mod.rs`][query-mod], which looks something like: +[`src/librustc/query/mod.rs`][query-mod], which looks something like: -[query-mod]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/query/index.html +[query-mod]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/query/index.html ```rust,ignore -define_queries! { <'tcx> - /// Records the type of every item. - [] fn type_of: TypeOfItem(DefId) -> Ty<'tcx>, +rustc_queries! { + Other { + /// Records the type of every item. + query type_of(key: DefId) -> Ty<'tcx> { + cache { key.is_local() } + } + } ... } ``` -Each line of the macro defines one query. The name is broken up like this: +Queries are grouped into categories (`Other`, `Codegen`, `TypeChecking`, etc.). +Each group contains one or more queries. Each query definition is broken up like +this: ```rust,ignore -[] fn type_of: TypeOfItem(DefId) -> Ty<'tcx>, -^^ ^^^^^^^ ^^^^^^^^^^ ^^^^^ ^^^^^^^^ -| | | | | -| | | | result type of query -| | | query key type -| | dep-node constructor +query type_of(key: DefId) -> Ty<'tcx> { ... } +^^ ^^^^^^^ ^^^^^ ^^^^^^^^ ^^^ +| | | | | +| | | | query modifiers +| | | result type of query +| | query key type | name of query -query flags +query keyword ``` Let's go over them one by one: -- **Query flags:** these are largely unused right now, but the intention - is that we'll be able to customize various aspects of how the query is - processed. +- **Query keyword:** indicates a start of a query definition. - **Name of query:** the name of the query method (`tcx.type_of(..)`). Also used as the name of a struct (`ty::queries::type_of`) that will be generated to represent this query. -- **Dep-node constructor:** indicates the constructor function that - connects this query to incremental compilation. Typically, this is a - `DepNode` variant, which can be added by modifying the - `define_dep_nodes!` macro invocation in - [`librustc/dep_graph/dep_node.rs`][dep-node]. - - However, sometimes we use a custom function, in which case the - name will be in snake case and the function will be defined at the - bottom of the file. This is typically used when the query key is - not a def-id, or just not the type that the dep-node expects. - **Query key type:** the type of the argument to this query. This type must implement the `ty::query::keys::Key` trait, which defines (for example) how to map it to a crate, and so forth. @@ -222,19 +217,18 @@ Let's go over them one by one: which is used to cheaply modify MIR in place. See the definition of `Steal` for more details. New uses of `Steal` should **not** be added without alerting `@rust-lang/compiler`. - -[dep-node]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/dep_graph/struct.DepNode.html +- **Query modifiers:** various flags and options that customize how the + query is processed. So, to add a query: -- Add an entry to `define_queries!` using the format above. -- Possibly add a corresponding entry to the dep-node macro. +- Add an entry to `rustc_queries!` using the format above. - Link the provider by modifying the appropriate `provide` method; or add a new one if needed and ensure that `rustc_driver` is invoking it. #### Query structs and descriptions -For each kind, the `define_queries` macro will generate a "query struct" +For each kind, the `rustc_queries` macro will generate a "query struct" named after the query. This struct is a kind of a place-holder describing the query. Each such struct implements the `self::config::QueryConfig` trait, which has associated types for the @@ -243,11 +237,14 @@ like this: ```rust,ignore // Dummy struct representing a particular kind of query: -pub struct type_of<'tcx> { phantom: PhantomData<&'tcx ()> } +pub struct type_of<'tcx> { data: PhantomData<&'tcx ()> } impl<'tcx> QueryConfig for type_of<'tcx> { type Key = DefId; type Value = Ty<'tcx>; + + const NAME: QueryName = QueryName::type_of; + const CATEGORY: ProfileCategory = ProfileCategory::Other; } ``` @@ -262,9 +259,24 @@ You can put new impls into the `config` module. They look something like this: ```rust,ignore impl<'tcx> QueryDescription for queries::type_of<'tcx> { fn describe(tcx: TyCtxt, key: DefId) -> String { - format!("computing the type of `{}`", tcx.item_path_str(key)) + format!("computing the type of `{}`", tcx.def_path_str(key)) + } +} +``` + +Another option is to add `desc` modifier: + +```rust,ignore +rustc_queries! { + Other { + /// Records the type of every item. + query type_of(key: DefId) -> Ty<'tcx> { + desc { |tcx| "computing the type of `{}`", tcx.def_path_str(key) } + } } } ``` +`rustc_queries` macro will generate an appropriate `impl` automatically. + [query-model]: queries/query-evaluation-model-in-detail.html From b95f3493890c017b70b8442c38e7ef2f9334731a Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 16 Apr 2019 15:29:25 -0500 Subject: [PATCH 535/648] fix MovePathIndex link --- .../moves_and_initialization/move_paths.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/src/borrow_check/moves_and_initialization/move_paths.md b/src/borrow_check/moves_and_initialization/move_paths.md index 8fd7b3f19..4d4db4116 100644 --- a/src/borrow_check/moves_and_initialization/move_paths.md +++ b/src/borrow_check/moves_and_initialization/move_paths.md @@ -32,13 +32,11 @@ efficiently. ## Move path indices -Although there is a [`MovePath`] data structure, they are never -referenced directly. Instead, all the code passes around *indices* of -type -[`MovePathIndex`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/indexes/struct.MovePathIndex.html). If -you need to get information about a move path, you use this index with -the [`move_paths` field of the `MoveData`][move_paths]. For example, -to convert a [`MovePathIndex`] `mpi` into a MIR [`Place`], you might +Although there is a [`MovePath`] data structure, they are never referenced +directly. Instead, all the code passes around *indices* of type +[`MovePathIndex`]. If you need to get information about a move path, you use +this index with the [`move_paths` field of the `MoveData`][move_paths]. For +example, to convert a [`MovePathIndex`] `mpi` into a MIR [`Place`], you might access the [`MovePath::place`] field like so: ```rust,ignore @@ -47,7 +45,7 @@ move_data.move_paths[mpi].place [move_paths]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MoveData.html#structfield.move_paths [`MovePath::place`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MovePath.html#structfield.place -[`MovePathIndex`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/indexes/struct.MovePathIndex.html +[`MovePathIndex`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MovePathIndex.html ## Building move paths From 2ef961e45465a5edfbd3d5c49e4a16cee356b44b Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 16 Apr 2019 14:13:38 -0500 Subject: [PATCH 536/648] move to subsection --- src/diag.md | 37 +++++++++++++++++++++++++++++++++++++ src/diag/json-format.md | 36 ------------------------------------ 2 files changed, 37 insertions(+), 36 deletions(-) diff --git a/src/diag.md b/src/diag.md index cfdd82aa2..31268e9f6 100644 --- a/src/diag.md +++ b/src/diag.md @@ -305,3 +305,40 @@ an appropriate mapping to the body of [`Lint::from_parser_lint_id`][fplid]. [`BufferedEarlyLintId`]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/early_buffered_lints/enum.BufferedEarlyLintId.html [fplid]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/lint/struct.Lint.html#method.from_parser_lint_id + +## JSON diagnostic output + +The compiler accepts an `--error-format json` flag to output +diagnostics as JSON objects (for the benefit of tools such as `cargo +fix` or the RLS). It looks like this— + +```console +$ rustc json_error_demo.rs --error-format json +{"message":"cannot add `&str` to `{integer}`","code":{"code":"E0277","explanation":"\nYou tried to use a type which doesn't implement some trait in a place which\nexpected that trait. Erroneous code example:\n\n```compile_fail,E0277\n// here we declare the Foo trait with a bar method\ntrait Foo {\n fn bar(&self);\n}\n\n// we now declare a function which takes an object implementing the Foo trait\nfn some_func(foo: T) {\n foo.bar();\n}\n\nfn main() {\n // we now call the method with the i32 type, which doesn't implement\n // the Foo trait\n some_func(5i32); // error: the trait bound `i32 : Foo` is not satisfied\n}\n```\n\nIn order to fix this error, verify that the type you're using does implement\nthe trait. Example:\n\n```\ntrait Foo {\n fn bar(&self);\n}\n\nfn some_func(foo: T) {\n foo.bar(); // we can now use this method since i32 implements the\n // Foo trait\n}\n\n// we implement the trait on the i32 type\nimpl Foo for i32 {\n fn bar(&self) {}\n}\n\nfn main() {\n some_func(5i32); // ok!\n}\n```\n\nOr in a generic context, an erroneous code example would look like:\n\n```compile_fail,E0277\nfn some_func(foo: T) {\n println!(\"{:?}\", foo); // error: the trait `core::fmt::Debug` is not\n // implemented for the type `T`\n}\n\nfn main() {\n // We now call the method with the i32 type,\n // which *does* implement the Debug trait.\n some_func(5i32);\n}\n```\n\nNote that the error here is in the definition of the generic function: Although\nwe only call it with a parameter that does implement `Debug`, the compiler\nstill rejects the function: It must work with all possible input types. In\norder to make this example compile, we need to restrict the generic type we're\naccepting:\n\n```\nuse std::fmt;\n\n// Restrict the input type to types that implement Debug.\nfn some_func(foo: T) {\n println!(\"{:?}\", foo);\n}\n\nfn main() {\n // Calling the method is still fine, as i32 implements Debug.\n some_func(5i32);\n\n // This would fail to compile now:\n // struct WithoutDebug;\n // some_func(WithoutDebug);\n}\n```\n\nRust only looks at the signature of the called function, as such it must\nalready specify all requirements that will be used for every type parameter.\n"},"level":"error","spans":[{"file_name":"json_error_demo.rs","byte_start":50,"byte_end":51,"line_start":4,"line_end":4,"column_start":7,"column_end":8,"is_primary":true,"text":[{"text":" a + b","highlight_start":7,"highlight_end":8}],"label":"no implementation for `{integer} + &str`","suggested_replacement":null,"suggestion_applicability":null,"expansion":null}],"children":[{"message":"the trait `std::ops::Add<&str>` is not implemented for `{integer}`","code":null,"level":"help","spans":[],"children":[],"rendered":null}],"rendered":"error[E0277]: cannot add `&str` to `{integer}`\n --> json_error_demo.rs:4:7\n |\n4 | a + b\n | ^ no implementation for `{integer} + &str`\n |\n = help: the trait `std::ops::Add<&str>` is not implemented for `{integer}`\n\n"} +{"message":"aborting due to previous error","code":null,"level":"error","spans":[],"children":[],"rendered":"error: aborting due to previous error\n\n"} +{"message":"For more information about this error, try `rustc --explain E0277`.","code":null,"level":"","spans":[],"children":[],"rendered":"For more information about this error, try `rustc --explain E0277`.\n"} +``` + +Note that the output is a series of lines, each of which is a JSON +object, but the series of lines taken together is, unfortunately, not +valid JSON, thwarting tools and tricks (such as [piping to `python3 -m +json.tool`](https://docs.python.org/3/library/json.html#module-json.tool)) +that require such. (One speculates that this was intentional for LSP +performance purposes, so that each line/object can be sent to RLS as +it is flushed?) + +Also note the "rendered" field, which contains the "human" output as a +string; this was introduced so that UI tests could both make use of +the structured JSON and see the "human" output (well, _sans_ colors) +without having to compile everything twice. + +The JSON emitter currently lives in libsyntax/json.rs. (But arguably +it should live in librustc_errors along with the "human" emitter? It's +not obvious to the present author why it wasn't moved from libsyntax +to librustc_errors at the same [time the "human" emitter was +moved](https://github.com/rust-lang/rust/commit/6ae3502134).) + +The JSON emitter defines [its own `Diagnostic` +struct](https://github.com/rust-lang/rust/blob/b2c6b8c29f13f8d1f242da89e587960b95337819/src/libsyntax/json.rs#L85-L99) +(and sub-structs) for the JSON serialization. Don't confuse this with +[`errors::Diagnostic`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.Diagnostic.html)! diff --git a/src/diag/json-format.md b/src/diag/json-format.md index 6ba555b1d..e69de29bb 100644 --- a/src/diag/json-format.md +++ b/src/diag/json-format.md @@ -1,36 +0,0 @@ -# JSON diagnostic output - -The compiler accepts an `--error-format json` flag to output -diagnostics as JSON objects (for the benefit of tools such as `cargo -fix` or the RLS). It looks like this— - -```ignore -$ rustc json_error_demo.rs --error-format json -{"message":"cannot add `&str` to `{integer}`","code":{"code":"E0277","explanation":"\nYou tried to use a type which doesn't implement some trait in a place which\nexpected that trait. Erroneous code example:\n\n```compile_fail,E0277\n// here we declare the Foo trait with a bar method\ntrait Foo {\n fn bar(&self);\n}\n\n// we now declare a function which takes an object implementing the Foo trait\nfn some_func(foo: T) {\n foo.bar();\n}\n\nfn main() {\n // we now call the method with the i32 type, which doesn't implement\n // the Foo trait\n some_func(5i32); // error: the trait bound `i32 : Foo` is not satisfied\n}\n```\n\nIn order to fix this error, verify that the type you're using does implement\nthe trait. Example:\n\n```\ntrait Foo {\n fn bar(&self);\n}\n\nfn some_func(foo: T) {\n foo.bar(); // we can now use this method since i32 implements the\n // Foo trait\n}\n\n// we implement the trait on the i32 type\nimpl Foo for i32 {\n fn bar(&self) {}\n}\n\nfn main() {\n some_func(5i32); // ok!\n}\n```\n\nOr in a generic context, an erroneous code example would look like:\n\n```compile_fail,E0277\nfn some_func(foo: T) {\n println!(\"{:?}\", foo); // error: the trait `core::fmt::Debug` is not\n // implemented for the type `T`\n}\n\nfn main() {\n // We now call the method with the i32 type,\n // which *does* implement the Debug trait.\n some_func(5i32);\n}\n```\n\nNote that the error here is in the definition of the generic function: Although\nwe only call it with a parameter that does implement `Debug`, the compiler\nstill rejects the function: It must work with all possible input types. In\norder to make this example compile, we need to restrict the generic type we're\naccepting:\n\n```\nuse std::fmt;\n\n// Restrict the input type to types that implement Debug.\nfn some_func(foo: T) {\n println!(\"{:?}\", foo);\n}\n\nfn main() {\n // Calling the method is still fine, as i32 implements Debug.\n some_func(5i32);\n\n // This would fail to compile now:\n // struct WithoutDebug;\n // some_func(WithoutDebug);\n}\n```\n\nRust only looks at the signature of the called function, as such it must\nalready specify all requirements that will be used for every type parameter.\n"},"level":"error","spans":[{"file_name":"json_error_demo.rs","byte_start":50,"byte_end":51,"line_start":4,"line_end":4,"column_start":7,"column_end":8,"is_primary":true,"text":[{"text":" a + b","highlight_start":7,"highlight_end":8}],"label":"no implementation for `{integer} + &str`","suggested_replacement":null,"suggestion_applicability":null,"expansion":null}],"children":[{"message":"the trait `std::ops::Add<&str>` is not implemented for `{integer}`","code":null,"level":"help","spans":[],"children":[],"rendered":null}],"rendered":"error[E0277]: cannot add `&str` to `{integer}`\n --> json_error_demo.rs:4:7\n |\n4 | a + b\n | ^ no implementation for `{integer} + &str`\n |\n = help: the trait `std::ops::Add<&str>` is not implemented for `{integer}`\n\n"} -{"message":"aborting due to previous error","code":null,"level":"error","spans":[],"children":[],"rendered":"error: aborting due to previous error\n\n"} -{"message":"For more information about this error, try `rustc --explain E0277`.","code":null,"level":"","spans":[],"children":[],"rendered":"For more information about this error, try `rustc --explain E0277`.\n"} -``` - -Note that the output is a series of lines, each of which is a JSON -object, but the series of lines taken together is, unfortunately, not -valid JSON, thwarting tools and tricks (such as [piping to `python3 -m -json.tool`](https://docs.python.org/3/library/json.html#module-json.tool)) -that require such. (One speculates that this was intentional for LSP -performance purposes, so that each line/object can be sent to RLS as -it is flushed?) - -Also note the "rendered" field, which contains the "human" output as a -string; this was introduced so that UI tests could both make use of -the structured JSON and see the "human" output (well, _sans_ colors) -without having to compile everything twice. - -The JSON emitter currently lives in libsyntax/json.rs. (But arguably -it should live in librustc_errors along with the "human" emitter? It's -not obvious to the present author why it wasn't moved from libsyntax -to librustc_errors at the same [time the "human" emitter was -moved](https://github.com/rust-lang/rust/commit/6ae3502134).) - -The JSON emitter defines [its own `Diagnostic` -struct](https://github.com/rust-lang/rust/blob/b2c6b8c29f13f8d1f242da89e587960b95337819/src/libsyntax/json.rs#L85-L99) -(and sub-structs) for the JSON serialization. Don't confuse this with -[`errors::Diagnostic`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.Diagnostic.html)! From aa7bb2bc5d6ae2688589ff066426eb5ad0ac24a0 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 16 Apr 2019 15:18:55 -0500 Subject: [PATCH 537/648] update chalk with new organization --- src/traits/chalk-overview.md | 51 ++++++++++++++++++------------------ src/traits/lowering-rules.md | 6 ++--- src/traits/wf.md | 10 +++---- 3 files changed, 34 insertions(+), 33 deletions(-) diff --git a/src/traits/chalk-overview.md b/src/traits/chalk-overview.md index 54046b22e..4d1260ddc 100644 --- a/src/traits/chalk-overview.md +++ b/src/traits/chalk-overview.md @@ -70,16 +70,17 @@ impls, and struct definitions. Parsing is often the first "phase" of transformation that a program goes through in order to become a format that chalk can understand. -### Rust Intermediate Representation ([rust_ir]) +### Rust Intermediate Representation ([chalk_rust_ir]) After getting the AST we convert it to a more convenient intermediate -representation called [`rust_ir`][rust_ir]. This is sort of analogous to the -[HIR] in Rust. The process of converting to IR is called *lowering*. +representation called [`chalk_rust_ir`][chalk_rust_ir]. This is sort of +analogous to the [HIR] in Rust. The process of converting to IR is called +*lowering*. -The [`rust_ir::Program`][rust_ir-program] struct contains some "rust things" +The [`chalk::program::Program`][chalk-program] struct contains some "rust things" but indexed and accessible in a different way. For example, if you have a type like `Foo`, we would represent `Foo` as a string in the AST but in -`rust_ir::Program`, we use numeric indices (`ItemId`). +`chalk::program::Program`, we use numeric indices (`ItemId`). The [IR source code][ir-code] contains the complete definition. @@ -120,14 +121,14 @@ forall { (Vec: Clone) :- (T: Clone) } This rule dictates that `Vec: Clone` is only satisfied if `T: Clone` is also satisfied (i.e. "provable"). -Similar to [`rust_ir::Program`][rust_ir-program] which has "rust-like +Similar to [`chalk::program::Program`][chalk-program] which has "rust-like things", chalk_ir defines [`ProgramEnvironment`] which which is "pure logic". The main field in that struct is `program_clauses`, which contains the [`ProgramClause`]s generated by the rules module. -#### Rules +### Rules ([chalk_rules]) -The `rules` module ([source code][rules-src]) defines the logic rules we use +The `chalk_rules` crate ([source code][chalk_rules]) defines the logic rules we use for each item in the Rust IR. It works by iterating over every trait, impl, etc. and emitting the rules that come from each one. @@ -136,13 +137,13 @@ etc. and emitting the rules that come from each one. #### Well-formedness checks As part of lowering to logic, we also do some "well formedness" checks. See -the [`rules::wf` source code][rules-wf-src] for where those are done. +the [`chalk_rules::wf` source code][rules-wf-src] for where those are done. *See also: [Well-formedness checking][wf-checking]* #### Coherence -The function `record_specialization_priorities` in the `coherence` module +The method `CoherenceSolver::specialization_priorities` in the `coherence` module ([source code][coherence-src]) checks "coherence", which means that it ensures that two impls of the same trait for the same type cannot exist. @@ -158,18 +159,19 @@ queries is called the *solver*. Chalk's functionality is broken up into the following crates: - [**chalk_engine**][chalk_engine]: Defines the core [SLG solver][slg]. +- [**chalk_rust_ir**][chalk_rust_ir], containing the "HIR-like" form of the AST - [**chalk_ir**][chalk_ir]: Defines chalk's internal representation of types, lifetimes, and goals. - [**chalk_solve**][chalk_solve]: Combines `chalk_ir` and `chalk_engine`, effectively. - [`chalk_engine::context`][engine-context] provides the necessary hooks. - [**chalk_parse**][chalk_parse]: Defines the raw AST and a parser. +- [**chalk_rules**][chalk_rules]: which implements logic rules converting + `chalk_rust_ir` to `chalk_ir` + - Defines the `coherence` module, which implements coherence rules - [**chalk**][doc-chalk]: Brings everything together. Defines the following modules: - - [`rust_ir`][rust_ir], containing the "HIR-like" form of the AST - - `rust_ir::lowering`, which converts AST to `rust_ir` - - `rules`, which implements logic rules converting `rust_ir` to `chalk_ir` - - `coherence`, which implements coherence rules + - `chalk::lowering`, which converts AST to `chalk_rust_ir` - Also includes [chalki][chalki], chalk's REPL. [Browse source code on GitHub](https://github.com/rust-lang/chalk) @@ -188,7 +190,7 @@ which is expected to lower to logic successfully, and a set of queries tests support specifying only a prefix of the output. **Lowering tests** check the stages that occur before we can issue queries -to the solver: the [lowering to rust_ir][chalk-test-lowering], and the +to the solver: the [lowering to chalk_rust_ir][chalk-test-lowering], and the [well-formedness checks][chalk-test-wf] that occur after that. ### Testing internals @@ -229,29 +231,28 @@ Likewise, lowering tests use the [`lowering_success!` and [universal quantification]: https://en.wikipedia.org/wiki/Universal_quantification [`ProgramClause`]: https://rust-lang.github.io/chalk/doc/chalk_ir/enum.ProgramClause.html -[`ProgramEnvironment`]: https://rust-lang.github.io/chalk/doc/chalk_ir/struct.ProgramEnvironment.html +[`ProgramEnvironment`]: http://rust-lang.github.io/chalk/doc/chalk/program_environment/struct.ProgramEnvironment.html [chalk_engine]: https://rust-lang.github.io/chalk/doc/chalk_engine/index.html [chalk_ir]: https://rust-lang.github.io/chalk/doc/chalk_ir/index.html [chalk_parse]: https://rust-lang.github.io/chalk/doc/chalk_parse/index.html [chalk_solve]: https://rust-lang.github.io/chalk/doc/chalk_solve/index.html +[chalk_rules]: https://rust-lang.github.io/chalk/doc/chalk_rules/index.html +[chalk_rust_ir]: https://rust-lang.github.io/chalk/doc/chalk_rust_ir/index.html [doc-chalk]: https://rust-lang.github.io/chalk/doc/chalk/index.html [engine-context]: https://rust-lang.github.io/chalk/doc/chalk_engine/context/index.html -[rust_ir-program]: https://rust-lang.github.io/chalk/doc/chalk/rust_ir/struct.Program.html -[rust_ir]: https://rust-lang.github.io/chalk/doc/chalk/rust_ir/index.html +[chalk-program]: http://rust-lang.github.io/chalk/doc/chalk/program/struct.Program.html -[binders-struct]: https://github.com/rust-lang/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/ir.rs#L661 -[chalk-ast]: https://github.com/rust-lang/chalk/blob/master/chalk-parse/src/ast.rs +[binders-struct]: http://rust-lang.github.io/chalk/doc/chalk_ir/struct.Binders.html +[chalk-ast]: http://rust-lang.github.io/chalk/doc/chalk_parse/ast/index.html [chalk-test-example]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L115 [chalk-test-lowering-example]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rust_ir/lowering/test.rs#L8-L31 [chalk-test-lowering]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rust_ir/lowering/test.rs [chalk-test-wf]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules/wf/test.rs#L1 [chalki]: https://rust-lang.github.io/chalk/doc/chalki/index.html [clause]: https://github.com/rust-lang/chalk/blob/master/GLOSSARY.md#clause -[coherence-src]: https://github.com/rust-lang/chalk/blob/master/src/coherence.rs -[ir-code]: https://github.com/rust-lang/chalk/blob/master/src/rust_ir.rs -[rules-environment]: https://github.com/rust-lang/chalk/blob/94a1941a021842a5fcb35cd043145c8faae59f08/src/rules.rs#L9 -[rules-src]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules.rs -[rules-wf-src]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules/wf.rs +[coherence-src]: http://rust-lang.github.io/chalk/doc/chalk_rules/coherence/index.html +[ir-code]: http://rust-lang.github.io/chalk/doc/chalk_rust_ir/ +[rules-wf-src]: http://rust-lang.github.io/chalk/doc/chalk_rules/wf/index.html [solve_goal]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L85 [test-lowering-macros]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test_util.rs#L21-L54 [test-macro]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L33 diff --git a/src/traits/lowering-rules.md b/src/traits/lowering-rules.md index 88a61ac4f..7f46e5738 100644 --- a/src/traits/lowering-rules.md +++ b/src/traits/lowering-rules.md @@ -28,10 +28,10 @@ comment like so: // Rule Foo-Bar-Baz The reference implementation of these rules is to be found in -[`chalk/src/rules.rs`][chalk_rules]. They are also ported in rustc in the -[`librustc_traits`][librustc_traits] crate. +[`chalk/chalk-rules/src/clauses.rs`][chalk_rules]. They are also ported in +rustc in the [`librustc_traits`][librustc_traits] crate. -[chalk_rules]: https://github.com/rust-lang-nursery/chalk/blob/master/src/rules.rs +[chalk_rules]: https://github.com/rust-lang/chalk/blob/master/chalk-rules/src/clauses.rs [librustc_traits]: https://github.com/rust-lang/rust/tree/master/src/librustc_traits ## Lowering where clauses diff --git a/src/traits/wf.md b/src/traits/wf.md index f0cb03caa..e2d15f1db 100644 --- a/src/traits/wf.md +++ b/src/traits/wf.md @@ -11,14 +11,14 @@ to prove it using the lowered rules we described in the [lowering rules](./lowering-rules.md) chapter. If we are able to prove it, we say that the construct is well-formed. If not, we report an error to the user. -Well-formedness checking happens in the [`src/rules/wf.rs`][wf] module in -chalk. After you have read this chapter, you may find useful to see an -extended set of examples in the [`src/rules/wf/test.rs`][wf_test] submodule. +Well-formedness checking happens in the [`chalk/chalk-rules/src/wf.rs`][wf] +module in chalk. After you have read this chapter, you may find useful to see +an extended set of examples in the [`chalk/src/test/wf.rs`][wf_test] submodule. The new-style WF checking has not been implemented in rustc yet. -[wf]: https://github.com/rust-lang-nursery/chalk/blob/master/src/rules/wf.rs -[wf_test]: https://github.com/rust-lang-nursery/chalk/blob/master/src/rules/wf/test.rs +[wf]: https://github.com/rust-lang/chalk/blob/master/chalk-rules/src/wf.rs +[wf_test]: https://github.com/rust-lang/chalk/blob/master/src/test/wf.rs We give here a complete reference of the generated goals for each Rust declaration. From 813cdf613e24313c61f4cdb05518a75d99d21c8f Mon Sep 17 00:00:00 2001 From: Alexey Shmalko Date: Wed, 17 Apr 2019 11:55:14 +0300 Subject: [PATCH 538/648] Update test-implementation chapter to current code `test_main_static` is now used instead of `test_static_main`. The libsyntax no longer generates a `TESTS` constant but rather passes all test cases directly into `test_main_static` as a slice. Update the guide accordingly. --- src/test-implementation.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/src/test-implementation.md b/src/test-implementation.md index 3c93ad619..969d6d2e4 100644 --- a/src/test-implementation.md +++ b/src/test-implementation.md @@ -81,20 +81,18 @@ Now that our tests are accessible from the root of our crate, we need to do something with them. `libsyntax` generates a module like so: ```rust,ignore -pub mod __test { +#[main] +pub fn main() { extern crate test; - const TESTS: &'static [self::test::TestDescAndFn] = &[/*...*/]; - - #[main] - pub fn main() { - self::test::test_static_main(TESTS); - } + test::test_main_static(&[&path::to::test1, /*...*/]); } ``` +where `path::to::test1` is a constant of type `test::TestDescAndFn`. + While this transformation is simple, it gives us a lot of insight into how tests are actually run. The tests are aggregated into an array and passed to -a test runner called `test_static_main`. We'll come back to exactly what +a test runner called `test_main_static`. We'll come back to exactly what `TestDescAndFn` is, but for now, the key takeaway is that there is a crate called [`test`][test] that is part of Rust core, that implements all of the runtime for testing. `test`'s interface is unstable, so the only stable way @@ -119,7 +117,7 @@ configuration information as well. `test` encodes this configuration data into a struct called [`TestDesc`][TestDesc]. For each test function in a crate, `libsyntax` will parse its attributes and generate a `TestDesc` instance. It then combines the `TestDesc` and test function into the -predictably named `TestDescAndFn` struct, that `test_static_main` operates +predictably named `TestDescAndFn` struct, that `test_main_static` operates on. For a given test, the generated `TestDescAndFn` instance looks like so: ```rust,ignore @@ -151,4 +149,4 @@ $ rustc my_mod.rs -Z unpretty=hir [Symbol]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Ident.html [Ident]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/ast/struct.Ident.html [eRFC]: https://github.com/rust-lang/rfcs/blob/master/text/2318-custom-test-frameworks.md -[libsyntax]: https://github.com/rust-lang/rust/tree/master/src/libsyntax \ No newline at end of file +[libsyntax]: https://github.com/rust-lang/rust/tree/master/src/libsyntax From 99e1b1d53656be08654df399fc200584aebb50e4 Mon Sep 17 00:00:00 2001 From: Alexey Shmalko Date: Thu, 18 Apr 2019 16:51:47 +0300 Subject: [PATCH 539/648] Update BodyId description It is now a newtype'd HirId, not NodeId. See https://github.com/rust-lang/rust/pull/58167. --- src/hir.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/hir.md b/src/hir.md index 34e478ee4..b10042270 100644 --- a/src/hir.md +++ b/src/hir.md @@ -87,9 +87,9 @@ sorts of identifiers in active use: offset within that item. - the key point of a [`HirId`] is that it is *relative* to some item (which is named via a [`DefId`]). -- [`BodyId`], this is an absolute identifier that refers to a specific +- [`BodyId`], this is an identifier that refers to a specific body (definition of a function or constant) in the crate. It is currently - effectively a "newtype'd" [`NodeId`]. + effectively a "newtype'd" [`HirId`]. - [`NodeId`], which is an absolute id that identifies a single node in the HIR tree. - While these are still in common use, **they are being slowly phased out**. From a0ab5fac4b09b765478fa2a1a3de390ca60c3d61 Mon Sep 17 00:00:00 2001 From: Alexey Shmalko Date: Sat, 20 Apr 2019 23:00:13 +0300 Subject: [PATCH 540/648] [canonicalization] fix result canonicalization example --- src/traits/canonical-queries.md | 2 +- src/traits/canonicalization.md | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/traits/canonical-queries.md b/src/traits/canonical-queries.md index cbf7d880d..e15bdaae2 100644 --- a/src/traits/canonical-queries.md +++ b/src/traits/canonical-queries.md @@ -173,7 +173,7 @@ variables have unbound inference variables in their type: `?T` represents the elements in the vector `t` and `?U` represents the value stored in the option `u`. Next, we invoke `foo`; comparing the signature of `foo` to its arguments, we wind up with `A = Vec` and -`B = ?U`.Therefore, the where clause on `foo` requires that `Vec: +`B = ?U`. Therefore, the where clause on `foo` requires that `Vec: Borrow`. This is thus our first example trait query. There are many possible solutions to the query `Vec: Borrow`; diff --git a/src/traits/canonicalization.md b/src/traits/canonicalization.md index ca81d6fd1..d6d57d785 100644 --- a/src/traits/canonicalization.md +++ b/src/traits/canonicalization.md @@ -177,8 +177,8 @@ The result would be as follows: ```text Canonical(QR) = for { certainty: Proven, - var_values: [Vec, '?1, ?2] - region_constraints: [?2: '?1], + var_values: [Vec, '?1, ?0] + region_constraints: [?0: '?1], value: (), } ``` @@ -213,8 +213,8 @@ and now we got back a canonical response: ```text for { certainty: Proven, - var_values: [Vec, '?1, ?2] - region_constraints: [?2: '?1], + var_values: [Vec, '?1, ?0] + region_constraints: [?0: '?1], value: (), } ``` @@ -250,7 +250,7 @@ for later verification. than eagerly instantiating all of the canonical values in the result with variables, we instead walk the vector of values, looking for cases where the value is just a canonical variable. In our example, -`values[2]` is `?C`, so that means we can deduce that `?C := ?B and +`values[2]` is `?C`, so that means we can deduce that `?C := ?B` and `'?D := 'static`. This gives us a partial set of values. Anything for which we do not find a value, we create an inference variable.) From 92219f29bb3666a605d71de394c9401b28a62c6a Mon Sep 17 00:00:00 2001 From: Alexey Shmalko Date: Sat, 20 Apr 2019 20:08:00 +0300 Subject: [PATCH 541/648] Update lowering-module test case --- src/traits/lowering-module.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/src/traits/lowering-module.md b/src/traits/lowering-module.md index 939484051..3f1515ef8 100644 --- a/src/traits/lowering-module.md +++ b/src/traits/lowering-module.md @@ -31,7 +31,7 @@ this writing, it looked like this: trait Foo { } -#[rustc_dump_program_clauses] //~ ERROR Implemented(T: Foo) :- +#[rustc_dump_program_clauses] //~ ERROR program clause dump impl Foo for T where T: Iterator { } fn main() { @@ -45,19 +45,18 @@ compiler will then invoke the `program_clauses_for` query on that item, and emit compiler errors that dump the clauses produced. These errors just exist for unit-testing, as we can then leverage the standard [ui test] mechanisms to check them. In this case, there is a -`//~ ERROR Implemented` annotation which is intentionally minimal (it -need only be a prefix of the error), but [the stderr file] contains +`//~ ERROR program clause dump` annotation which is always the same for +`#[rustc_dump_program_clauses]`, but [the stderr file] contains the full details: ```text -error: Implemented(T: Foo) :- ProjectionEq(::Item == i32), TypeOutlives(T \ -: 'static), Implemented(T: std::iter::Iterator), Implemented(T: std::marker::Sized). - --> $DIR/lower_impl.rs:15:1 +error: program clause dump + --> $DIR/lower_impl.rs:5:1 | -LL | #[rustc_dump_program_clauses] //~ ERROR Implemented(T: Foo) :- +LL | #[rustc_dump_program_clauses] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -error: aborting due to previous error + | + = note: forall { Implemented(T: Foo) :- ProjectionEq(::Item == i32), TypeOutlives(T: 'static), Implemented(T: std::iter::Iterator), Implemented(T: std::marker::Sized). } ``` [chalkify]: https://github.com/rust-lang/rust/tree/master/src/test/ui/chalkify From ad07f7fac5a68007f8f7d87f70ebc9bf60293909 Mon Sep 17 00:00:00 2001 From: Oliver Scherer Date: Wed, 24 Apr 2019 15:57:47 +0200 Subject: [PATCH 542/648] Explain new powers of the `treat-err-as-bug` flag --- src/compiler-debugging.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index bce5959dc..2e45e5f02 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -70,8 +70,10 @@ stack backtrace: [getting-a-backtrace-for-errors]: #getting-a-backtrace-for-errors If you want to get a backtrace to the point where the compiler emits -an error message, you can pass the `-Z treat-err-as-bug`, which -will make the compiler panic on the first error it sees. +an error message, you can pass the `-Z treat-err-as-bug=n`, which +will make the compiler skip `n` errors or `delay_span_bug` calls and then +panic on the next one. If you leave off `=n`, the compiler will assume `0` for +`n` and thus panic on the first error it encounters. This can also help when debugging `delay_span_bug` calls - it will make the first `delay_span_bug` call panic, which will give you a useful backtrace. From dca6649b722260351a49e2e39dce05b5da56a230 Mon Sep 17 00:00:00 2001 From: Matthew Jasper Date: Wed, 24 Apr 2019 20:53:08 +0100 Subject: [PATCH 543/648] Add documentation for two-phase borrows --- src/SUMMARY.md | 1 + src/borrow_check/two_phase_borrows.md | 100 ++++++++++++++++++++++++++ 2 files changed, 101 insertions(+) create mode 100644 src/borrow_check/two_phase_borrows.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 099c65a73..477b9a707 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -76,6 +76,7 @@ - [Move paths](./borrow_check/moves_and_initialization/move_paths.md) - [MIR type checker](./borrow_check/type_check.md) - [Region inference](./borrow_check/region_inference.md) + - [Two-phase-borrows](./borrow_check/two_phase_borrows.md) - [Constant evaluation](./const-eval.md) - [miri const evaluator](./miri.md) - [Parameter Environments](./param_env.md) diff --git a/src/borrow_check/two_phase_borrows.md b/src/borrow_check/two_phase_borrows.md new file mode 100644 index 000000000..753ffa5b2 --- /dev/null +++ b/src/borrow_check/two_phase_borrows.md @@ -0,0 +1,100 @@ +# Two-phase borrows + +Two-phase borrows are a more permissive version of mutable borrows that allow +nested method calls such as `vec.push(vec.len())`. Such borrows first act as +shared borrows in a "reservation" phase and can later be "activated" into a +full mutable borrow. + +Only certain implicit mutable borrows can be two-phase, any `&mut` or `ref mut` +in the source code is never a two-phase borrow. The cases where we generate a +two-phase borrow are: + +1. The autoref borrow when calling a method with a mutable reference receiver. +2. A mutable reborrow in function arguments. +3. The implicit mutable borrow in an overloaded compound assignment operator. + +To give some examples: + +```rust +// In the source code + +// Case 1: +let mut v = Vec::new(); +v.push(v.len()); +let r = &mut Vec::new(); +r.push(r.len()); + +// Case 2: +std::mem::replace(r, vec![1, r.len()]); + +// Case 3: +let mut x = std::num::Wrapping(2); +x += x; +``` + +Expanding these enough to show the two-phase borrows: + +```rust,ignore +// Case 1: +let mut v = Vec::new(); +let temp1 = &two_phase v; +let temp2 = v.len(); +Vec::push(temp1, temp2); +let r = &mut Vec::new(); +let temp3 = &two_phase *r; +let temp4 = r.len(); +Vec::push(temp3, temp4); + +// Case 2: +let temp5 = &two_phase *r; +let temp6 = vec![1, r.len()]; +std::mem::replace(temp5, temp6); + +// Case 3: +let mut x = std::num::Wrapping(2); +let temp7 = &two_phase x; +let temp8 = x; +std::ops::AddAssign::add_assign(temp7, temp8); +``` + +Whether a borrow can be two-phase is tracked by a flag on the [`AutoBorrow`] +after type checking, which is then [converted] to a [`BorrowKind`] during MIR +construction. + +Each two-phase borrow is assigned to a temporary that is only used once. As +such we can define: + +* The point where the temporary is assigned to is called the *reservation* + point of the two-phase borrow. +* The point where the temporary is used, which is effectively always a + function call, is called the *activation* point. + +The activation points are found using the [`GatherBorrows`] visitor. The +[`BorrowData`] then holds both the reservation and activation points for the +borrow. + +[`AutoBorrow`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/adjustment/enum.AutoBorrow.html +[converted]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/hair/cx/expr/trait.ToBorrowKind.html#method.to_borrow_kind +[`BorrowKind`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/enum.BorrowKind.html +[`GatherBorrows`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/visit/trait.Visitor.html#method.visit_local +[`BorrowData`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/borrow_set/struct.BorrowData.html + +## Checking two-phase borrows + +Two-phase borrows are treated as if they were mutable borrows with the +following exceptions: + +1. At every location in the MIR we [check] if any two-phase borrows are + activated at this location. If a live two phase borrow is activated at a + location, then we check that there are no borrows that conflict with the + two-phase borrow. +2. At the reservation point we error if there are conflicting live *mutable* + borrows. And lint if there are any conflicting shared borrows. +3. Between the reservation and the activation point, the two-phase borrow acts + as a shared borrow. We determine (in [`is_active`]) if we're at such a point + by using the [`Dominators`] for the MIR graph. +4. After the activation point, the two-phase borrow acts as a mutable borrow. + +[check]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/struct.MirBorrowckCtxt.html#method.check_activations +[`Dominators`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_structures/graph/dominators/struct.Dominators.html +[`is_active`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/path_utils/fn.is_active.html From e905e3aa17f64686a9da9dba5415e1571ba81ed8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?I=C3=B1aki=20Garay?= Date: Thu, 25 Apr 2019 19:24:36 -0300 Subject: [PATCH 544/648] Fixed broken chalk links --- src/traits/lowering-rules.md | 4 ++-- src/traits/wf.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/traits/lowering-rules.md b/src/traits/lowering-rules.md index 7f46e5738..bbebf1450 100644 --- a/src/traits/lowering-rules.md +++ b/src/traits/lowering-rules.md @@ -28,10 +28,10 @@ comment like so: // Rule Foo-Bar-Baz The reference implementation of these rules is to be found in -[`chalk/chalk-rules/src/clauses.rs`][chalk_rules]. They are also ported in +[`chalk/chalk-solve/src/clauses.rs`][chalk_rules]. They are also ported in rustc in the [`librustc_traits`][librustc_traits] crate. -[chalk_rules]: https://github.com/rust-lang/chalk/blob/master/chalk-rules/src/clauses.rs +[chalk_rules]: https://github.com/rust-lang/chalk/blob/master/chalk-solve/src/clauses.rs [librustc_traits]: https://github.com/rust-lang/rust/tree/master/src/librustc_traits ## Lowering where clauses diff --git a/src/traits/wf.md b/src/traits/wf.md index e2d15f1db..0ced345a1 100644 --- a/src/traits/wf.md +++ b/src/traits/wf.md @@ -11,13 +11,13 @@ to prove it using the lowered rules we described in the [lowering rules](./lowering-rules.md) chapter. If we are able to prove it, we say that the construct is well-formed. If not, we report an error to the user. -Well-formedness checking happens in the [`chalk/chalk-rules/src/wf.rs`][wf] +Well-formedness checking happens in the [`chalk/chalk-solve/src/wf.rs`][wf] module in chalk. After you have read this chapter, you may find useful to see an extended set of examples in the [`chalk/src/test/wf.rs`][wf_test] submodule. The new-style WF checking has not been implemented in rustc yet. -[wf]: https://github.com/rust-lang/chalk/blob/master/chalk-rules/src/wf.rs +[wf]: https://github.com/rust-lang/chalk/blob/master/chalk-solve/src/wf.rs [wf_test]: https://github.com/rust-lang/chalk/blob/master/src/test/wf.rs We give here a complete reference of the generated goals for each Rust From 1ff2ac0c2aa4fc0a8fe31acdfdee4a4b9c9ec127 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Thu, 25 Apr 2019 20:18:55 -0500 Subject: [PATCH 545/648] use nightly rust for ci --- .travis.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.travis.yml b/.travis.yml index 55ea20587..fc6f4f05e 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,4 +1,6 @@ language: rust +rust: +- nightly cache: - cargo before_install: From b68e57b705f4799b7f06467a92b0c080982bd80c Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 29 Apr 2019 07:26:21 -0700 Subject: [PATCH 546/648] Bring the updating LLVM guide up to date --- src/codegen/updating-llvm.md | 62 ++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 35 deletions(-) diff --git a/src/codegen/updating-llvm.md b/src/codegen/updating-llvm.md index e29e041b2..b26c475f7 100644 --- a/src/codegen/updating-llvm.md +++ b/src/codegen/updating-llvm.md @@ -36,24 +36,25 @@ For updates of LLVM that typically just update a bug, we cherry-pick the bugfix to the branch we're already using. The steps for this are: 1. Make sure the bugfix is in upstream LLVM. -2. Identify the branch that rustc is currently using. The `src/llvm` submodule - is always pinned to a branch of the - [rust-lang/llvm](https://github.com/rust-lang/llvm) repository. -3. Fork the rust-lang/llvm repository -4. Check out the appropriate branch (typically named `rust-llvm-release-*`) +2. Identify the branch that rustc is currently using. The `src/llvm-project` + submodule is always pinned to a branch of the + [rust-lang/llvm-project](https://github.com/rust-lang/llvm-project) repository. +3. Fork the rust-lang/llvm-project repository +4. Check out the appropriate branch (typically named `rustc/a.b-yyyy-mm-dd`) 5. Cherry-pick the upstream commit onto the branch 6. Push this branch to your fork -7. Send a Pull Request to rust-lang/llvm to the same branch as before +7. Send a Pull Request to rust-lang/llvm-project to the same branch as before 8. Wait for the PR to be merged -9. Send a PR to rust-lang/rust updating the `src/llvm` submodule with your bugfix +9. Send a PR to rust-lang/rust updating the `src/llvm-project` submodule with + your bugfix 10. Wait for PR to be merged The tl;dr; is that we can cherry-pick bugfixes at any time and pull them back -into the rust-lang/llvm branch that we're using, and getting it into the +into the rust-lang/llvm-project branch that we're using, and getting it into the compiler is just updating the submodule via a PR! Example PRs look like: -[#56313](https://github.com/rust-lang/rust/pull/56313) +[#59089](https://github.com/rust-lang/rust/pull/59089) ## Feature updates @@ -66,37 +67,28 @@ lot more work. This is where we can't reasonably cherry-pick commits backwards so we need to do a full update. There's a lot of stuff to do here, so let's go through each in detail. -1. Create new branches in all repositories for this update. Branches should be - named `rust-llvm-release-X-Y-Z-vA` where `X.Y.Z` is the LLVM version and `A` - is just increasing based on if there's previous branches of this name. All - repositories here should be branched at the same time from the upstream LLVM - projects, we currently use https://github.com/llvm-mirror repositories. The - list of repositories that need a new branch are: - - * rust-lang/llvm - * rust-lang/compiler-rt - * rust-lang/lld - * rust-lang-nursery/lldb - * rust-lang-nursery/clang - -2. Apply Rust-specific patches to LLVM repositories. All features and bugfixes - are upstream, but there's often some weird build-related patches that don't - make sense to upstream which we have on our repositories. These patches are - typically the latest patches on the branch. All repositories, except `clang`, - currently have Rust-specific patches. +1. Create a new branch in the rust-lang/llvm-project repository. This branch + should be named `rustc/a.b-yyyy-mm-dd` where `a.b` is the current version + number of LLVM in-tree at the time of the branch and the remaining part is + today's date. + +2. Apply Rust-specific patches to the llvm-project repository. All features and + bugfixes are upstream, but there's often some weird build-related patches + that don't make sense to upstream which we have on our repositories. These + patches are around the latest patches in the rust-lang/llvm-project branch + that rustc is currently using. 3. Update the `compiler-rt` submodule in the - `rust-lang-nursery/compiler-builtins` repository. Push this update to a - `rust-llvm-release-*` branch of the `compiler-builtins` repository. + `rust-lang-nursery/compiler-builtins` repository. Push this update to the + same branch name of the `llvm-project` submodule to the + of the `rust-lang/compiler-rt` repository. Then push this update to a branch + of `compiler-builtins` with the same-named branch. Note that this step is + frequently optional since we may not need to update `compiler-rt`. 4. Prepare a commit to rust-lang/rust - * Update `src/llvm` - * Update `src/tools/lld` - * Update `src/tools/lldb` - * Update `src/tools/clang` - * Update `src/libcompiler_builtins - * Edit `src/rustllvm/llvm-rebuild-trigger` to update its contents + * Update `src/llvm-project` + * Update `compiler-builtins` crate in `Cargo.lock` (if necessary) 5. Build your commit. Make sure you've committed the previous changes to ensure submodule updates aren't reverted. Some commands you should execute are: From 786d85c10e602effdc7439bb2e5def811ba5d086 Mon Sep 17 00:00:00 2001 From: Nicolas Date: Wed, 1 May 2019 03:58:22 -0300 Subject: [PATCH 547/648] Remove IRC from discussion chats --- src/compiler-team.md | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/src/compiler-team.md b/src/compiler-team.md index b837c038c..0f0808b79 100644 --- a/src/compiler-team.md +++ b/src/compiler-team.md @@ -9,12 +9,9 @@ contributions to rustc and its design. ## Discussion -Currently the compiler team chats in a number of places. There is an -ongoing [thread] on the internals board about trying to find a permanent -home. In any case, you can find people in one of three places at the moment: +Currently the compiler team chats in 2 places: -- The `#rustc` channel on mozilla's IRC (`irc.mozilla.org`) -- The `t-compiler` stream on [the Zulip instance](https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler) +- The `t-compiler` stream on [the Zulip instance][zulip] - The `compiler` channel on the [rust-lang discord](https://discord.gg/rust-lang) ## Expert map @@ -50,14 +47,11 @@ The meeting currently takes place on Thursdays at 10am Boston time (UTC-4 typically, but daylight savings time sometimes makes things complicated). -The meeting is held over a "chat medium" — it used to be IRC, but we -are currently in the process of evaluating other alternatives. Check -the [etherpad] to find the current home (and see -[this internals thread][thread] for some ongoing discussion). +The meeting is held over a "chat medium", currently on [zulip]. [etherpad]: https://public.etherpad-mozilla.org/p/rust-compiler-meeting -[thread]: https://internals.rust-lang.org/t/where-should-the-compiler-team-and-perhaps-working-groups-chat/7894 [procedure]: https://forge.rust-lang.org/rustc-bug-fix-procedure.html +[zulip]: https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler ## Team membership From b05cfab80bdffd50da2390a45e1f4424a62d0f61 Mon Sep 17 00:00:00 2001 From: guilherme Date: Fri, 26 Apr 2019 23:47:16 +0100 Subject: [PATCH 548/648] Fix link in walkthrough --- src/walkthrough.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/walkthrough.md b/src/walkthrough.md index 2823426be..e2e96fced 100644 --- a/src/walkthrough.md +++ b/src/walkthrough.md @@ -258,6 +258,8 @@ After this, [a PR is made][stab] to remove the feature gate, enabling the featur default (on the 2018 edition). A note is added to the [Release notes][relnotes] about the feature. +[stab]: https://github.com/rust-lang/rust/pull/56245 + Steps to stabilize the feature can be found at [Stabilizing Features](./stabilization_guide.md). [relnotes]: https://github.com/rust-lang/rust/blob/master/RELEASES.md From 2ecb52cb2dd088b994c03cf44cf2aa3a8095f70d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?I=C3=B1aki=20Garay?= Date: Fri, 3 May 2019 12:37:09 -0300 Subject: [PATCH 549/648] Added mention of universal ctags --- src/how-to-build-and-run.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 241852b19..6d5bcb916 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -300,11 +300,19 @@ in other sections: ### ctags -One of the challenges with rustc is that the RLS can't handle it, since it's a bootstrapping -compiler. This makes code navigation difficult. One solution is to use `ctags`. The following -script can be used to set it up: [https://github.com/nikomatsakis/rust-etags][etags]. +One of the challenges with rustc is that the RLS can't handle it, since it's a +bootstrapping compiler. This makes code navigation difficult. One solution is to +use `ctags`. -CTAGS integrates into emacs and vim quite easily. The following can then be +`ctags` has a long history and several variants. Exhuberant CTags seems to be +quite commonly distributed but it does not have out-of-box Rust support. Some +distributions seem to use [Universal Ctags][utags], which is a maintained fork +and does have built-in Rust support. + +The following script can be used to set up Exhuberant Ctags: +[https://github.com/nikomatsakis/rust-etags][etags]. + +`ctags` integrates into emacs and vim quite easily. The following can then be used to build and generate tags: ```console @@ -315,6 +323,7 @@ This allows you to do "jump-to-def" with whatever functions were around when you last built, which is ridiculously useful. [etags]: https://github.com/nikomatsakis/rust-etags +[utags]: https://github.com/universal-ctags/ctags ### Cleaning out build directories From 2b8280f113c6715f047c6550f530c4064f796027 Mon Sep 17 00:00:00 2001 From: Yuki Okushi Date: Fri, 3 May 2019 15:45:04 +0900 Subject: [PATCH 550/648] Rename to RUSTC_LOG --- src/compiler-debugging.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index 2e45e5f02..db7f08e46 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -149,7 +149,7 @@ $ # Cool, now I have a backtrace for the error These crates are used in compiler for logging: * [log] -* [env-logger]: check the link to see the full `RUST_LOG` syntax +* [env-logger]: check the link to see the full `RUSTC_LOG` syntax [log]: https://docs.rs/log/0.4.6/log/index.html [env-logger]: https://docs.rs/env_logger/0.4.3/env_logger/ @@ -159,9 +159,9 @@ at many points. These are very useful to at least narrow down the location of a bug if not to find it entirely, or just to orient yourself as to why the compiler is doing a particular thing. -To see the logs, you need to set the `RUST_LOG` environment variable to +To see the logs, you need to set the `RUSTC_LOG` environment variable to your log filter, e.g. to get the logs for a specific module, you can run the -compiler as `RUST_LOG=module::path rustc my-file.rs`. All `debug!` output will +compiler as `RUSTC_LOG=module::path rustc my-file.rs`. All `debug!` output will then appear in standard error. **Note that unless you use a very strict filter, the logger will emit a lot of @@ -174,16 +174,16 @@ So to put it together. ```bash # This puts the output of all debug calls in `librustc/traits` into # standard error, which might fill your console backscroll. -$ RUST_LOG=rustc::traits rustc +local my-file.rs +$ RUSTC_LOG=rustc::traits rustc +local my-file.rs # This puts the output of all debug calls in `librustc/traits` in # `traits-log`, so you can then see it with a text editor. -$ RUST_LOG=rustc::traits rustc +local my-file.rs 2>traits-log +$ RUSTC_LOG=rustc::traits rustc +local my-file.rs 2>traits-log # Not recommended. This will show the output of all `debug!` calls # in the Rust compiler, and there are a *lot* of them, so it will be # hard to find anything. -$ RUST_LOG=debug rustc +local my-file.rs 2>all-log +$ RUSTC_LOG=debug rustc +local my-file.rs 2>all-log # This will show the output of all `info!` calls in `rustc_trans`. # @@ -192,7 +192,7 @@ $ RUST_LOG=debug rustc +local my-file.rs 2>all-log # which function triggers an LLVM assertion, and this is an `info!` # log rather than a `debug!` log so it will work on the official # compilers. -$ RUST_LOG=rustc_trans=info rustc +local my-file.rs +$ RUSTC_LOG=rustc_trans=info rustc +local my-file.rs ``` ### How to keep or remove `debug!` and `trace!` calls from the resulting binary @@ -201,7 +201,7 @@ While calls to `error!`, `warn!` and `info!` are included in every build of the calls to `debug!` and `trace!` are only included in the program if `debug-assertions=yes` is turned on in config.toml (it is turned off by default), so if you don't see `DEBUG` logs, especially -if you run the compiler with `RUST_LOG=rustc rustc some.rs` and only see +if you run the compiler with `RUSTC_LOG=rustc rustc some.rs` and only see `INFO` logs, make sure that `debug-assertions=yes` is turned on in your config.toml. @@ -230,7 +230,7 @@ If in the module `rustc::foo` you have a statement debug!("{:?}", random_operation(tcx)); ``` -Then if someone runs a debug `rustc` with `RUST_LOG=rustc::bar`, then +Then if someone runs a debug `rustc` with `RUSTC_LOG=rustc::bar`, then `random_operation()` will run. This means that you should not put anything too expensive or likely to crash From 3cb727b62b953d59b4360d39aa68b6dc8f157655 Mon Sep 17 00:00:00 2001 From: Tom Mulvaney Date: Wed, 17 Apr 2019 18:29:11 +0100 Subject: [PATCH 551/648] Fix typo, 'which' repeated twice --- src/traits/chalk-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/traits/chalk-overview.md b/src/traits/chalk-overview.md index 4d1260ddc..54f6c3394 100644 --- a/src/traits/chalk-overview.md +++ b/src/traits/chalk-overview.md @@ -122,7 +122,7 @@ This rule dictates that `Vec: Clone` is only satisfied if `T: Clone` is also satisfied (i.e. "provable"). Similar to [`chalk::program::Program`][chalk-program] which has "rust-like -things", chalk_ir defines [`ProgramEnvironment`] which which is "pure logic". +things", chalk_ir defines [`ProgramEnvironment`] which is "pure logic". The main field in that struct is `program_clauses`, which contains the [`ProgramClause`]s generated by the rules module. From 243863e48700f35641d3f39b0a9f666e9bbd2444 Mon Sep 17 00:00:00 2001 From: Michael Woerister Date: Tue, 14 May 2019 13:10:04 +0200 Subject: [PATCH 552/648] Add documentation about profile-guided optimization. --- src/SUMMARY.md | 1 + src/profile-guided-optimization.md | 132 +++++++++++++++++++++++++++++ 2 files changed, 133 insertions(+) create mode 100644 src/profile-guided-optimization.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 477b9a707..c33d15f3c 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -85,6 +85,7 @@ - [Debugging LLVM](./codegen/debugging.md) - [Emitting Diagnostics](./diag.md) - [JSON diagnostic format](./diag/json-format.md) +- [Profile-guided Optimization](./profile-guided-optimization.md) --- diff --git a/src/profile-guided-optimization.md b/src/profile-guided-optimization.md new file mode 100644 index 000000000..fb897e901 --- /dev/null +++ b/src/profile-guided-optimization.md @@ -0,0 +1,132 @@ +# Profile Guided Optimization + +`rustc` supports doing profile-guided optimization (PGO). +This chapter describes what PGO is and how the support for it is +implemented in `rustc`. + +## What Is Profiled-Guided Optimization? + +The basic concept of PGO is to collect data about the typical execution of +a program (e.g. which branches it is likely to take) and then use this data +to inform optimizations such as inlining, machine-code layout, +register allocation, etc. + +There are different ways of collecting data about a program's execution. +One is to run the program inside a profiler (such as `perf`) and another +is to create an instrumented binary, that is, a binary that has data +collection built into it, and run that. +The latter usually provides more accurate data. + +## How is PGO implemented in `rustc`? + +`rustc` current PGO implementation relies entirely on LLVM. +LLVM actually [supports multiple forms][clang-pgo] of PGO: + +[clang-pgo]: https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization + +- Sampling-based PGO where an external profiling tool like `perf` is used + to collect data about a program's execution. +- GCOV-based profiling, where code coverage infrastructure is used to collect + profiling information. +- Front-end based instrumentation, where the compiler front-end (e.g. Clang) + inserts instrumentation intrinsics into the LLVM IR it generates. +- IR-level instrumentation, where LLVM inserts the instrumentation intrinsics + itself during optimization passes. + +`rustc` supports only the last approach, IR-level instrumentation, mainly +because it is almost exclusively implemented in LLVM and needs little +maintenance on the Rust side. Fortunately, it is also the most modern approach, +yielding the best results. + +So, we are dealing with an instrumentation-based approach, i.e. profiling data +is generated by a specially instrumented version of the program that's being +optimized. Instrumentation-based PGO has two components: a compile-time +component and run-time component, and one needs to understand the overall +workflow to see how they interact. + +### Overall Workflow + +Generating a PGO-optimized program involves the following four steps: + +1. Compile the program with instrumentation enabled (e.g. `rustc -Cprofile-generate main.rs`) +2. Run the instrumented program (e.g. `./main`) which generates a `default-.profraw` file +3. Convert the `.profraw` file into a `.profdata` file using LLVM's `llvm-profdata` tool. +4. Compile the program again, this time making use of the profiling data + (e.g. `rustc -Cprofile-use=merged.profdata main.rs`) + +### Compile-Time Aspects + +Depending on which step in the above workflow we are in, two different things +can happen at compile time: + +#### Create Binaries with Instrumentation + +As mentioned above, the profiling instrumentation is added by LLVM. +`rustc` instructs LLVM to do so [by setting the appropriate][pgo-gen-passmanager] +flags when creating LLVM `PassManager`s: + +```C + // `PMBR` is an `LLVMPassManagerBuilderRef` + unwrap(PMBR)->EnablePGOInstrGen = true; + // Instrumented binaries have a default output path for the `.profraw` file + // hard-coded into them: + unwrap(PMBR)->PGOInstrGen = PGOGenPath; +``` + +`rustc` also has to make sure that some of the symbols from LLVM's profiling +runtime are not removed [by marking the with the right export level][pgo-gen-symbols]. + +[pgo-gen-passmanager]: https://github.com/rust-lang/rust/blob/1.34.1/src/rustllvm/PassWrapper.cpp#L412-L416 +[pgo-gen-symbols]:https://github.com/rust-lang/rust/blob/1.34.1/src/librustc_codegen_ssa/back/symbol_export.rs#L212-L225 + + +#### Compile Binaries Where Optimizations Make Use Of Profiling Data + +In the final step of the workflow described above, the program is compiled +again, with the compiler using the gathered profiling data in order to drive +optimization decisions. `rustc` again leaves most of the work to LLVM here, +basically [just telling][pgo-use-passmanager] the LLVM `PassManagerBuilder` +where the profiling data can be found: + +```C + unwrap(PMBR)->PGOInstrUse = PGOUsePath; +``` + +[pgo-use-passmanager]: https://github.com/rust-lang/rust/blob/1.34.1/src/rustllvm/PassWrapper.cpp#L417-L420 + +LLVM does the rest (e.g. setting branch weights, marking functions with +`cold` or `inlinehint`, etc). + + +### Runtime Aspects + +Instrumentation-based approaches always also have a runtime component, i.e. +once we have an instrumented program, that program needs to be run in order +to generate profiling data, and collecting and persisting this profiling +data needs some infrastructure in place. + +In the case of LLVM, these runtime components are implemented in +[compiler-rt][compiler-rt-profile] and statically linked into any instrumented +binaries. +The `rustc` version of this can be found in `src/libprofiler_builtins` which +basically packs the C code from `compiler-rt` into a Rust crate. + +In order for `libprofiler_builtins` to be built, `profiler = true` must be set +in `rustc`'s `config.toml`. + +[compiler-rt-profile]: https://github.com/llvm/llvm-project/tree/master/compiler-rt/lib/profile + +## Testing PGO + +Since the PGO workflow spans multiple compiler invocations most testing happens +in [run-make tests][rmake-tests] (the relevant tests have `pgo` in their name). +There is also a [codegen test][codegen-test] that checks that some expected +instrumentation artifacts show up in LLVM IR. + +[rmake-tests]: https://github.com/rust-lang/rust/tree/master/src/test/run-make-fulldeps +[codegen-test]: https://github.com/rust-lang/rust/blob/master/src/test/codegen/pgo-instrumentation.rs + +## Additional Information + +Clang's documentation contains a good overview on PGO in LLVM here: +https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization From 2972cf43a871c6df1b5c4c2efb5f27e7a9dfcb15 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?I=C3=B1aki=20Garay?= Date: Wed, 15 May 2019 11:21:26 -0300 Subject: [PATCH 553/648] Fixed links broken by merging chalks rules and solve --- src/traits/chalk-overview.md | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/src/traits/chalk-overview.md b/src/traits/chalk-overview.md index 54f6c3394..0a95585f6 100644 --- a/src/traits/chalk-overview.md +++ b/src/traits/chalk-overview.md @@ -126,10 +126,10 @@ things", chalk_ir defines [`ProgramEnvironment`] which is "pure logic". The main field in that struct is `program_clauses`, which contains the [`ProgramClause`]s generated by the rules module. -### Rules ([chalk_rules]) +### Rules ([chalk_solve]) -The `chalk_rules` crate ([source code][chalk_rules]) defines the logic rules we use -for each item in the Rust IR. It works by iterating over every trait, impl, +The `chalk_solve` crate ([source code][chalk_solve]) defines the logic rules we +use for each item in the Rust IR. It works by iterating over every trait, impl, etc. and emitting the rules that come from each one. *See also: [Lowering Rules][lowering-rules]* @@ -137,7 +137,7 @@ etc. and emitting the rules that come from each one. #### Well-formedness checks As part of lowering to logic, we also do some "well formedness" checks. See -the [`chalk_rules::wf` source code][rules-wf-src] for where those are done. +the [`chalk_solve::wf` source code][solve-wf-src] for where those are done. *See also: [Well-formedness checking][wf-checking]* @@ -163,12 +163,11 @@ Chalk's functionality is broken up into the following crates: - [**chalk_ir**][chalk_ir]: Defines chalk's internal representation of types, lifetimes, and goals. - [**chalk_solve**][chalk_solve]: Combines `chalk_ir` and `chalk_engine`, - effectively. + effectively, which implements logic rules converting `chalk_rust_ir` to + `chalk_ir` + - Defines the `coherence` module, which implements coherence rules - [`chalk_engine::context`][engine-context] provides the necessary hooks. - [**chalk_parse**][chalk_parse]: Defines the raw AST and a parser. -- [**chalk_rules**][chalk_rules]: which implements logic rules converting - `chalk_rust_ir` to `chalk_ir` - - Defines the `coherence` module, which implements coherence rules - [**chalk**][doc-chalk]: Brings everything together. Defines the following modules: - `chalk::lowering`, which converts AST to `chalk_rust_ir` @@ -236,7 +235,6 @@ Likewise, lowering tests use the [`lowering_success!` and [chalk_ir]: https://rust-lang.github.io/chalk/doc/chalk_ir/index.html [chalk_parse]: https://rust-lang.github.io/chalk/doc/chalk_parse/index.html [chalk_solve]: https://rust-lang.github.io/chalk/doc/chalk_solve/index.html -[chalk_rules]: https://rust-lang.github.io/chalk/doc/chalk_rules/index.html [chalk_rust_ir]: https://rust-lang.github.io/chalk/doc/chalk_rust_ir/index.html [doc-chalk]: https://rust-lang.github.io/chalk/doc/chalk/index.html [engine-context]: https://rust-lang.github.io/chalk/doc/chalk_engine/context/index.html @@ -250,9 +248,9 @@ Likewise, lowering tests use the [`lowering_success!` and [chalk-test-wf]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/rules/wf/test.rs#L1 [chalki]: https://rust-lang.github.io/chalk/doc/chalki/index.html [clause]: https://github.com/rust-lang/chalk/blob/master/GLOSSARY.md#clause -[coherence-src]: http://rust-lang.github.io/chalk/doc/chalk_rules/coherence/index.html +[coherence-src]: http://rust-lang.github.io/chalk/doc/chalk_solve/coherence/index.html [ir-code]: http://rust-lang.github.io/chalk/doc/chalk_rust_ir/ -[rules-wf-src]: http://rust-lang.github.io/chalk/doc/chalk_rules/wf/index.html +[solve-wf-src]: http://rust-lang.github.io/chalk/doc/chalk_solve/wf/index.html [solve_goal]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L85 [test-lowering-macros]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test_util.rs#L21-L54 [test-macro]: https://github.com/rust-lang/chalk/blob/4bce000801de31bf45c02f742a5fce335c9f035f/src/test.rs#L33 From 72bafaca4f1b958a471d178c0483e7390e63384e Mon Sep 17 00:00:00 2001 From: varkor Date: Wed, 15 May 2019 08:58:20 +0100 Subject: [PATCH 554/648] Mention running tests for subdirectories --- src/how-to-build-and-run.md | 4 ++++ src/tests/running.md | 21 ++++++++++++++++++--- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 6d5bcb916..9b205c7b1 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -297,6 +297,10 @@ in other sections: more details): - `./x.py test --stage 1 src/libstd` – runs the `#[test]` tests from libstd - `./x.py test --stage 1 src/test/run-pass` – runs the `run-pass` test suite + - `./x.py test --stage 1 src/test/ui/const-generics` - runs all the tests in + the `const-generics/` subdirectory of the `ui` test suite + - `./x.py test --stage 1 src/test/ui/const-generics/const-types.rs` - runs + the single test `const-types.rs` from the `ui` test suite ### ctags diff --git a/src/tests/running.md b/src/tests/running.md index 1250fa77b..5a05c156f 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -47,6 +47,19 @@ the debuginfo test suite: > ./x.py test --stage 1 src/test/debuginfo ``` +If you only need to test a specific subdirectory of tests for any +given test suite, you can pass that directory to `x.py test`: + +```bash +> ./x.py test --stage 1 src/test/ui/const-generics +``` + +Likewise, you can test a single file by passing its path: + +```bash +> ./x.py test --stage 1 src/test/ui/const-generics/const-test.rs +``` + ### Run only the tidy script ```bash @@ -82,8 +95,9 @@ work well with procedural macros or custom derive tests. ## Running an individual test Another common thing that people want to do is to run an **individual -test**, often the test they are trying to fix. One way to do this is -to invoke `x.py` with the `--test-args` option: +test**, often the test they are trying to fix. As mentioned earlier, +you may pass the full file path to achieve this, or alternatively one +may invoke `x.py` with the `--test-args` option: ```bash > ./x.py test --stage 1 src/test/ui --test-args issue-1234 @@ -91,7 +105,8 @@ to invoke `x.py` with the `--test-args` option: Under the hood, the test runner invokes the standard rust test runner (the same one you get with `#[test]`), so this command would wind up -filtering for tests that include "issue-1234" in the name. +filtering for tests that include "issue-1234" in the name. (Thus +`--test-args` is a good way to run a collection of related tests.) ## Using incremental compilation From 673b4a4b4ca04d9ec1ec57222da82189dad7ffaa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?I=C3=B1aki=20Garay?= Date: Wed, 17 Apr 2019 10:44:59 -0300 Subject: [PATCH 555/648] Added rustc phases diagram and explanation Added rustc build phase diagram and explanation --- src/SUMMARY.md | 2 +- src/how-to-build-and-run.md | 270 ++++++++++++++++++++++++++++++------ src/img/rustc_stages.svg | 2 + 3 files changed, 230 insertions(+), 44 deletions(-) create mode 100644 src/img/rustc_stages.svg diff --git a/src/SUMMARY.md b/src/SUMMARY.md index c33d15f3c..ca8f4adaa 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -6,7 +6,7 @@ - [Part 1: Building, debugging, and contributing to Rustc](./part-1-intro.md) - [About the compiler team](./compiler-team.md) -- [How to build the compiler and run what you built](./how-to-build-and-run.md) +- [How to Build and Run the Compiler](./how-to-build-and-run.md) - [Build and Install distribution artifacts](./build-install-distribution-artifacts.md) - [Documenting Compiler](./compiler-documenting.md) - [The compiler testing framework](./tests/intro.md) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 9b205c7b1..4eb4023c4 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -1,4 +1,4 @@ -# How to build the compiler and run what you built +# How to Build and Run the Compiler The compiler is built using a tool called `x.py`. You will need to have Python installed to run it. But before we get to that, if you're going to @@ -29,7 +29,7 @@ assertions = true [rust] # This enables some assertions, but more importantly it enables the `debug!` -# logging macros that are essential for debugging rustc. +# logging macros that are essential for debugging `rustc`. debug-assertions = true # This will make your build more parallel; it costs a bit of runtime @@ -43,31 +43,39 @@ debuginfo = true debuginfo-lines = true ``` -### What is x.py? +### What is `x.py`? -x.py is the script used to orchestrate the tooling in the rustc repository. -It is the script that can build docs, run tests, and compile rustc. -It is the now preferred way to build rustc and it replaces the old makefiles -from before. Below are the different ways to utilize x.py in order to +`x.py` is the script used to orchestrate the tooling in the `rustc` repository. +It is the script that can build docs, run tests, and compile `rustc`. +It is the now preferred way to build `rustc` and it replaces the old makefiles +from before. Below are the different ways to utilize `x.py` in order to effectively deal with the repo for various common tasks. -### Running x.py and building a stage1 compiler +### Running `x.py` and building a stage1 compiler One thing to keep in mind is that `rustc` is a _bootstrapping_ compiler. That is, since `rustc` is written in Rust, we need to use an older version of the compiler to compile the newer version. In -particular, the newer version of the compiler, `libstd`, and other -tooling may use some unstable features internally. The result is that -compiling `rustc` is done in stages: - -- **Stage 0:** the stage0 compiler is usually the current _beta_ compiler - (`x.py` will download it for you); you can configure `x.py` to use something - else, though. +particular, the newer version of the compiler and some of the artifacts needed +to build it, such as `libstd` and other tooling, may use some unstable features +internally, requiring a specific version which understands these unstable +features. + +The result is that compiling `rustc` is done in stages: + +- **Stage 0:** the stage0 compiler is usually (you can configure `x.py` to use + something else) the current _beta_ `rustc` compiler and its associated dynamic + libraries (which `x.py` will download for you). This stage0 compiler is then + used only to compile `rustbuild`, `std`, `test`, and `rustc`. When compiling + `test` and `rustc`, this stage0 compiler uses the freshly compiled `std`. + There are two concepts at play here: a compiler (with its set of dependencies) + and its 'target' or 'object' libraries (`std`, `test`, and `rustc`). + Both are staged, but in a staggered manner. - **Stage 1:** the code in your clone (for new version) is then compiled with the stage0 compiler to produce the stage1 compiler. However, it was built with an older compiler (stage0), so to - optimize the stage1 compiler we go to next stage. - - (In theory, the stage1 compiler is functionally identical to the + optimize the stage1 compiler we go to next the stage. + - In theory, the stage1 compiler is functionally identical to the stage2 compiler, but in practice there are subtle differences. In particular, the stage1 compiler itself was built by stage0 and hence not by the source in your working directory: this means that @@ -75,22 +83,197 @@ compiling `rustc` is done in stages: symbol names that would have been made by the stage1 compiler. This can be important when using dynamic linking (e.g., with derives. Sometimes this means that some tests don't work when run - with stage1.) + with stage1. - **Stage 2:** we rebuild our stage1 compiler with itself to produce the stage2 compiler (i.e. it builds itself) to have all the _latest optimizations_. (By default, we copy the stage1 libraries for use by the stage2 compiler, since they ought to be identical.) -- _(Optional)_ **Stage 3**: to sanity check of our new compiler, we +- _(Optional)_ **Stage 3**: to sanity check our new compiler, we can build the libraries with the stage2 compiler. The result ought to be identical to before, unless something has broken. +#### A note on stage meanings + +When running `x.py` you will see output such as: + +```txt +Building stage0 std artifacts +Copying stage0 std from stage0 +Building stage0 test artifacts +Copying stage0 test from stage0 +Building stage0 compiler artifacts +Copying stage0 rustc from stage0 +Building LLVM for x86_64-apple-darwin +Building stage0 codegen artifacts +Assembling stage1 compiler +Building stage1 std artifacts +Copying stage1 std from stage1 +Building stage1 test artifacts +Copying stage1 test from stage1 +Building stage1 compiler artifacts +Copying stage1 rustc from stage1 +Building stage1 codegen artifacts +Assembling stage2 compiler +Uplifting stage1 std +Copying stage2 std from stage1 +Generating unstable book md files +Building stage0 tool unstable-book-gen +Building stage0 tool rustbook +Documenting standalone +Building rustdoc for stage2 +Documenting book redirect pages +Documenting stage2 std +Building rustdoc for stage1 +Documenting stage2 test +Documenting stage2 whitelisted compiler +Documenting stage2 compiler +Documenting stage2 rustdoc +Documenting error index +Uplifting stage1 test +Copying stage2 test from stage1 +Uplifting stage1 rustc +Copying stage2 rustc from stage1 +Building stage2 tool error_index_generator +``` + +A deeper look into `x.py`'s phases can be seen here: + +A diagram of the rustc compilation phases + +Keep in mind this diagram is a simplification, i.e. `rustdoc` can be built at +different stages, the process is a bit different when passing flags such as +`--keep-stage`, or if there are non-host targets. + +The following tables indicate the outputs of various stage actions: + +| Stage 0 Action | Output | +|-----------------------------------------------------------|----------------------------------------------| +| `beta` extracted | `build/HOST/stage0` | +| `stage0` builds `bootstrap` | `build/bootstrap` | +| `stage0` builds `libstd` | `build/HOST/stage0-std/TARGET` | +| copy `stage0-std` (HOST only) | `build/HOST/stage0-sysroot/lib/rustlib/HOST` | +| `stage0` builds `libtest` with `stage0-sysroot` | `build/HOST/stage0-test/TARGET` | +| copy `stage0-test` (HOST only) | `build/HOST/stage0-sysroot/lib/rustlib/HOST` | +| `stage0` builds `rustc` with `stage0-sysroot` | `build/HOST/stage0-rustc/HOST` | +| copy `stage0-rustc (except executable)` | `build/HOST/stage0-sysroot/lib/rustlib/HOST` | +| build `llvm` | `build/HOST/llvm` | +| `stage0` builds `codegen` with `stage0-sysroot` | `build/HOST/stage0-codgen/HOST` | +| `stage0` builds `rustdoc` with `stage0-sysroot` | `build/HOST/stage0-tools/HOST` | + +`--stage=0` stops here. + +| Stage 1 Action | Output | +|-----------------------------------------------------|---------------------------------------| +| copy (uplift) `stage0-rustc` executable to `stage1` | `build/HOST/stage1/bin` | +| copy (uplift) `stage0-codegen` to `stage1` | `build/HOST/stage1/lib` | +| copy (uplift) `stage0-sysroot` to `stage1` | `build/HOST/stage1/lib` | +| `stage1` builds `libstd` | `build/HOST/stage1-std/TARGET` | +| copy `stage1-std` (HOST only) | `build/HOST/stage1/lib/rustlib/HOST` | +| `stage1` builds `libtest` | `build/HOST/stage1-test/TARGET` | +| copy `stage1-test` (HOST only) | `build/HOST/stage1/lib/rustlib/HOST` | +| `stage1` builds `rustc` | `build/HOST/stage1-rustc/HOST` | +| copy `stage1-rustc` (except executable) | `build/HOST/stage1/lib/rustlib/HOST` | +| `stage1` builds `codegen` | `build/HOST/stage1-codegen/HOST` | + +`--stage=1` stops here. + +| Stage 2 Action | Output | +|-------------------------------------------|-----------------------------------------------------------------| +| copy (uplift) `stage1-rustc` executable | `build/HOST/stage2/bin` | +| copy (uplift) `stage1-sysroot` | `build/HOST/stage2/lib and build/HOST/stage2/lib/rustlib/HOST` | +| `stage2` builds `libstd` (except HOST?) | `build/HOST/stage2-std/TARGET` | +| copy `stage2-std` (not HOST targets) | `build/HOST/stage2/lib/rustlib/TARGET` | +| `stage2` builds `libtest` (except HOST?) | `build/HOST/stage2-test/TARGET` | +| copy `stage2-test` (not HOST targets) | `build/HOST/stage2/lib/rustlib/TARGET` | +| `stage2` builds `rustdoc` | `build/HOST/stage2-tools/HOST` | +| copy `rustdoc` | `build/HOST/stage2/bin` | + +`--stage=2` stops here. + +Note that the convention `x.py` uses is that: +- A "stage N artifact" is an artifact that is _produced_ by the stage N compiler. +- The "stage (N+1) compiler" is assembled from "stage N artifacts". +- A `--stage N` flag means build _with_ stage N. + +In short, _stage 0 uses the stage0 compiler to create stage0 artifacts which +will later be uplifted to stage1_. + +Every time any of the main artifacts (`std`, `test`, `rustc`) are compiled, two +steps are performed. +When `std` is compiled by a stage N compiler, that `std` will be linked to +programs built by the stage N compiler (including test and `rustc` built later +on). It will also be used by the stage (N+1) compiler to link against itself. +This is somewhat intuitive if one thinks of the stage (N+1) compiler as "just" +another program we are building with the stage N compiler. In some ways, `rustc` +(the binary, not the `rustbuild` step) could be thought of as one of the few +`no_core` binaries out there. + +So "stage0 std artifacts" are in fact the output of the downloaded stage0 +compiler, and are going to be used for anything built by the stage0 compiler: +e.g. `rustc`, `test` artifacts. When it announces that it is "building stage1 +std artifacts" it has moved on to the next bootstrapping phase. This pattern +continues in latter stages. + +Also note that building host `std` and target `std` are different based on the +stage (e.g. see in the table how stage2 only builds non-host `std` targets. +This is because during stage2, the host `std` is uplifted from the "stage 1" +`std` -- specifically, when "Building stage 1 artifacts" is announced, it is +later copied into stage2 as well (both the compiler's `libdir` and the +`sysroot`). + +This `std` is pretty much necessary for any useful work with the compiler. +Specifically, it's used as the `std` for programs compiled by the newly compiled +compiler (so when you compile `fn main() { }` it is linked to the last `std` +compiled with `x.py build --stage 1 src/libstd`). + +The `rustc` generated by the stage0 compiler is linked to the freshly-built +`libstd`, which means that for the most part only `std` needs to be cfg-gated, +so that `rustc` can use featured added to std immediately after their addition, +without need for them to get into the downloaded beta. The `libstd` built by the +`stage1/bin/rustc` compiler, also known as "stage1 std artifacts", is not +necessarily ABI-compatible with that compiler. +That is, the `rustc` binary most likely could not use this `std` itself. +It is however ABI-compatible with any programs that the `stage1/bin/rustc` +binary builds (including itself), so in that sense they're paired. + +This is also where `--keep-stage 1 src/libstd` comes into play. Since most +changes to the compiler don't actually change the ABI, once you've produced a +`libstd` in stage 1, you can probably just reuse it with a different compiler. +If the ABI hasn't changed, you're good to go, no need to spend the time +recompiling that `std`. +`--keep-stage` simply assumes the previous compile is fine and copies those +artifacts into the appropriate place, skipping the cargo invocation. + +The reason we first build `std`, then `test`, then `rustc`, is largely just +because we want to minimize `cfg(stage0)` in the code for `rustc`. +Currently `rustc` is always linked against a "new" `std`/`test` so it doesn't +ever need to be concerned with differences in std; it can assume that the std is +as fresh as possible. + +The reason we need to build it twice is because of ABI compatibility. +The beta compiler has it's own ABI, and then the `stage1/bin/rustc` compiler +will produce programs/libraries with the new ABI. +We used to build three times, but because we assume that the ABI is constant +within a codebase, we presume that the libraries produced by the "stage2" +compiler (produced by the `stage1/bin/rustc` compiler) is ABI-compatible with +the `stage1/bin/rustc` compiler's produced libraries. +What this means is that we can skip that final compilation -- and simply use the +same libraries as the `stage2/bin/rustc` compiler uses itself for programs it +links against. + +This `stage2/bin/rustc` compiler is shipped to end-users, along with the +`stage 1 {std,test,rustc}` artifacts. + +If you want to learn more about `x.py`, read its README.md +[here](https://github.com/rust-lang/rust/blob/master/src/bootstrap/README.md). + #### Build Flags -There are other flags you can pass to the build portion of x.py that can be +There are other flags you can pass to the build command of `x.py` that can be beneficial to cutting down compile times or fitting other things you might need to change. They are: -```bash +```txt Options: -v, --verbose use verbose output (-vv for very verbose) -i, --incremental use incremental compilation @@ -127,16 +310,17 @@ probably the best "go to" command for building a local rust: This may *look* like it only builds libstd, but that is not the case. What this command does is the following: -- Build libstd using the stage0 compiler (using incremental) -- Build librustc using the stage0 compiler (using incremental) +- Build `libstd` using the stage0 compiler (using incremental) +- Build `librustc` using the stage0 compiler (using incremental) - This produces the stage1 compiler - Build libstd using the stage1 compiler (cannot use incremental) This final product (stage1 compiler + libs built using that compiler) -is what you need to build other rust programs. +is what you need to build other rust programs (unless you use `#![no_std]` or +`#![no_core]`). -Note that the command includes the `-i` switch. This enables incremental -compilation. This will be used to speed up the first two steps of the process: +The command includes the `-i` switch which enables incremental compilation. +This will be used to speed up the first two steps of the process: in particular, if you make a small change, we ought to be able to use your old results to make producing the stage1 **compiler** faster. @@ -145,15 +329,15 @@ stage1 libraries. This is because incremental only works when you run the *same compiler* twice in a row. In this case, we are building a *new stage1 compiler* every time. Therefore, the old incremental results may not apply. **As a result, you will probably find that -building the stage1 libstd is a bottleneck for you** -- but fear not, +building the stage1 `libstd` is a bottleneck for you** -- but fear not, there is a (hacky) workaround. See [the section on "recommended workflows"](#workflow) below. -Note that this whole command just gives you a subset of the full rustc -build. The **full** rustc build (what you get if you just say `./x.py +Note that this whole command just gives you a subset of the full `rustc` +build. The **full** `rustc` build (what you get if you just say `./x.py build`) has quite a few more steps: -- Build librustc and rustc with the stage1 compiler. +- Build `librustc` and `rustc` with the stage1 compiler. - The resulting compiler here is called the "stage2" compiler. - Build libstd with stage2 compiler. - Build librustdoc and a bunch of other things with the stage2 compiler. @@ -185,12 +369,11 @@ compile. Using these commands you can test that it compiles before doing a bigger build to make sure it works with the compiler. As shown before you can also pass flags at the end such as --stage. - ### Creating a rustup toolchain -Once you have successfully built rustc, you will have created a bunch +Once you have successfully built `rustc`, you will have created a bunch of files in your `build` directory. In order to actually run the -resulting rustc, we recommend creating rustup toolchains. The first +resulting `rustc`, we recommend creating rustup toolchains. The first one will run the stage1 compiler (which we built above). The second will execute the stage2 compiler (which we did not build, but which you will likely need to build at some point; for example, if you want @@ -207,7 +390,7 @@ The `` would typically be one of the following: - Mac: `x86_64-apple-darwin` - Windows: `x86_64-pc-windows-msvc` -Now you can run the rustc you built with. If you run with `-vV`, you +Now you can run the `rustc` you built with. If you run with `-vV`, you should see a version number ending in `-dev`, indicating a build from your local environment: @@ -226,8 +409,7 @@ LLVM version: 4.0 ### Suggested workflows for faster builds of the compiler -There are two workflows that are useful for faster builds of the -compiler. +There are two workflows that are useful for faster builds of the compiler. **Check, check, and check again.** The first workflow, which is useful when doing simple refactorings, is to run `./x.py check` @@ -257,12 +439,15 @@ The sequence of commands you want is as follows: - Initial build: `./x.py build -i --stage 1 src/libstd` - As [documented above](#command), this will build a functional - stage1 compiler + stage1 compiler as part of running all stage0 commands (which include + building a `libstd` compatible with the stage1 compiler) as well as the + first few steps of the "stage 1 actions" up to "stage1 (sysroot stage1) + builds libstd". - Subsequent builds: `./x.py build -i --stage 1 src/libstd --keep-stage 1` - Note that we added the `--keep-stage 1` flag here -The effect of `--keep-stage 1` is that we just *assume* that the old -standard library can be re-used. If you are editing the compiler, this +As mentioned, the effect of `--keep-stage 1` is that we just *assume* that the +old standard library can be re-used. If you are editing the compiler, this is almost always true: you haven't changed the standard library, after all. But sometimes, it's not true: for example, if you are editing the "metadata" part of the compiler, which controls how the compiler @@ -276,15 +461,14 @@ using `--keep-stage 1`** -- for example, strange should simply remove the `--keep-stage 1` from the command and rebuild. That ought to fix the problem. -You can also use `--keep-stage 1` when running tests. Something like -this: +You can also use `--keep-stage 1` when running tests. Something like this: - Initial test run: `./x.py test -i --stage 1 src/test/ui` - Subsequent test run: `./x.py test -i --stage 1 src/test/ui --keep-stage 1` -### Other x.py commands +### Other `x.py` commands -Here are a few other useful x.py commands. We'll cover some of them in detail +Here are a few other useful `x.py` commands. We'll cover some of them in detail in other sections: - Building things: diff --git a/src/img/rustc_stages.svg b/src/img/rustc_stages.svg new file mode 100644 index 000000000..b6671aa7c --- /dev/null +++ b/src/img/rustc_stages.svg @@ -0,0 +1,2 @@ + +
Download
Download
beta
beta
stage0
stage0
Builds
Builds
stage0
stage0
Outputs
Outputs
libstd
libstd
Copy
Copy
stage0-std
stage0-std
stage0-sysroot
stage0-sysroot
Builds
Builds
stage0
stage0
Outputs
Outputs
libtest
libtest
Copy
Copy
stage0-test
stage0-test
stage0-sysroot
stage0-sysroot
Builds
Builds
stage0
stage0
Outputs
Outputs
rustc
rustc
Copy
Copy
stage0-rustc
stage0-rustc
stage0-sysroot
stage0-sysroot
llvm
llvm
Builds
Builds
stage0
stage0
Outputs
Outputs
codegen
codegen
stage0-codegen
stage0-codegen
Stage 0
Stage 0
Copy
Copy
stage0-rustc
stage0-codegen
stage0-sysroot
[Not supported by viewer]
stage1
stage1
Builds
Builds
stage1
stage1
Outputs
Outputs
libstd
libstd
Copy
Copy
stage1-std
stage1-std
stage1/lib/rustlib
stage1/lib/rustlib
Builds
Builds
stage1
stage1
Outputs
Outputs
libtest
libtest
Copy
Copy
stage1-test
stage1-test
stage1/lib/rustlib
stage1/lib/rustlib
Builds
Builds
stage1
stage1
Outputs
Outputs
rustc
rustc
Copy
Copy
stage1-rustc
stage1-rustc
stage1/lib/rustlib
stage1/lib/rustlib
Builds
Builds
stage1
stage1
Outputs
Outputs
codegen
codegen
stage1-codegen
stage1-codegen
Stage 1
[Not supported by viewer]
Copy
Copy
stage1-rustc
stage1-codegen
stage1/lib/rustlib
[Not supported by viewer]
stage2
stage2
Builds
Builds
stage2
stage2
Outputs
Outputs
rustdoc
rustdoc
Copy
[Not supported by viewer]
stage2-tools
stage2-tools
Builds
Builds
stage0
stage0
bootstrap
bootstrap
llvm
llvm
stage2
stage2
Stage 2
[Not supported by viewer] \ No newline at end of file From 14b243de252da65e26dc3f60676d288f8c2af8b6 Mon Sep 17 00:00:00 2001 From: Julian Wollersberger Date: Fri, 17 May 2019 17:38:29 +0200 Subject: [PATCH 556/648] Renamed the file and title of the diagnostics chapter. When I recently searched for infos in this guide on how to create a warning, I couldn't find any. Later I found it through #14. The reason was that I didn't know the term 'diagnostics' and that it is the collective term for errors, warnings and lints. Renaming the chapter to include the word 'error' should help. I think also including 'warning' in the title shouldn't be neccessary, because it's close enought. --- src/SUMMARY.md | 4 ++-- src/appendix/code-index.md | 2 +- src/{diag.md => diagnostics.md} | 2 +- src/{diag => diagnostics}/json-format.md | 0 src/implementing_new_features.md | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) rename src/{diag.md => diagnostics.md} (99%) rename src/{diag => diagnostics}/json-format.md (100%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index ca8f4adaa..3d5dd44b6 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -21,6 +21,8 @@ - [with the linux perf tool](./profiling/with_perf.md) - [Coding conventions](./conventions.md) - [crates.io Dependencies](./crates-io.md) +- [Emitting Errors and other Diagnostics](diagnostics.md) + - [JSON diagnostic format](diagnostics/json-format.md) --- @@ -83,8 +85,6 @@ - [Code Generation](./codegen.md) - [Updating LLVM](./codegen/updating-llvm.md) - [Debugging LLVM](./codegen/debugging.md) -- [Emitting Diagnostics](./diag.md) - - [JSON diagnostic format](./diag/json-format.md) - [Profile-guided Optimization](./profile-guided-optimization.md) --- diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index 3c71d6bba..222c9b2db 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -38,7 +38,7 @@ Item | Kind | Short description | Chapter | [Type checking]: ../type-checking.html [The `ty` modules]: ../ty.html [Rustdoc]: ../rustdoc.html -[Emitting Diagnostics]: ../diag.html +[Emitting Diagnostics]: ../diagnostics.html [Macro expansion]: ../macro-expansion.html [Name resolution]: ../name-resolution.html [Parameter Environment]: ../param_env.html diff --git a/src/diag.md b/src/diagnostics.md similarity index 99% rename from src/diag.md rename to src/diagnostics.md index 31268e9f6..f3d518764 100644 --- a/src/diag.md +++ b/src/diagnostics.md @@ -1,4 +1,4 @@ -# Emitting Diagnostics +# Emitting Errors and other Diagnostics A lot of effort has been put into making `rustc` have great error messages. This chapter is about how to emit compile errors and lints from the compiler. diff --git a/src/diag/json-format.md b/src/diagnostics/json-format.md similarity index 100% rename from src/diag/json-format.md rename to src/diagnostics/json-format.md diff --git a/src/implementing_new_features.md b/src/implementing_new_features.md index 8105e1474..6d1327fab 100644 --- a/src/implementing_new_features.md +++ b/src/implementing_new_features.md @@ -49,7 +49,7 @@ In some cases, a feature or bugfix might break some existing programs in some edge cases. In that case, you might want to do a crater run to assess the impact and possibly add a future-compatibility lint, similar to those used for -[edition-gated lints](./diag.md#edition-gated-lints). +[edition-gated lints](diagnostics.md#edition-gated-lints). ### Stability From 2755796a2926bb946fe794b42ea65bf664c5e9e2 Mon Sep 17 00:00:00 2001 From: Santiago Pastorino Date: Tue, 28 May 2019 17:13:08 -0300 Subject: [PATCH 557/648] Add more info subsection with links to forge and rustc api docs --- src/about-this-guide.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/src/about-this-guide.md b/src/about-this-guide.md index 30a3b8870..8376fb2a3 100644 --- a/src/about-this-guide.md +++ b/src/about-this-guide.md @@ -16,4 +16,15 @@ be found at the [GitHub repository]. If you find any mistakes in the guide, please file an issue about it, or even better, open a PR with a correction! +## Other places to find information + +You might also find the following sites useful: + +- [Rustc API docs] -- rustdoc documentation for the compiler +- [Forge] -- contains documentation about rust infrastructure, team procedures, and more +- [compiler-team] -- the home-base for the rust compiler team, with description of the team procedures, active working groups, and the team calendar. + [GitHub repository]: https://github.com/rust-lang/rustc-guide/ +[Rustc API docs]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ +[Forge]: https://forge.rust-lang.org/ +[compiler-team]: https://github.com/rust-lang/compiler-team/ From c2ced29a082e201fe51e974c62ff6ddb862ffb11 Mon Sep 17 00:00:00 2001 From: Vallentin Date: Thu, 30 May 2019 21:09:46 +0200 Subject: [PATCH 558/648] Fixed misspelling --- src/borrow_check/moves_and_initialization.md | 2 +- src/queries/query-evaluation-model-in-detail.md | 2 +- src/traits/slg.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/borrow_check/moves_and_initialization.md b/src/borrow_check/moves_and_initialization.md index d1530d6c0..043db2f53 100644 --- a/src/borrow_check/moves_and_initialization.md +++ b/src/borrow_check/moves_and_initialization.md @@ -7,7 +7,7 @@ figuring out where moves occur and tracking those. ## Initialization and moves From a user's perspective, initialization -- giving a variable some -value -- and moves -- transfering ownership to another place -- might +value -- and moves -- transferring ownership to another place -- might seem like distinct topics. Indeed, our borrow checker error messages often talk about them differently. But **within the borrow checker**, they are not nearly as separate. Roughly speaking, the borrow checker diff --git a/src/queries/query-evaluation-model-in-detail.md b/src/queries/query-evaluation-model-in-detail.md index 47412acef..7772f36be 100644 --- a/src/queries/query-evaluation-model-in-detail.md +++ b/src/queries/query-evaluation-model-in-detail.md @@ -204,7 +204,7 @@ OK as long as the mutation is not observable. This is achieved by two things: This is not an ideal setup because of the manual intervention needed, so it should be used sparingly and only when it is well known which queries might access a given result. In practice, however, stealing has not turned out to be -much of a maintainance burden. +much of a maintenance burden. To summarize: "Steal queries" break some of the rules in a controlled way. There are checks in place that make sure that nothing can go silently wrong. diff --git a/src/traits/slg.md b/src/traits/slg.md index 98547575d..948d3fc8a 100644 --- a/src/traits/slg.md +++ b/src/traits/slg.md @@ -147,7 +147,7 @@ are both represented with an index.) For each strand, we also optionally store a *selected subgoal*. This is the subgoal after the turnstile (`:-`) that we are currently trying -to prove in this strand. Initally, when a strand is first created, +to prove in this strand. Initially, when a strand is first created, there is no selected subgoal. [`ExClause`]: https://rust-lang.github.io/chalk/doc/chalk_engine/struct.ExClause.html From 52d423e1b3fe7e59d4a4fc910bbaee3b794114dd Mon Sep 17 00:00:00 2001 From: mark Date: Sun, 2 Jun 2019 19:28:14 -0500 Subject: [PATCH 559/648] fix long line --- src/about-this-guide.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/about-this-guide.md b/src/about-this-guide.md index 8376fb2a3..df060a208 100644 --- a/src/about-this-guide.md +++ b/src/about-this-guide.md @@ -22,7 +22,8 @@ You might also find the following sites useful: - [Rustc API docs] -- rustdoc documentation for the compiler - [Forge] -- contains documentation about rust infrastructure, team procedures, and more -- [compiler-team] -- the home-base for the rust compiler team, with description of the team procedures, active working groups, and the team calendar. +- [compiler-team] -- the home-base for the rust compiler team, with description + of the team procedures, active working groups, and the team calendar. [GitHub repository]: https://github.com/rust-lang/rustc-guide/ [Rustc API docs]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ From 3ac9cfc9c9ab2e366feebf18718112737f572352 Mon Sep 17 00:00:00 2001 From: Vadim Petrochenkov Date: Fri, 31 May 2019 13:54:39 +0400 Subject: [PATCH 560/648] Update information about debuginfo configuration --- src/compiler-debugging.md | 2 +- src/how-to-build-and-run.md | 8 +++----- src/profiling/with_perf.md | 8 ++++---- 3 files changed, 8 insertions(+), 10 deletions(-) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index db7f08e46..304f75d80 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -47,7 +47,7 @@ stack backtrace: ``` If you want line numbers for the stack trace, you can enable -`debuginfo-lines=true` or `debuginfo=true` in your config.toml and rebuild the +`debuginfo-level = 1` or `debuginfo-level = 2` in your config.toml and rebuild the compiler. Then the backtrace will look like this: ```text diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 4eb4023c4..d0ba0ea57 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -36,11 +36,9 @@ debug-assertions = true # performance perhaps (less inlining) but it's worth it. codegen-units = 0 -# I always enable full debuginfo, though debuginfo-lines is more important. -debuginfo = true - -# Gives you line numbers for backtraces. -debuginfo-lines = true +# I always enable full debuginfo, however the line debuginfo (which is a more important part +# and enables e.g. line numbers in backtraces) is enabled by `debuginfo-level = 1` too. +debuginfo-level = 2 ``` ### What is `x.py`? diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index 258899400..2634cc260 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -7,15 +7,15 @@ This is a guide for how to profile rustc with [perf](https://perf.wiki.kernel.or - Get a clean checkout of rust-lang/master, or whatever it is you want to profile. - Set the following settings in your `config.toml`: - - `debuginfo-lines = true` - - `use-jemalloc = false` — lets you do memory use profiling with valgrind + - `debuginfo-level = 1` - enables line debuginfo + - `use-jemalloc = false` - lets you do memory use profiling with valgrind - leave everything else the defaults - Run `./x.py build` to get a full build - Make a rustup toolchain pointing to that result - see [the "build and run" section for instructions][b-a-r] [b-a-r]: ../how-to-build-and-run.html#toolchain - + ## Gathering a perf profile perf is an excellent tool on linux that can be used to gather and @@ -295,7 +295,7 @@ altogether ("total") and the percent of time spent in **just that function and not some callee of that function** (self). Usually "total" is the more interesting number, but not always. -### Relative percentages +### Relative percentages By default, all in perf-focus are relative to the **total program execution**. This is useful to help you keep perspective — often as From 2290cd10370f071582cba5fe771e69dbe44ab0a5 Mon Sep 17 00:00:00 2001 From: mdsimmo Date: Tue, 4 Jun 2019 13:44:58 +1000 Subject: [PATCH 561/648] correct indentation Indentation consistently 4 spaces and Part 1/Part2 contain everything else --- src/SUMMARY.md | 157 ++++++++++++++++++++++++------------------------- 1 file changed, 77 insertions(+), 80 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 3d5dd44b6..2b519da75 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -5,87 +5,84 @@ --- - [Part 1: Building, debugging, and contributing to Rustc](./part-1-intro.md) -- [About the compiler team](./compiler-team.md) -- [How to Build and Run the Compiler](./how-to-build-and-run.md) - - [Build and Install distribution artifacts](./build-install-distribution-artifacts.md) - - [Documenting Compiler](./compiler-documenting.md) -- [The compiler testing framework](./tests/intro.md) - - [Running tests](./tests/running.md) - - [Adding new tests](./tests/adding.md) - - [Using `compiletest` + commands to control test execution](./compiletest.md) -- [Walkthrough: a typical contribution](./walkthrough.md) -- [Implementing new features](./implementing_new_features.md) -- [Stabilizing Features](./stabilization_guide.md) -- [Debugging the Compiler](./compiler-debugging.md) -- [Profiling the compiler](./profiling.md) - - [with the linux perf tool](./profiling/with_perf.md) -- [Coding conventions](./conventions.md) -- [crates.io Dependencies](./crates-io.md) -- [Emitting Errors and other Diagnostics](diagnostics.md) - - [JSON diagnostic format](diagnostics/json-format.md) - ---- - + - [About the compiler team](./compiler-team.md) + - [How to Build and Run the Compiler](./how-to-build-and-run.md) + - [Build and Install distribution artifacts](./build-install-distribution-artifacts.md) + - [Documenting Compiler](./compiler-documenting.md) + - [The compiler testing framework](./tests/intro.md) + - [Running tests](./tests/running.md) + - [Adding new tests](./tests/adding.md) + - [Using `compiletest` + commands to control test execution](./compiletest.md) + - [Walkthrough: a typical contribution](./walkthrough.md) + - [Implementing new features](./implementing_new_features.md) + - [Stabilizing Features](./stabilization_guide.md) + - [Debugging the Compiler](./compiler-debugging.md) + - [Profiling the compiler](./profiling.md) + - [with the linux perf tool](./profiling/with_perf.md) + - [Coding conventions](./conventions.md) + - [crates.io Dependencies](./crates-io.md) + - [Emitting Errors and other Diagnostics](diagnostics.md) + - [JSON diagnostic format](diagnostics/json-format.md) - [Part 2: How rustc works](./part-2-intro.md) -- [High-level overview of the compiler source](./high-level-overview.md) -- [The Rustc Driver and Interface](./rustc-driver.md) - - [Rustdoc](./rustdoc.md) -- [Queries: demand-driven compilation](./query.md) - - [The Query Evaluation Model in Detail](./queries/query-evaluation-model-in-detail.md) - - [Incremental compilation](./queries/incremental-compilation.md) - - [Incremental compilation In Detail](./queries/incremental-compilation-in-detail.md) - - [Debugging and Testing](./incrcomp-debugging.md) -- [The parser](./the-parser.md) -- [`#[test]` Implementation](./test-implementation.md) -- [Macro expansion](./macro-expansion.md) -- [Name resolution](./name-resolution.md) -- [The HIR (High-level IR)](./hir.md) - - [Lowering AST to HIR](./lowering.md) - - [Debugging](./hir-debugging.md) -- [The `ty` module: representing types](./ty.md) -- [Kinds](./kinds.md) -- [Type inference](./type-inference.md) -- [Trait solving (old-style)](./traits/resolution.md) - - [Higher-ranked trait bounds](./traits/hrtb.md) - - [Caching subtleties](./traits/caching.md) - - [Specialization](./traits/specialization.md) -- [Trait solving (new-style)](./traits/index.md) - - [Lowering to logic](./traits/lowering-to-logic.md) - - [Goals and clauses](./traits/goals-and-clauses.md) - - [Equality and associated types](./traits/associated-types.md) - - [Implied bounds](./traits/implied-bounds.md) - - [Region constraints](./traits/regions.md) - - [The lowering module in rustc](./traits/lowering-module.md) - - [Lowering rules](./traits/lowering-rules.md) - - [Well-formedness checking](./traits/wf.md) - - [Canonical queries](./traits/canonical-queries.md) - - [Canonicalization](./traits/canonicalization.md) - - [The SLG solver](./traits/slg.md) - - [An Overview of Chalk](./traits/chalk-overview.md) - - [Bibliography](./traits/bibliography.md) -- [Type checking](./type-checking.md) - - [Method Lookup](./method-lookup.md) - - [Variance](./variance.md) - - [Existential Types](./existential-types.md) -- [The MIR (Mid-level IR)](./mir/index.md) - - [MIR construction](./mir/construction.md) - - [MIR visitor and traversal](./mir/visitor.md) - - [MIR passes: getting the MIR for a function](./mir/passes.md) - - [MIR optimizations](./mir/optimizations.md) - - [Debugging](./mir/debugging.md) -- [The borrow checker](./borrow_check.md) - - [Tracking moves and initialization](./borrow_check/moves_and_initialization.md) - - [Move paths](./borrow_check/moves_and_initialization/move_paths.md) - - [MIR type checker](./borrow_check/type_check.md) - - [Region inference](./borrow_check/region_inference.md) - - [Two-phase-borrows](./borrow_check/two_phase_borrows.md) -- [Constant evaluation](./const-eval.md) - - [miri const evaluator](./miri.md) -- [Parameter Environments](./param_env.md) -- [Code Generation](./codegen.md) - - [Updating LLVM](./codegen/updating-llvm.md) - - [Debugging LLVM](./codegen/debugging.md) -- [Profile-guided Optimization](./profile-guided-optimization.md) + - [High-level overview of the compiler source](./high-level-overview.md) + - [The Rustc Driver and Interface](./rustc-driver.md) + - [Rustdoc](./rustdoc.md) + - [Queries: demand-driven compilation](./query.md) + - [The Query Evaluation Model in Detail](./queries/query-evaluation-model-in-detail.md) + - [Incremental compilation](./queries/incremental-compilation.md) + - [Incremental compilation In Detail](./queries/incremental-compilation-in-detail.md) + - [Debugging and Testing](./incrcomp-debugging.md) + - [The parser](./the-parser.md) + - [`#[test]` Implementation](./test-implementation.md) + - [Macro expansion](./macro-expansion.md) + - [Name resolution](./name-resolution.md) + - [The HIR (High-level IR)](./hir.md) + - [Lowering AST to HIR](./lowering.md) + - [Debugging](./hir-debugging.md) + - [The `ty` module: representing types](./ty.md) + - [Kinds](./kinds.md) + - [Type inference](./type-inference.md) + - [Trait solving (old-style)](./traits/resolution.md) + - [Higher-ranked trait bounds](./traits/hrtb.md) + - [Caching subtleties](./traits/caching.md) + - [Specialization](./traits/specialization.md) + - [Trait solving (new-style)](./traits/index.md) + - [Lowering to logic](./traits/lowering-to-logic.md) + - [Goals and clauses](./traits/goals-and-clauses.md) + - [Equality and associated types](./traits/associated-types.md) + - [Implied bounds](./traits/implied-bounds.md) + - [Region constraints](./traits/regions.md) + - [The lowering module in rustc](./traits/lowering-module.md) + - [Lowering rules](./traits/lowering-rules.md) + - [Well-formedness checking](./traits/wf.md) + - [Canonical queries](./traits/canonical-queries.md) + - [Canonicalization](./traits/canonicalization.md) + - [The SLG solver](./traits/slg.md) + - [An Overview of Chalk](./traits/chalk-overview.md) + - [Bibliography](./traits/bibliography.md) + - [Type checking](./type-checking.md) + - [Method Lookup](./method-lookup.md) + - [Variance](./variance.md) + - [Existential Types](./existential-types.md) + - [The MIR (Mid-level IR)](./mir/index.md) + - [MIR construction](./mir/construction.md) + - [MIR visitor and traversal](./mir/visitor.md) + - [MIR passes: getting the MIR for a function](./mir/passes.md) + - [MIR optimizations](./mir/optimizations.md) + - [Debugging](./mir/debugging.md) + - [The borrow checker](./borrow_check.md) + - [Tracking moves and initialization](./borrow_check/moves_and_initialization.md) + - [Move paths](./borrow_check/moves_and_initialization/move_paths.md) + - [MIR type checker](./borrow_check/type_check.md) + - [Region inference](./borrow_check/region_inference.md) + - [Two-phase-borrows](./borrow_check/two_phase_borrows.md) + - [Constant evaluation](./const-eval.md) + - [miri const evaluator](./miri.md) + - [Parameter Environments](./param_env.md) + - [Code Generation](./codegen.md) + - [Updating LLVM](./codegen/updating-llvm.md) + - [Debugging LLVM](./codegen/debugging.md) + - [Profile-guided Optimization](./profile-guided-optimization.md) --- From 9015f537ab88bd8db00bfdc3144b452409d8d915 Mon Sep 17 00:00:00 2001 From: mdsimmo Date: Tue, 4 Jun 2019 14:11:06 +1000 Subject: [PATCH 562/648] Replaced tabs with spaces --- src/SUMMARY.md | 154 ++++++++++++++++++++++++------------------------- 1 file changed, 77 insertions(+), 77 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 2b519da75..e83898520 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -5,84 +5,84 @@ --- - [Part 1: Building, debugging, and contributing to Rustc](./part-1-intro.md) - - [About the compiler team](./compiler-team.md) - - [How to Build and Run the Compiler](./how-to-build-and-run.md) - - [Build and Install distribution artifacts](./build-install-distribution-artifacts.md) - - [Documenting Compiler](./compiler-documenting.md) - - [The compiler testing framework](./tests/intro.md) - - [Running tests](./tests/running.md) - - [Adding new tests](./tests/adding.md) - - [Using `compiletest` + commands to control test execution](./compiletest.md) - - [Walkthrough: a typical contribution](./walkthrough.md) - - [Implementing new features](./implementing_new_features.md) - - [Stabilizing Features](./stabilization_guide.md) - - [Debugging the Compiler](./compiler-debugging.md) - - [Profiling the compiler](./profiling.md) - - [with the linux perf tool](./profiling/with_perf.md) - - [Coding conventions](./conventions.md) - - [crates.io Dependencies](./crates-io.md) - - [Emitting Errors and other Diagnostics](diagnostics.md) - - [JSON diagnostic format](diagnostics/json-format.md) + - [About the compiler team](./compiler-team.md) + - [How to Build and Run the Compiler](./how-to-build-and-run.md) + - [Build and Install distribution artifacts](./build-install-distribution-artifacts.md) + - [Documenting Compiler](./compiler-documenting.md) + - [The compiler testing framework](./tests/intro.md) + - [Running tests](./tests/running.md) + - [Adding new tests](./tests/adding.md) + - [Using `compiletest` + commands to control test execution](./compiletest.md) + - [Walkthrough: a typical contribution](./walkthrough.md) + - [Implementing new features](./implementing_new_features.md) + - [Stabilizing Features](./stabilization_guide.md) + - [Debugging the Compiler](./compiler-debugging.md) + - [Profiling the compiler](./profiling.md) + - [with the linux perf tool](./profiling/with_perf.md) + - [Coding conventions](./conventions.md) + - [crates.io Dependencies](./crates-io.md) + - [Emitting Errors and other Diagnostics](diagnostics.md) + - [JSON diagnostic format](diagnostics/json-format.md) - [Part 2: How rustc works](./part-2-intro.md) - - [High-level overview of the compiler source](./high-level-overview.md) - - [The Rustc Driver and Interface](./rustc-driver.md) - - [Rustdoc](./rustdoc.md) - - [Queries: demand-driven compilation](./query.md) - - [The Query Evaluation Model in Detail](./queries/query-evaluation-model-in-detail.md) - - [Incremental compilation](./queries/incremental-compilation.md) - - [Incremental compilation In Detail](./queries/incremental-compilation-in-detail.md) - - [Debugging and Testing](./incrcomp-debugging.md) - - [The parser](./the-parser.md) - - [`#[test]` Implementation](./test-implementation.md) - - [Macro expansion](./macro-expansion.md) - - [Name resolution](./name-resolution.md) - - [The HIR (High-level IR)](./hir.md) - - [Lowering AST to HIR](./lowering.md) - - [Debugging](./hir-debugging.md) - - [The `ty` module: representing types](./ty.md) - - [Kinds](./kinds.md) - - [Type inference](./type-inference.md) - - [Trait solving (old-style)](./traits/resolution.md) - - [Higher-ranked trait bounds](./traits/hrtb.md) - - [Caching subtleties](./traits/caching.md) - - [Specialization](./traits/specialization.md) - - [Trait solving (new-style)](./traits/index.md) - - [Lowering to logic](./traits/lowering-to-logic.md) - - [Goals and clauses](./traits/goals-and-clauses.md) - - [Equality and associated types](./traits/associated-types.md) - - [Implied bounds](./traits/implied-bounds.md) - - [Region constraints](./traits/regions.md) - - [The lowering module in rustc](./traits/lowering-module.md) - - [Lowering rules](./traits/lowering-rules.md) - - [Well-formedness checking](./traits/wf.md) - - [Canonical queries](./traits/canonical-queries.md) - - [Canonicalization](./traits/canonicalization.md) - - [The SLG solver](./traits/slg.md) - - [An Overview of Chalk](./traits/chalk-overview.md) - - [Bibliography](./traits/bibliography.md) - - [Type checking](./type-checking.md) - - [Method Lookup](./method-lookup.md) - - [Variance](./variance.md) - - [Existential Types](./existential-types.md) - - [The MIR (Mid-level IR)](./mir/index.md) - - [MIR construction](./mir/construction.md) - - [MIR visitor and traversal](./mir/visitor.md) - - [MIR passes: getting the MIR for a function](./mir/passes.md) - - [MIR optimizations](./mir/optimizations.md) - - [Debugging](./mir/debugging.md) - - [The borrow checker](./borrow_check.md) - - [Tracking moves and initialization](./borrow_check/moves_and_initialization.md) - - [Move paths](./borrow_check/moves_and_initialization/move_paths.md) - - [MIR type checker](./borrow_check/type_check.md) - - [Region inference](./borrow_check/region_inference.md) - - [Two-phase-borrows](./borrow_check/two_phase_borrows.md) - - [Constant evaluation](./const-eval.md) - - [miri const evaluator](./miri.md) - - [Parameter Environments](./param_env.md) - - [Code Generation](./codegen.md) - - [Updating LLVM](./codegen/updating-llvm.md) - - [Debugging LLVM](./codegen/debugging.md) - - [Profile-guided Optimization](./profile-guided-optimization.md) + - [High-level overview of the compiler source](./high-level-overview.md) + - [The Rustc Driver and Interface](./rustc-driver.md) + - [Rustdoc](./rustdoc.md) + - [Queries: demand-driven compilation](./query.md) + - [The Query Evaluation Model in Detail](./queries/query-evaluation-model-in-detail.md) + - [Incremental compilation](./queries/incremental-compilation.md) + - [Incremental compilation In Detail](./queries/incremental-compilation-in-detail.md) + - [Debugging and Testing](./incrcomp-debugging.md) + - [The parser](./the-parser.md) + - [`#[test]` Implementation](./test-implementation.md) + - [Macro expansion](./macro-expansion.md) + - [Name resolution](./name-resolution.md) + - [The HIR (High-level IR)](./hir.md) + - [Lowering AST to HIR](./lowering.md) + - [Debugging](./hir-debugging.md) + - [The `ty` module: representing types](./ty.md) + - [Kinds](./kinds.md) + - [Type inference](./type-inference.md) + - [Trait solving (old-style)](./traits/resolution.md) + - [Higher-ranked trait bounds](./traits/hrtb.md) + - [Caching subtleties](./traits/caching.md) + - [Specialization](./traits/specialization.md) + - [Trait solving (new-style)](./traits/index.md) + - [Lowering to logic](./traits/lowering-to-logic.md) + - [Goals and clauses](./traits/goals-and-clauses.md) + - [Equality and associated types](./traits/associated-types.md) + - [Implied bounds](./traits/implied-bounds.md) + - [Region constraints](./traits/regions.md) + - [The lowering module in rustc](./traits/lowering-module.md) + - [Lowering rules](./traits/lowering-rules.md) + - [Well-formedness checking](./traits/wf.md) + - [Canonical queries](./traits/canonical-queries.md) + - [Canonicalization](./traits/canonicalization.md) + - [The SLG solver](./traits/slg.md) + - [An Overview of Chalk](./traits/chalk-overview.md) + - [Bibliography](./traits/bibliography.md) + - [Type checking](./type-checking.md) + - [Method Lookup](./method-lookup.md) + - [Variance](./variance.md) + - [Existential Types](./existential-types.md) + - [The MIR (Mid-level IR)](./mir/index.md) + - [MIR construction](./mir/construction.md) + - [MIR visitor and traversal](./mir/visitor.md) + - [MIR passes: getting the MIR for a function](./mir/passes.md) + - [MIR optimizations](./mir/optimizations.md) + - [Debugging](./mir/debugging.md) + - [The borrow checker](./borrow_check.md) + - [Tracking moves and initialization](./borrow_check/moves_and_initialization.md) + - [Move paths](./borrow_check/moves_and_initialization/move_paths.md) + - [MIR type checker](./borrow_check/type_check.md) + - [Region inference](./borrow_check/region_inference.md) + - [Two-phase-borrows](./borrow_check/two_phase_borrows.md) + - [Constant evaluation](./const-eval.md) + - [miri const evaluator](./miri.md) + - [Parameter Environments](./param_env.md) + - [Code Generation](./codegen.md) + - [Updating LLVM](./codegen/updating-llvm.md) + - [Debugging LLVM](./codegen/debugging.md) + - [Profile-guided Optimization](./profile-guided-optimization.md) --- From af46ee444949b1826d2ba413cdb5025aa91adfba Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Fri, 7 Jun 2019 15:34:08 -0500 Subject: [PATCH 563/648] use debug instead of debuginfo-level --- src/how-to-build-and-run.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index d0ba0ea57..ca9065ced 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -36,9 +36,10 @@ debug-assertions = true # performance perhaps (less inlining) but it's worth it. codegen-units = 0 -# I always enable full debuginfo, however the line debuginfo (which is a more important part -# and enables e.g. line numbers in backtraces) is enabled by `debuginfo-level = 1` too. -debuginfo-level = 2 +# This enables full debuginfo (`debuginfo-level = 2`). The line debuginfo (which is a +# more important part and enables e.g. line numbers in backtraces) is also enabled by +# `debuginfo-level = 1`. +debug = true ``` ### What is `x.py`? From 53961054c84be0f5e6189aafcf30d0996d1bff40 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Fri, 7 Jun 2019 15:51:16 -0500 Subject: [PATCH 564/648] Update compiler-debugging.md --- src/compiler-debugging.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index 304f75d80..0126c2503 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -46,9 +46,10 @@ stack backtrace: 37: rustc_driver::run_compiler ``` -If you want line numbers for the stack trace, you can enable -`debuginfo-level = 1` or `debuginfo-level = 2` in your config.toml and rebuild the -compiler. Then the backtrace will look like this: +If you want line numbers for the stack trace, you can enable `debug = true` in +your config.toml and rebuild the compiler (`debuginfo-level = 1` will also add +line numbers, but `debug = true` gives full debuginfo). Then the backtrace will +look like this: ```text stack backtrace: From 14f353e9d810897138c62020363a0a7f71902f43 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Fri, 7 Jun 2019 16:06:09 -0500 Subject: [PATCH 565/648] Update how-to-build-and-run.md --- src/how-to-build-and-run.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index ca9065ced..e0ce70111 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -28,17 +28,15 @@ settings (and possibly others, such as `llvm.ccache`): assertions = true [rust] -# This enables some assertions, but more importantly it enables the `debug!` -# logging macros that are essential for debugging `rustc`. -debug-assertions = true - # This will make your build more parallel; it costs a bit of runtime # performance perhaps (less inlining) but it's worth it. codegen-units = 0 -# This enables full debuginfo (`debuginfo-level = 2`). The line debuginfo (which is a -# more important part and enables e.g. line numbers in backtraces) is also enabled by -# `debuginfo-level = 1`. +# This enables full debuginfo and debug assertions. The line debuginfo is also +# enabled by `debuginfo-level = 1`. Full debuginfo is also enabled by +# `debuginfo-level = 2`. Debug assertions can also be enabled with +# `debug-assertions = true`. Note that `debug = true` will make your build +# slower, so you may want to try individually enabling debuginfo and assertions. debug = true ``` From eda82ca8c9554a27bb24e4e2ca5a3ed67ffa5f16 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Fri, 7 Jun 2019 16:07:45 -0500 Subject: [PATCH 566/648] Update how-to-build-and-run.md --- src/how-to-build-and-run.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index e0ce70111..f44179bc9 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -36,7 +36,8 @@ codegen-units = 0 # enabled by `debuginfo-level = 1`. Full debuginfo is also enabled by # `debuginfo-level = 2`. Debug assertions can also be enabled with # `debug-assertions = true`. Note that `debug = true` will make your build -# slower, so you may want to try individually enabling debuginfo and assertions. +# slower, so you may want to try individually enabling debuginfo and assertions +# or enable only line debuginfo which is basically free. debug = true ``` From caee05203e8bb65bb9005d13fb25316f80c331b7 Mon Sep 17 00:00:00 2001 From: Santiago Pastorino Date: Wed, 12 Jun 2019 17:00:07 -0300 Subject: [PATCH 567/648] Add triagebot --- triagebot.toml | 1 + 1 file changed, 1 insertion(+) create mode 100644 triagebot.toml diff --git a/triagebot.toml b/triagebot.toml new file mode 100644 index 000000000..fa0824ac5 --- /dev/null +++ b/triagebot.toml @@ -0,0 +1 @@ +[assign] From d97d111c2e2ddcb2007c2211bbea60165785a473 Mon Sep 17 00:00:00 2001 From: Jonas Schievink Date: Wed, 12 Jun 2019 20:01:27 +0200 Subject: [PATCH 568/648] Fix a few typos in type inference chapter Also try to test the top example --- src/type-inference.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/type-inference.md b/src/type-inference.md index a3d012b5d..d94c06379 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -6,14 +6,14 @@ expression. It is what allows Rust to work with fewer or no type annotations, making things easier for users: -```rust,ignore +```rust fn main() { let mut things = vec![]; - things.push("thing") + things.push("thing"); } ``` -Here, the type of `things` is *inferenced* to be `&str` because that's the value +Here, the type of `things` is *inferred* to be `Vec<&str>` because of the value we push into `things`. The type inference is based on the standard Hindley-Milner (HM) type inference From 885ab7e860cf7c115f96e053f93fee0e9f25eadf Mon Sep 17 00:00:00 2001 From: Jonathan Behrens Date: Sat, 8 Jun 2019 19:11:04 -0400 Subject: [PATCH 569/648] Changes to config.toml require a clean --- src/how-to-build-and-run.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index f44179bc9..92fbcb0fd 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -41,6 +41,9 @@ codegen-units = 0 debug = true ``` +If you have already built `rustc`, then you may have to execute `rm -rf build` for subsequent +configuration changes to take effect. + ### What is `x.py`? `x.py` is the script used to orchestrate the tooling in the `rustc` repository. From 8dfb8c1f9ec7117f317374c5022ad44d83677055 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sat, 15 Jun 2019 16:08:35 -0500 Subject: [PATCH 570/648] add note about rebuilding llvm --- src/how-to-build-and-run.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 92fbcb0fd..84efffc84 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -42,7 +42,9 @@ debug = true ``` If you have already built `rustc`, then you may have to execute `rm -rf build` for subsequent -configuration changes to take effect. +configuration changes to take effect. Note that `./x.py clean` will not cause a +rebuild of LLVM, so if your configuration change affects LLVM, you will need to +manually `rm -rf build/` before rebuilding. ### What is `x.py`? From ea6f6b516b24c70498863e74b47d6a7293764e73 Mon Sep 17 00:00:00 2001 From: Blitzerr Date: Thu, 1 Nov 2018 22:14:05 -0700 Subject: [PATCH 571/648] Notes about closure de-sugaring --- src/SUMMARY.md | 1 + src/closure.md | 159 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 160 insertions(+) create mode 100644 src/closure.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index e83898520..273b409ec 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -39,6 +39,7 @@ - [The HIR (High-level IR)](./hir.md) - [Lowering AST to HIR](./lowering.md) - [Debugging](./hir-debugging.md) + - [Closure expansion](./closure.md) - [The `ty` module: representing types](./ty.md) - [Kinds](./kinds.md) - [Type inference](./type-inference.md) diff --git a/src/closure.md b/src/closure.md new file mode 100644 index 000000000..3bf965983 --- /dev/null +++ b/src/closure.md @@ -0,0 +1,159 @@ +# Closure Expansion in rustc + +Let's start with a few examples + +### Example 1 +```rust +fn closure(f: impl Fn()) { + f(); +} + +fn main() { + let x: i32 = 10; + closure(|| println!("Hi {}", x)); // The closure just reads x. + println!("Value of x after return {}", x); +} +``` +Let's say the above is the content of a file called immut.rs. If we compile immut.rs using the command +``` +rustc +stage1 immut.rs -Zdump-mir=all +``` +we will see a newly generated directory in our current working directory called mir_dump, which will +contain several files. If we look at file `rustc.main.-------.mir_map.0.mir`, we will find, among +other things, it also contains this line: + +```rust,ignore +_4 = &_1; // bb0[6]: scope 1 at immut.rs:7:13: 7:36 +_3 = [closure@immut.rs:7:13: 7:36] { x: move _4 }; // bb0[7]: scope 1 at immut.rs:7:13: 7:36 +``` +Here in first line `_4 = &_1;`, the mir_dump tells us that x was borrowed as an immutable reference. +This is what we would hope as our closure just reads x. + +### Example 2 +```rust +fn closure(mut f: impl FnMut()) { + f(); +} + +fn main() { + let mut x: i32 = 10; + closure(|| { + x += 10; // The closure mutates the value of x + println!("Hi {}", x) + }); + println!("Value of x after return {}", x); +} +``` + +```rust,ignore +_4 = &mut _1; // bb0[6]: scope 1 at mut.rs:7:13: 10:6 +_3 = [closure@mut.rs:7:13: 10:6] { x: move _4 }; // bb0[7]: scope 1 at mut.rs:7:13: 10:6 +``` +This time along, in the line `_4 = &mut _1;`, we see that the borrow is changed to mutable borrow. +fair enough as the closure increments x by 10. + +### Example 3 +```rust +fn closure(f: impl FnOnce()) { + f(); +} + +fn main() { + let x = vec![21]; + closure(|| { + drop(x); // Makes x unusable after the fact. + }); + // println!("Value of x after return {:?}", x); +} +``` + +```rust,ignore +_6 = [closure@move.rs:7:13: 9:6] { x: move _1 }; // bb16[3]: scope 1 at move.rs:7:13: 9:6 +``` +Here, x is directly moved into the closure and the access to it will not be permitted after the +closure. + + +Now let's dive into rustc code and see how all these inferences are done by the compiler. + +Let's start with defining a term that we will be using quite a bit in the rest of the discussion - +*upvar*. An **upvar** is a variable that is local to the function, where the closure is defined. So, +in the above examples, **x** will be an upvar to the closure. They are also sometimes referred to as +the *free variables* meaning they are not bound to the context of the closure. +`src/librustc/ty/query/mod.rs` defines a query called *freevars* for this purpose. + +So, we know that other than lazy invocation, one other thing that the distinguishes a closure from a +normal function is that it can use the upvars. Because, it borrows these upvars from its surrounding +context, therfore the compiler has to determine the upvar's borrow type. The compiler starts with +assigning an immutable borrow type and lowers the restriction (that is, changes it from +**immutable** to **mutable** to **move**) as needed, based on the usage. In the Example 1 above, the +closure only uses the variable for printing but does not modify it in any way and therefore, in the +mir_dump, we find the borrow type for the upvar x to be immutable. In example 2, however the +closure modifies x and increments it by some value. Because of this mutation, the compiler, which +started off assigning x as an immutable reference type, has to adjust it as mutable reference. +Likewise in the third example, the closure drops the vector and therefore this requires the variable +x to be moved into the closure. Depending on the borrow kind, the closure has to implement the +appropriate trait. Fn trait for immutable borrow, FnMut for mutable borrow and FnOnce for move +semantics. + +Most of the code related to the closure is in the src/librustc_typeck/check/upvar.rs file and the +data structures are declared in the file src/librustc/ty/mod.rs. + +Before we go any further, let's discuss how we can examine the flow of coontrol through the rustc +codebase. For the closure part specifically, I would set the RUST_LOG as under and collect the +output in a file + +``` +RUST_LOG=rustc_typeck::check::upvar rustc +stage1 -Zdump-mir=all <.rs file to compile> 2> +``` + +This uses the stage1 compiler. + +The other option is to step through the code using lldb or gdb. + +``` +1. rust-lldb build/x86_64-apple-darwin/stage1/bin/rustc test.rs +2. b upvar.rs:134 // Setting the breakpoint on a certain line in the upvar.rs file +3. r // Run the program until it hits the breakpoint +``` + +Let's start with the file: `upvar.rs`. This file has something called the euv::ExprUseVisitor which +walks the source of the closure and it gets called back for each upvar that is borrowed, mutated or +moved. + +```rust +fn main() { + let x = vec![21]; + let _cl = || { + let y = x[0]; // 1. + x[0] += 1; // 2. + }; +} +``` + +In the above example, our visitor will be called twice, for the lines marked 1 and 2, once as a +shared borrow and another one as a mutable borrow. It will also tell as what was borrowed. The +callbacks get invoked at the delegate. The delegate is of type `struct InferBorrowKind` which has a +few fields but the one we are interested in is the `adjust_upvar_captures` which is of type +`FxHashMap>` which tells us for each upvar, which mode of borrow did we +require. The modes of borrow can be ByValue (moved) or ByRef (borrowed) and for ByRef borrows, it +can be one among shared, shallow, unique or mut as defined in the `src/librustc/mir/mod.rs` + +The method callbacks are the method implementations of the euv::Delegate trait for InferBorrowKind. +**consume** callback is for *move* of a variable, **borrow** callback if there is a *borrow* of some +kind, shared or mutable and **mutate** when we see an *assignment* of something. We will see that +all these callbacks have a common argument *cmt* which stands for category, Mutability and Type and +is defined in *src/librustc/middle/mem_categorization.rs*. Borrowing from the code comments *cmt *is +a complete categorization of a value indicating where it originated and how it is located, as well +as the mutability of the memory in which the value is stored.** Based on the callback (consume, +borrow etc.), we will call the relevant *adjust_upvar_borrow_kind_for_* and pass the cmt +along. Once the borrow type is adjusted, we store it in the table, which basically says for this +closure, these set of borrows were made. + +``` +self.tables + .borrow_mut() + .upvar_capture_map + .extend(delegate.adjust_upvar_captures); +``` From 9a4ff92b8e151590e98db4c4d9d1b0e8d085533d Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sat, 23 Mar 2019 13:47:03 -0500 Subject: [PATCH 572/648] Some edits to address review comments --- src/closure.md | 174 ++++++++++++++++++++++++++++++++----------------- 1 file changed, 114 insertions(+), 60 deletions(-) diff --git a/src/closure.md b/src/closure.md index 3bf965983..a31893b0a 100644 --- a/src/closure.md +++ b/src/closure.md @@ -1,8 +1,24 @@ # Closure Expansion in rustc -Let's start with a few examples +This section describes how rustc handles closures. Closures in Rust are +effectively "desugared" into structs that contain the values they use (or +references to the values they use) from their creator's stack frame. rustc has +the job of figuring out which values a closure uses and how, so it can decide +whether to capture a given variable by shared reference, mutable reference, or +by move. rustc also has to figure out which the closure traits ([`Fn`][fn], +[`FnMut`][fn_mut], or [`FnOnce`][fn_once]) a closure is capable of +implementing. + +[fn]: https://doc.rust-lang.org/std/ops/trait.Fn.html +[fn_mut]:https://doc.rust-lang.org/std/ops/trait.FnMut.html +[fn_once]: https://doc.rust-lang.org/std/ops/trait.FnOnce.html + +Let's start with a few examples: ### Example 1 + +To start, let's take a look at how the closure in the following example is desugared: + ```rust fn closure(f: impl Fn()) { f(); @@ -14,22 +30,37 @@ fn main() { println!("Value of x after return {}", x); } ``` -Let's say the above is the content of a file called immut.rs. If we compile immut.rs using the command -``` -rustc +stage1 immut.rs -Zdump-mir=all + +Let's say the above is the content of a file called `immut.rs`. If we compile +`immut.rs` using the following command. The [`-Zdump-mir=all`][dump-mir] flag will cause +`rustc` to generate and dump the [MIR][mir] to a directory called `mir_dump`. +```console +> rustc +stage1 immut.rs -Zdump-mir=all ``` -we will see a newly generated directory in our current working directory called mir_dump, which will -contain several files. If we look at file `rustc.main.-------.mir_map.0.mir`, we will find, among + +[mir]: ./mir/index.md +[dump-mir]: ./mir/passes.md + +After we run this command, we will see a newly generated directory in our +current working directory called `mir_dump`, which will contain several files. +If we look at file `rustc.main.-------.mir_map.0.mir`, we will find, among other things, it also contains this line: ```rust,ignore -_4 = &_1; // bb0[6]: scope 1 at immut.rs:7:13: 7:36 -_3 = [closure@immut.rs:7:13: 7:36] { x: move _4 }; // bb0[7]: scope 1 at immut.rs:7:13: 7:36 +_4 = &_1; +_3 = [closure@immut.rs:7:13: 7:36] { x: move _4 }; ``` -Here in first line `_4 = &_1;`, the mir_dump tells us that x was borrowed as an immutable reference. -This is what we would hope as our closure just reads x. + +Note that in the MIR examples in this chapter, `_1` is `x`. + +Here in first line `_4 = &_1;`, the `mir_dump` tells us that `x` was borrowed +as an immutable reference. This is what we would hope as our closure just +reads `x`. ### Example 2 + +Here is another example: + ```rust fn closure(mut f: impl FnMut()) { f(); @@ -46,13 +77,16 @@ fn main() { ``` ```rust,ignore -_4 = &mut _1; // bb0[6]: scope 1 at mut.rs:7:13: 10:6 -_3 = [closure@mut.rs:7:13: 10:6] { x: move _4 }; // bb0[7]: scope 1 at mut.rs:7:13: 10:6 +_4 = &mut _1; +_3 = [closure@mut.rs:7:13: 10:6] { x: move _4 }; ``` This time along, in the line `_4 = &mut _1;`, we see that the borrow is changed to mutable borrow. -fair enough as the closure increments x by 10. +Fair enough! The closure increments `x` by 10. ### Example 3 + +One more example: + ```rust fn closure(f: impl FnOnce()) { f(); @@ -70,57 +104,66 @@ fn main() { ```rust,ignore _6 = [closure@move.rs:7:13: 9:6] { x: move _1 }; // bb16[3]: scope 1 at move.rs:7:13: 9:6 ``` -Here, x is directly moved into the closure and the access to it will not be permitted after the +Here, `x` is directly moved into the closure and the access to it will not be permitted after the closure. +## Inferences in the compiler Now let's dive into rustc code and see how all these inferences are done by the compiler. Let's start with defining a term that we will be using quite a bit in the rest of the discussion - -*upvar*. An **upvar** is a variable that is local to the function, where the closure is defined. So, +*upvar*. An **upvar** is a variable that is local to the function where the closure is defined. So, in the above examples, **x** will be an upvar to the closure. They are also sometimes referred to as the *free variables* meaning they are not bound to the context of the closure. -`src/librustc/ty/query/mod.rs` defines a query called *freevars* for this purpose. +[`src/librustc/ty/query/mod.rs`][freevars] defines a query called *freevars* for this purpose. -So, we know that other than lazy invocation, one other thing that the distinguishes a closure from a -normal function is that it can use the upvars. Because, it borrows these upvars from its surrounding -context, therfore the compiler has to determine the upvar's borrow type. The compiler starts with +[freevars]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/query/queries/struct.freevars.html + +Other than lazy invocation, one other thing that the distinguishes a closure from a +normal function is that it can use the upvars. It borrows these upvars from its surrounding +context; therfore the compiler has to determine the upvar's borrow type. The compiler starts with assigning an immutable borrow type and lowers the restriction (that is, changes it from **immutable** to **mutable** to **move**) as needed, based on the usage. In the Example 1 above, the closure only uses the variable for printing but does not modify it in any way and therefore, in the -mir_dump, we find the borrow type for the upvar x to be immutable. In example 2, however the -closure modifies x and increments it by some value. Because of this mutation, the compiler, which -started off assigning x as an immutable reference type, has to adjust it as mutable reference. +`mir_dump`, we find the borrow type for the upvar `x` to be immutable. In example 2, however, the +closure modifies `x` and increments it by some value. Because of this mutation, the compiler, which +started off assigning `x` as an immutable reference type, has to adjust it as a mutable reference. Likewise in the third example, the closure drops the vector and therefore this requires the variable -x to be moved into the closure. Depending on the borrow kind, the closure has to implement the -appropriate trait. Fn trait for immutable borrow, FnMut for mutable borrow and FnOnce for move -semantics. +`x` to be moved into the closure. Depending on the borrow kind, the closure has to implement the +appropriate trait: `Fn` trait for immutable borrow, `FnMut` for mutable borrow, +and `FnOnce` for move semantics. + +Most of the code related to the closure is in the +[`src/librustc_typeck/check/upvar.rs`][upvar] file and the data structures are +declared in the file [`src/librustc/ty/mod.rs`][ty]. -Most of the code related to the closure is in the src/librustc_typeck/check/upvar.rs file and the -data structures are declared in the file src/librustc/ty/mod.rs. +[upvar]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_typeck/check/upvar/index.html +[ty]:https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/index.html Before we go any further, let's discuss how we can examine the flow of coontrol through the rustc -codebase. For the closure part specifically, I would set the RUST_LOG as under and collect the -output in a file +codebase. For closures specifically, set the `RUST_LOG` env variable as below and collect the +output in a file: -``` -RUST_LOG=rustc_typeck::check::upvar rustc +stage1 -Zdump-mir=all <.rs file to compile> 2> +```console +> RUST_LOG=rustc_typeck::check::upvar rustc +stage1 -Zdump-mir=all \ + <.rs file to compile> 2> ``` -This uses the stage1 compiler. +This uses the stage1 compiler and enables `debug!` logging for the +`rustc_typeck::check::upvar` module. The other option is to step through the code using lldb or gdb. -``` -1. rust-lldb build/x86_64-apple-darwin/stage1/bin/rustc test.rs -2. b upvar.rs:134 // Setting the breakpoint on a certain line in the upvar.rs file -3. r // Run the program until it hits the breakpoint -``` +1. `rust-lldb build/x86_64-apple-darwin/stage1/bin/rustc test.rs` +2. In lldb: + 1. `b upvar.rs:134` // Setting the breakpoint on a certain line in the upvar.rs file` + 2. `r` // Run the program until it hits the breakpoint + +Let's start with [`upvar.rs`][upvar]. This file has something called +the [`euv::ExprUseVisitor`][euv] which walks the source of the closure and +invokes a callbackfor each upvar that is borrowed, mutated, or moved. -Let's start with the file: `upvar.rs`. This file has something called the euv::ExprUseVisitor which -walks the source of the closure and it gets called back for each upvar that is borrowed, mutated or -moved. +[euv]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/middle/expr_use_visitor/struct.ExprUseVisitor.html ```rust fn main() { @@ -132,28 +175,39 @@ fn main() { } ``` -In the above example, our visitor will be called twice, for the lines marked 1 and 2, once as a -shared borrow and another one as a mutable borrow. It will also tell as what was borrowed. The -callbacks get invoked at the delegate. The delegate is of type `struct InferBorrowKind` which has a -few fields but the one we are interested in is the `adjust_upvar_captures` which is of type -`FxHashMap>` which tells us for each upvar, which mode of borrow did we -require. The modes of borrow can be ByValue (moved) or ByRef (borrowed) and for ByRef borrows, it -can be one among shared, shallow, unique or mut as defined in the `src/librustc/mir/mod.rs` - -The method callbacks are the method implementations of the euv::Delegate trait for InferBorrowKind. -**consume** callback is for *move* of a variable, **borrow** callback if there is a *borrow* of some -kind, shared or mutable and **mutate** when we see an *assignment* of something. We will see that -all these callbacks have a common argument *cmt* which stands for category, Mutability and Type and -is defined in *src/librustc/middle/mem_categorization.rs*. Borrowing from the code comments *cmt *is -a complete categorization of a value indicating where it originated and how it is located, as well -as the mutability of the memory in which the value is stored.** Based on the callback (consume, -borrow etc.), we will call the relevant *adjust_upvar_borrow_kind_for_* and pass the cmt -along. Once the borrow type is adjusted, we store it in the table, which basically says for this -closure, these set of borrows were made. +In the above example, our visitor will be called twice, for the lines marked 1 and 2, once for a +shared borrow and another one for a mutable borrow. It will also tell us what was borrowed. -``` +The callbacks are defined by implementing the [`Delegate`][delegate] trait. The +[`InferBorrowKind`][ibk] type implements `Delegate` and keeps a map that +records for each upvar which mode of borrow was required. The modes of borrow +can be `ByValue` (moved) or `ByRef` (borrowed). For `ByRef` borrows, it can be +`shared`, `shallow`, `unique` or `mut` as defined in the +[`src/librustc/mir/mod.rs`][mir_mod]. + +[mir_mod]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/index.html + +`Delegate` defines a few different methods (the different callbacks): +**consume**: for *move* of a variable, **borrow** for a *borrow* of some kind +(shared or mutable), and **mutate** when we see an *assignment* of something. + +All of these callbacks have a common argument *cmt* which stands for Category, +Mutability and Type and is defined in +[`src/librustc/middle/mem_categorization.rs`][cmt]. Borrowing from the code +comments, "`cmt` is a complete categorization of a value indicating where it +originated and how it is located, as well as the mutability of the memory in +which the value is stored". Based on the callback (consume, borrow etc.), we +will call the relevant *adjust_upvar_borrow_kind_for_* and pass the +`cmt` along. Once the borrow type is adjusted, we store it in the table, which +basically says what borrows were made for each closure. + +```rust,ignore self.tables .borrow_mut() .upvar_capture_map .extend(delegate.adjust_upvar_captures); ``` + +[delegate]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/middle/expr_use_visitor/trait.Delegate.html +[ibk]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_typeck/check/upvar/struct.InferBorrowKind.html +[cmt]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/middle/mem_categorization/index.html From c15c60938a65d5a260b2dc14c58424278b59da39 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sat, 15 Jun 2019 16:54:08 -0500 Subject: [PATCH 573/648] fix ci failures, typos, broken links --- src/borrow_check/two_phase_borrows.md | 2 +- src/closure.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/borrow_check/two_phase_borrows.md b/src/borrow_check/two_phase_borrows.md index 753ffa5b2..09e7fbefe 100644 --- a/src/borrow_check/two_phase_borrows.md +++ b/src/borrow_check/two_phase_borrows.md @@ -15,7 +15,7 @@ two-phase borrow are: To give some examples: -```rust +```rust2018 // In the source code // Case 1: diff --git a/src/closure.md b/src/closure.md index a31893b0a..77f2b42c3 100644 --- a/src/closure.md +++ b/src/closure.md @@ -115,9 +115,9 @@ Let's start with defining a term that we will be using quite a bit in the rest o *upvar*. An **upvar** is a variable that is local to the function where the closure is defined. So, in the above examples, **x** will be an upvar to the closure. They are also sometimes referred to as the *free variables* meaning they are not bound to the context of the closure. -[`src/librustc/ty/query/mod.rs`][freevars] defines a query called *freevars* for this purpose. +[`src/librustc/ty/query/mod.rs`][upvars] defines a query called *upvars* for this purpose. -[freevars]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/query/queries/struct.freevars.html +[upvars]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/query/queries/struct.upvars.html Other than lazy invocation, one other thing that the distinguishes a closure from a normal function is that it can use the upvars. It borrows these upvars from its surrounding @@ -167,7 +167,7 @@ invokes a callbackfor each upvar that is borrowed, mutated, or moved. ```rust fn main() { - let x = vec![21]; + let mut x = vec![21]; let _cl = || { let y = x[0]; // 1. x[0] += 1; // 2. From f675e3694154c11b1ee18ecfb62c3e757c8fdaf2 Mon Sep 17 00:00:00 2001 From: Eduard-Mihai Burtescu Date: Fri, 14 Jun 2019 19:17:23 +0300 Subject: [PATCH 574/648] Update for TyCtxt<'a, 'gcx, 'tcx> -> TyCtxt<'tcx>. --- src/appendix/code-index.md | 2 +- src/appendix/glossary.md | 1 - src/compiler-debugging.md | 10 +++---- src/conventions.md | 3 +- src/mir/visitor.md | 2 +- src/profiling/with_perf.md | 6 ++-- src/query.md | 16 +++++------ src/ty.md | 58 +++++--------------------------------- src/type-inference.md | 5 ++-- 9 files changed, 27 insertions(+), 76 deletions(-) diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index 222c9b2db..52a7f42d6 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -29,7 +29,7 @@ Item | Kind | Short description | Chapter | `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules] | [src/librustc/ty/trait_def.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/trait_def/struct.TraitDef.html) `TraitRef` | struct | The combination of a trait and its input types (e.g. `P0: Trait`) | [Trait Solving: Goals and Clauses], [Trait Solving: Lowering impls] | [src/librustc/ty/sty.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TraitRef.html) `Ty<'tcx>` | struct | This is the internal representation of a type used for type checking | [Type checking] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/type.Ty.html) -`TyCtxt<'cx, 'tcx, 'tcx>` | struct | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries | [The `ty` modules] | [src/librustc/ty/context.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html) +`TyCtxt<'tcx>` | struct | The "typing context". This is the central data structure in the compiler. It is the context that you use to perform all manner of queries | [The `ty` modules] | [src/librustc/ty/context.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html) [The HIR]: ../hir.html [Identifiers in the HIR]: ../hir.html#hir-id diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index a5d5bf56a..46c5c9f0f 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -25,7 +25,6 @@ early-bound lifetime | a lifetime region that is substituted at its definiti empty type | see "uninhabited type". Fat pointer | a two word value carrying the address of some value, along with some further information necessary to put the value to use. Rust includes two kinds of "fat pointers": references to slices, and trait objects. A reference to a slice carries the starting address of the slice and its length. A trait object carries a value's address and a pointer to the trait's implementation appropriate to that value. "Fat pointers" are also known as "wide pointers", and "double pointers". free variable | a "free variable" is one that is not bound within an expression or term; see [the background chapter for more](./background.html#free-vs-bound) -'gcx | the lifetime of the global arena ([see more](../ty.html)) generics | the set of generic type parameters defined on a type or item HIR | the High-level IR, created by lowering and desugaring the AST ([see more](../hir.html)) HirId | identifies a particular node in the HIR by combining a def-id with an "intra-definition offset". diff --git a/src/compiler-debugging.md b/src/compiler-debugging.md index 0126c2503..8f3a331aa 100644 --- a/src/compiler-debugging.md +++ b/src/compiler-debugging.md @@ -48,7 +48,7 @@ stack backtrace: If you want line numbers for the stack trace, you can enable `debug = true` in your config.toml and rebuild the compiler (`debuginfo-level = 1` will also add -line numbers, but `debug = true` gives full debuginfo). Then the backtrace will +line numbers, but `debug = true` gives full debuginfo). Then the backtrace will look like this: ```text @@ -129,11 +129,11 @@ note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. stack backtrace: (~~~ IRRELEVANT PART OF BACKTRACE REMOVED BY ME ~~~) - 7: rustc::traits::error_reporting::>::report_selection_error + 7: rustc::traits::error_reporting::> + ::report_selection_error at /home/user/rust/src/librustc/traits/error_reporting.rs:823 - 8: rustc::traits::error_reporting::>::report_fulfillment_errors + 8: rustc::traits::error_reporting::> + ::report_fulfillment_errors at /home/user/rust/src/librustc/traits/error_reporting.rs:160 at /home/user/rust/src/librustc/traits/error_reporting.rs:112 9: rustc_typeck::check::FnCtxt::select_obligations_where_possible diff --git a/src/conventions.md b/src/conventions.md index d9f462a05..a941ebff9 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -141,8 +141,7 @@ to the compiler. - `cx` tends to be short for "context" and is often used as a suffix. For example, `tcx` is a common name for the [Typing Context][tcx]. -- [`'tcx` and `'gcx`][tcx] are used as the lifetime names for the Typing - Context. +- [`'tcx`][tcx] is used as the lifetim names for the Typing Context. - Because `crate` is a keyword, if you need a variable to represent something crate-related, often the spelling is changed to `krate`. diff --git a/src/mir/visitor.md b/src/mir/visitor.md index ad00bc3f1..a5e8e9a0b 100644 --- a/src/mir/visitor.md +++ b/src/mir/visitor.md @@ -15,7 +15,7 @@ state you will need while processing MIR: ```rust,ignore struct MyVisitor<...> { - tcx: TyCtxt<'cx, 'tcx, 'tcx>, + tcx: TyCtxt<'tcx>, ... } ``` diff --git a/src/profiling/with_perf.md b/src/profiling/with_perf.md index 2634cc260..7582eece5 100644 --- a/src/profiling/with_perf.md +++ b/src/profiling/with_perf.md @@ -272,9 +272,9 @@ Tree : : | rustc_mir::borrow_check::nll::type_check::type_check_internal (13% total, 0% self) : : : | core::ops::function::FnOnce::call_once (5% total, 0% self) : : : : | rustc_mir::borrow_check::nll::type_check::liveness::generate (5% total, 3% self) -: : : | as rustc::mir::visit::Visitor<'tcx>>::visit_mir (3% total, 0% self) +: : : | as rustc::mir::visit::Visitor<'tcx>>::visit_mir (3% total, 0% self) : | rustc::mir::visit::Visitor::visit_mir (8% total, 6% self) -: | as rustc_mir::dataflow::DataflowResultsConsumer<'cx, 'tcx>>::visit_statement_entry (5% total, 0% self) +: | as rustc_mir::dataflow::DataflowResultsConsumer<'cx, 'tcx>>::visit_statement_entry (5% total, 0% self) : | rustc_mir::dataflow::do_dataflow (3% total, 0% self) ``` @@ -321,7 +321,7 @@ Tree | matched `{do_mir_borrowck}` (100% total, 0% self) : | rustc_mir::borrow_check::nll::compute_regions (47% total, 0% self) [...] : | rustc::mir::visit::Visitor::visit_mir (19% total, 15% self) [...] -: | as rustc_mir::dataflow::DataflowResultsConsumer<'cx, 'tcx>>::visit_statement_entry (13% total, 0% self) [...] +: | as rustc_mir::dataflow::DataflowResultsConsumer<'cx, 'tcx>>::visit_statement_entry (13% total, 0% self) [...] : | rustc_mir::dataflow::do_dataflow (8% total, 1% self) [...] ``` diff --git a/src/query.md b/src/query.md index 4d80187aa..4ab1d079e 100644 --- a/src/query.md +++ b/src/query.md @@ -82,17 +82,15 @@ on how that works). Providers always have the same signature: ```rust,ignore -fn provider<'cx, 'tcx>(tcx: TyCtxt<'cx, 'tcx, 'tcx>, - key: QUERY_KEY) - -> QUERY_RESULT -{ +fn provider<'tcx>( + tcx: TyCtxt<'tcx>, + key: QUERY_KEY, +) -> QUERY_RESULT { ... } ``` -Providers take two arguments: the `tcx` and the query key. Note also -that they take the *global* tcx (i.e. they use the `'tcx` lifetime -twice), rather than taking a tcx with some active inference context. +Providers take two arguments: the `tcx` and the query key. They return the result of the query. #### How providers are setup @@ -103,7 +101,7 @@ is basically a big list of function pointers: ```rust,ignore struct Providers { - type_of: for<'cx, 'tcx> fn(TyCtxt<'cx, 'tcx, 'tcx>, DefId) -> Ty<'tcx>, + type_of: for<'tcx> fn(TyCtxt<'tcx>, DefId) -> Ty<'tcx>, ... } ``` @@ -144,7 +142,7 @@ pub fn provide(providers: &mut Providers) { }; } -fn fubar<'cx, 'tcx>(tcx: TyCtxt<'cx, 'tcx>, key: DefId) -> Fubar<'tcx> { ... } +fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: DefId) -> Fubar<'tcx> { ... } ``` N.B. Most of the `rustc_*` crates only provide **local diff --git a/src/ty.md b/src/ty.md index d9979bc0b..72405d297 100644 --- a/src/ty.md +++ b/src/ty.md @@ -11,63 +11,19 @@ compiler. It is the context that you use to perform all manner of queries. The struct `TyCtxt` defines a reference to this shared context: ```rust,ignore -tcx: TyCtxt<'a, 'gcx, 'tcx> -// -- ---- ---- -// | | | -// | | innermost arena lifetime (if any) -// | "global arena" lifetime -// lifetime of this reference +tcx: TyCtxt<'tcx> +// ---- +// | +// arena lifetime ``` -As you can see, the `TyCtxt` type takes three lifetime parameters. -These lifetimes are perhaps the most complex thing to understand about -the tcx. During Rust compilation, we allocate most of our memory in +As you can see, the `TyCtxt` type takes a lifetime parameter. +During Rust compilation, we allocate most of our memory in **arenas**, which are basically pools of memory that get freed all at -once. When you see a reference with a lifetime like `'tcx` or `'gcx`, +once. When you see a reference with a lifetime like `'tcx`, you know that it refers to arena-allocated data (or data that lives as long as the arenas, anyhow). -We use two distinct levels of arenas. The outer level is the "global -arena". This arena lasts for the entire compilation: so anything you -allocate in there is only freed once compilation is basically over -(actually, when we shift to executing LLVM). - -To reduce peak memory usage, when we do type inference, we also use an -inner level of arena. These arenas get thrown away once type inference -is over. This is done because type inference generates a lot of -"throw-away" types that are not particularly interesting after type -inference completes, so keeping around those allocations would be -wasteful. - -Often, we wish to write code that explicitly asserts that it is not -taking place during inference. In that case, there is no "local" -arena, and all the types that you can access are allocated in the -global arena. To express this, the idea is to use the same lifetime -for the `'gcx` and `'tcx` parameters of `TyCtxt`. Just to be a touch -confusing, we tend to use the name `'tcx` in such contexts. Here is an -example: - -```rust,ignore -fn not_in_inference<'a, 'tcx>(tcx: TyCtxt<'a, 'tcx, 'tcx>, def_id: DefId) { - // ---- ---- - // Using the same lifetime here asserts - // that the innermost arena accessible through - // this reference *is* the global arena. -} -``` - -In contrast, if we want to code that can be usable during type inference, then -you need to declare a distinct `'gcx` and `'tcx` lifetime parameter: - -```rust,ignore -fn maybe_in_inference<'a, 'gcx, 'tcx>(tcx: TyCtxt<'a, 'gcx, 'tcx>, def_id: DefId) { - // ---- ---- - // Using different lifetimes here means that - // the innermost arena *may* be distinct - // from the global arena (but doesn't have to be). -} -``` - ### Allocating and working with types Rust types are represented using the `Ty<'tcx>` defined in the `ty` diff --git a/src/type-inference.md b/src/type-inference.md index d94c06379..d4734525f 100644 --- a/src/type-inference.md +++ b/src/type-inference.md @@ -50,9 +50,8 @@ function and disposed of after it returns. [ty-ch]: ty.html -Within the closure, `infcx` has the type `InferCtxt<'cx, 'gcx, 'tcx>` -for some fresh `'cx` and `'tcx` – the latter corresponds to the lifetime of -this temporary arena, and the `'cx` is the lifetime of the `InferCtxt` itself. +Within the closure, `infcx` has the type `InferCtxt<'cx, 'tcx>` for some +fresh `'cx`, while `'tcx` is the same as outside the inference context. (Again, see the [`ty` chapter][ty-ch] for more details on this setup.) The `tcx.infer_ctxt` method actually returns a builder, which means From f55e97c145cf37fd664db2e0e2f2d05df328bf4f Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Sat, 15 Jun 2019 17:00:21 -0500 Subject: [PATCH 575/648] fix typos --- src/appendix/glossary.md | 2 +- src/conventions.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 46c5c9f0f..3d77def91 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -65,7 +65,7 @@ soundness | soundness is a technical term in type theory. Roughly span | a location in the user's source code, used for error reporting primarily. These are like a file-name/line-number/column tuple on steroids: they carry a start/end point, and also track macro expansions and compiler desugaring. All while being packed into a few bytes (really, it's an index into a table). See the Span datatype for more. substs | the substitutions for a given generic type or item (e.g. the `i32`, `u32` in `HashMap`) tcx | the "typing context", main data structure of the compiler ([see more](../ty.html)) -'tcx | the lifetime of the currently active inference context ([see more](../ty.html)) +'tcx | the lifetime of the allocation arena ([see more](../ty.html)) trait reference | the name of a trait along with a suitable set of input type/lifetimes ([see more](../traits/goals-and-clauses.html#trait-ref)) token | the smallest unit of parsing. Tokens are produced after lexing ([see more](../the-parser.html)). [TLS] | Thread-Local Storage. Variables may be defined so that each thread has its own copy (rather than all threads sharing the variable). This has some interactions with LLVM. Not all platforms support TLS. diff --git a/src/conventions.md b/src/conventions.md index a941ebff9..3660f3379 100644 --- a/src/conventions.md +++ b/src/conventions.md @@ -141,7 +141,7 @@ to the compiler. - `cx` tends to be short for "context" and is often used as a suffix. For example, `tcx` is a common name for the [Typing Context][tcx]. -- [`'tcx`][tcx] is used as the lifetim names for the Typing Context. +- [`'tcx`][tcx] is used as the lifetime name for the Typing Context. - Because `crate` is a keyword, if you need a variable to represent something crate-related, often the spelling is changed to `krate`. From 76a7f1085e9ec293da1b3bf58cfe105aec3b443f Mon Sep 17 00:00:00 2001 From: Amanjeev Sethi Date: Sun, 5 May 2019 15:15:09 -0400 Subject: [PATCH 576/648] Added Rustc Debugger Support Chapter --- src/SUMMARY.md | 1 + src/debugging-support-in-rustc.md | 321 ++++++++++++++++++++++++++++++ 2 files changed, 322 insertions(+) create mode 100644 src/debugging-support-in-rustc.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 273b409ec..663357f00 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -84,6 +84,7 @@ - [Updating LLVM](./codegen/updating-llvm.md) - [Debugging LLVM](./codegen/debugging.md) - [Profile-guided Optimization](./profile-guided-optimization.md) + - [Debugging Support in Rust Compiler](./debugging-support-in-rustc.md) --- diff --git a/src/debugging-support-in-rustc.md b/src/debugging-support-in-rustc.md new file mode 100644 index 000000000..1775b07af --- /dev/null +++ b/src/debugging-support-in-rustc.md @@ -0,0 +1,321 @@ +# Debugging support in the Rust compiler + +This document explains the state of debugging tools support in the Rust compiler (rustc). +The document gives an overview of debugging tools like GDB, LLDB etc. and infrastrcture +around Rust compiler to debug Rust code. If you want to learn how to debug the Rust compiler +itself, then you must see [Debugging the Compiler] page. + +The material is gathered from YouTube video [Tom Tromey discusses debugging support in rustc]. + +## Preliminaries + +### Debuggers + +According to Wikipedia + +> A [debugger or debugging tool] is a computer program that is used to test and debug +> other programs (the "target" program). + +Writing a debugger from scratch for a language requires a lot of work, especially if +debuggers have to be supported on various platforms. GDB and LLDB, however, can be +extended to support debugging a language. This is the path that Rust has chosen. +This document's main goal is to document the said debuggers support in Rust compiler. + +### DWARF + +According to the [DWARF] standard website + +> DWARF is a debugging file format used by many compilers and debuggers to support source level +> debugging. It addresses the requirements of a number of procedural languages, +> such as C, C++, and Fortran, and is designed to be extensible to other languages. +> DWARF is architecture independent and applicable to any processor or operating system. +> It is widely used on Unix, Linux and other operating systems, +> as well as in stand-alone environments. + +DWARF reader is a program that consumes the DWARF format and creates debugger compatible output. +This program may live in the compiler itself. DWARF uses a data structure called +Debugging Information Entry (DIE) which stores the information as "tags" to denote functions, +variables etc., e.g., `DW_TAG_variable`, `DW_TAG_pointer_type`, `DW_TAG_subprogram` etc. +You can also invent your own tags and attributes. + +## Supported debuggers + +### GDB + +We have our own fork of GDB - [https://github.com/rust-dev-tools/gdb] + +#### Rust expression parser + +To be able to show debug output we need an expression parser. +This (GDB) expression parser is written in [Bison] and is only a subset of Rust expressions. +This means that this parser can parse only a subset of Rust expressions. +GDB parser was written from scratch and has no relation to any other parser. +For example, this parser is not related to Rustc's parser. + +GDB has Rust like value and type output. It can print values and types in a way +that look like Rust syntax in the output. Or when you print a type as [ptype] in GDB, +it also looks like Rust source code. Checkout the documentation in the [manual for GDB/Rust]. + +#### Parser extensions + +Expression parser has a couple of extensions in it to facilitate features that you cannot do +with Rust. Some limitations are listed in the [manual for GDB/Rust]. There is some special +code in the DWARF reader in GDB to support the extensions. + +A couple of examples of DWARF reader support needed are as follows - + +1. Enum: Needed for support for enum types. The Rustc writes the information about enum into +DWARF and GDB reads the DWARF to understand where is the tag field or is there a tag +field or is the tag slot shared with non-zero optimization etc. + +2. Dissect trait objects: DWARF extension where the trait object's description in the DWARF +also points to a stub description of the corresponding vtable which in turn points to the +concrete type for which this trait object exists. This means that you can do a `print *object` +for that trait object, and GDB will understand how to find the correct type of the payload in +the trait object. + +**TODO**: Figure out if the following should be mentioned in the GDB-Rust document rather than +this guide page so there is no duplication. This is regarding the following comments: + +[This comment by Tom](https://github.com/rust-lang/rustc-guide/pull/316#discussion_r284027340) +> gdb's Rust extensions and limitations are documented in the gdb manual: +https://sourceware.org/gdb/onlinedocs/gdb/Rust.html -- however, this neglects to mention that +gdb convenience variables and registers follow the gdb $ convention, and that the Rust parser +implements the gdb @ extension. + +[This question by Aman](https://github.com/rust-lang/rustc-guide/pull/316#discussion_r285401353) +> @tromey do you think we should mention this part in the GDB-Rust document rather than this +document so there is no duplication etc.? + +#### Developer notes + +* This work is now upstream. Bugs can be reported in [GDB Bugzilla]. + +### LLDB + +We have our own fork of LLDB - [https://github.com/rust-lang/lldb] + +Fork of LLVM project - [https://github.com/rust-lang/llvm-project] + +LLDB currently only works on macOS because of a dependency issue. This issue was easier to +solve for macOS as compared to Linux. However, Tom has a possible solution which can enable +us to ship LLDB everywhere. + +#### Rust expression parser + +This expression parser is written in C++. It is a type of [Recursive Descent parser]. +Implements slightly less of the Rust language than GDB. LLDB has Rust like value and type output. + +#### Parser extensions + +There is some special code in the DWARF reader in LLDB to support the extensions. +A couple of examples of DWARF reader support needed are as follows - + +1. Enum: Needed for support for enum types. The Rustc writes the information about +enum into DWARF and LLDB reads the DWARF to understand where is the tag field or +is there a tag field or is the tag slot shared with non-zero optimization etc. +In other words, it has enum support as well. + +#### Developer notes + +* None of the LLDB work is upstream. This [rust-lang/lldb wiki page] explains a few details. +* The reason for forking LLDB is that LLDB recently removed all the other language plugins +due to lack of maintenance. +* LLDB has a plugin architecture but that does not work for language support. +* LLDB is available via Rust build (`rustup`). +* GDB generally works better on Linux. + +## DWARF and Rustc + +[DWARF] is the standard way compilers generate debugging information that debuggers read. +It is _the_ debugging format on macOS and Linux. It is a multi-language, extensible format +and is mostly good enough for Rust's purposes. Hence, the current implementation reuses DWARF's +concepts. This is true even if some of the concepts in DWARF do not align with Rust +semantically because generally there can be some kind of mapping between the two. + +We have some DWARF extensions that the Rust compiler emits and the debuggers understand that +are _not_ in the DWARF standard. + +* Rust compiler will emit DWARF for a virtual table, and this `vtable` object will have a + `DW_AT_containing_type` that points to the real type. This lets debuggers dissect a trait object + pointer to correctly find the payload. E.g., here's such a DIE, from a test case in the gdb + repository: + + ```asm + <1><1a9>: Abbrev Number: 3 (DW_TAG_structure_type) + <1aa> DW_AT_containing_type: <0x1b4> + <1ae> DW_AT_name : (indirect string, offset: 0x23d): vtable + <1b2> DW_AT_byte_size : 0 + <1b3> DW_AT_alignment : 8 + ``` + +* The other extension is that the Rust compiler can emit a tagless discriminated union. + See [DWARF feature request] for this item. + +### Current limitations of DWARF + +* Traits - require a bigger change than normal to DWARF, on how to represent Traits in DWARF. +* DWARF provides no way to differentiate between Structs and Tuples. Rust compiler emits +fields with `__0` and debuggers look for a sequence of such names to overcome this limitation. +For example, in this case the debugger would look at a field via `x.__0` instead of `x.0`. +This is resolved via the Rust parser in the debugger so now you can do `x.0`. + +DWARF relies on debuggers to know some information about platform ABI. +Rust does not do that all the time. + +## Developer notes + +This section is from the talk about certain aspects of development. + +## What is missing + +### Shipping GDB in Rustup + +Tracking issue: [https://github.com/rust-lang/rust/issues/34457] + +Shipping GDB requires change to Rustup delivery system. To manage Rustup build size and +times we need to build GDB separately, on its own and somehow provide the artifacts produced +to be included in the final build. However, if we can ship GDB with rustup, it will simplify +the development process by having compiler emit new debug info which can be readily consumed. + +Main issue in achieving this is setting up dependencies. One such dependency is Python. That +is why we have our own fork of GDB because one of the drivers is patched on Rust's side to +check the correct version of Python (Python 2.7 in this case. *Note: Python3 is not chosen +for this purpose because Python's stable ABI is limited and is not sufficient for GDB's needs. +See [https://docs.python.org/3/c-api/stable.html]*). + +This is to keep updates to debugger as fast as possible as we make changes to the debugging symbols. +In essence, to ship the debugger as soon as new debugging info is added. GDB only releases +every six months or so. However, the changes that are +not related to Rust itself should ideally be first merged to upstream eventually. + +### Code signing for LLDB debug server on macOS + +According to Wikipedia, [System Integrity Protection] is + +> System Integrity Protection (SIP, sometimes referred to as rootless) is a security feature +> of Apple's macOS operating system introduced in OS X El Capitan. It comprises a number of +> mechanisms that are enforced by the kernel. A centerpiece is the protection of system-owned +> files and directories against modifications by processes without a specific "entitlement", +> even when executed by the root user or a user with root privileges (sudo). + +It prevents processes using `ptrace` syscall. If a process wants to use `ptrace` it has to be +code signed. The certificate that signs it has to be trusted on your machine. + +See [Apple developer documentation for System Integrity Protection]. + +We may need to sign up with Apple and get the keys to do this signing. Tom has looked into if +Mozilla cannot do this because it is at the maximum number of +keys it is allowed to sign. Tom does not know if Mozilla could get more keys. + +Alternatively, Tom suggests that maybe a Rust legal entity is needed to get the keys via Apple. +This problem is not technical in nature. If we had such a key we could sign GDB as well and +ship that. + +### DWARF and Traits + +Rust traits are not emitted into DWARF at all. The impact of this is calling a method `x.method()` +does not work as is. The reason being that method is implemented by a trait, as opposed +to a type. That information is not present so finding trait methods is missing. + +DWARF has a notion of interface types (possibly added for Java). Tom's idea was to use this +interface type as traits. + +DWARF only deals with concrete names, not the reference types. So, a given implementation of a +trait for a type would be one of these interfaces (`DW_tag_interface` type). Also, the type for +which it is implemented would describe all the interfaces this type implements. This requires a +DWARF extension. + +Issue on Github: [https://github.com/rust-lang/rust/issues/33014] + +## Typical process for a Debug Info change (LLVM) + +LLVM has Debug Info (DI) builders. This is the primary thing that Rust calls into. +This is why we need to change LLVM first because that is emitted first and not DWARF directly. +This is a kind of metadata that you construct and hand-off to LLVM. For the Rustc/LLVM hand-off +some LLVM DI builder methods are called to construct representation of a type. + +The steps of this process are as follows - + +1. LLVM needs changing. + + LLVM does not emit Interface types at all, so this needs to be implemented in the LLVM first. + + Get sign off on LLVM maintainers that this is a good idea. + +2. Change the DWARF extension. + +3. Update the debuggers. + + Update DWARF readers, expression evaluators. + +4. Update Rust compiler. + + Change it to emit this new information. + +### Procedural macro stepping + +A deeply profound question is that how do you actually debug a procedural macro? +What is the location you emit for a macro expansion? Consider some of the following cases - + +* You can emit location of the invocation of the macro. +* You can emit the location of the definition of the macro. +* You can emit locations of the content of the macro. + +RFC: [https://github.com/rust-lang/rfcs/pull/2117] + +Focus is to let macros decide what to do. This can be achieved by having some kind of attribute +that lets the macro tell the compiler where the line marker should be. This affects where you +set the breakpoints and what happens when you step it. + +## Future work + +#### Name mangling changes + +* New demangler in `libiberty` (gcc source tree). +* New demangler in LLVM or LLDB. + +**TODO**: Check the location of the demangler source. +[Question on Github](https://github.com/rust-lang/rustc-guide/pull/316#discussion_r283062536). + +#### Reuse Rust compiler for expressions + +This is an important idea because debuggers by and large do not try to implement type +inference. You need to be much more explicit when you type into the debugger than your +actual source code. So, you cannot just copy and paste an expression from your source +code to debugger and expect the same answer but this would be nice. This can be helped +by using compiler. + +It is certainly doable but it is a large project. You certainly need a bridge to the +debugger because the debugger alone has access to the memory. Both GDB (gcc) and LLDB (clang) +have this feature. LLDB uses Clang to compile code to JIT and GDB can do the same with GCC. + +Both debuggers expression evaluation implement both a superset and a subset of Rust. +They implement just the expression language but they also add some extensions like GDB has +convenience variables. Therefore, if you are taking this route then you not only need +to do this bridge but may have to add some mode to let the compiler understand some extensions. + +#### Windows debugging (PDB) is missing + +This is a complete unknown. + +[Tom Tromey discusses debugging support in rustc]: https://www.youtube.com/watch?v=elBxMRSNYr4 +[Debugging the Compiler]: compiler-debugging.md +[debugger or debugging tool]: https://en.wikipedia.org/wiki/Debugger +[Bison]: https://www.gnu.org/software/bison/ +[ptype]: https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_109.html +[rust-lang/lldb wiki page]: https://github.com/rust-lang/lldb/wiki +[DWARF]: http://dwarfstd.org +[manual for GDB/Rust]: https://sourceware.org/gdb/onlinedocs/gdb/Rust.html +[GDB Bugzilla]: https://sourceware.org/bugzilla/ +[Recursive Descent parser]: https://en.wikipedia.org/wiki/Recursive_descent_parser +[System Integrity Protection]: https://en.wikipedia.org/wiki/System_Integrity_Protection +[https://github.com/rust-dev-tools/gdb]: https://github.com/rust-dev-tools/gdb +[DWARF feature request]: http://dwarfstd.org/ShowIssue.php?issue=180517.2 +[https://docs.python.org/3/c-api/stable.html]: https://docs.python.org/3/c-api/stable.html +[https://github.com/rust-lang/rfcs/pull/2117]: https://github.com/rust-lang/rfcs/pull/2117 +[https://github.com/rust-lang/rust/issues/33014]: https://github.com/rust-lang/rust/issues/33014 +[https://github.com/rust-lang/rust/issues/34457]: https://github.com/rust-lang/rust/issues/34457 +[Apple developer documentation for System Integrity Protection]: https://developer.apple.com/library/archive/releasenotes/MacOSX/WhatsNewInOSX/Articles/MacOSX10_11.html#//apple_ref/doc/uid/TP40016227-SW11 +[https://github.com/rust-lang/lldb]: https://github.com/rust-lang/lldb +[https://github.com/rust-lang/llvm-project]: https://github.com/rust-lang/llvm-project From 43ac5672c393b949cd7f765cc2b9a0725757cfab Mon Sep 17 00:00:00 2001 From: b41sh Date: Mon, 24 Jun 2019 21:43:27 +0800 Subject: [PATCH 577/648] fix compiler-team --- src/compiler-team.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/compiler-team.md b/src/compiler-team.md index 0f0808b79..968e2f142 100644 --- a/src/compiler-team.md +++ b/src/compiler-team.md @@ -95,7 +95,7 @@ The guidelines for reviewers are as follows: ### high-five -Once you have r+ rights, you can also be added to the high-five +Once you have r+ rights, you can also be added to the [high-five][hi5] rotation. high-five is the bot that assigns incoming PRs to reviewers. If you are added, you will be randomly selected to review PRs. If you find you are assigned a PR that you don't feel comfortable From 6f50f15f7a33bbe6ce528a699b66ce22b5a1949c Mon Sep 17 00:00:00 2001 From: flip1995 Date: Mon, 24 Jun 2019 12:59:12 +0200 Subject: [PATCH 578/648] Change stage0 cfg_attr to bootstrap --- src/stabilization_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index 56da5cdaf..76446150c 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -125,7 +125,7 @@ writing, the next stable release (i.e. what is currently beta) was Next search for the feature string (in this case, `pub_restricted`) in the codebase to find where it appears. Change uses of `#![feature(XXX)]` from the `libstd` and any rustc crates to be -`#![cfg_attr(stage0, feature(XXX))]`. This includes the feature-gate +`#![cfg_attr(bootstrap, feature(XXX))]`. This includes the feature-gate only for stage0, which is built using the current beta (this is needed because the feature is still unstable in the current beta). From 9f3678dece076f2f8bdb36a9e09a10333349028c Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Wed, 19 Jun 2019 13:11:01 -0500 Subject: [PATCH 579/648] Update mdbook --- ci/install.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ci/install.sh b/ci/install.sh index 835f23192..d22d2c6c1 100755 --- a/ci/install.sh +++ b/ci/install.sh @@ -20,5 +20,5 @@ function cargo_install() { fi } -cargo_install mdbook 0.2.3 -cargo_install mdbook-linkcheck 0.2.3 +cargo_install mdbook 0.3.0 +cargo_install mdbook-linkcheck 0.2.4 From d16c32661a70cfe82806c887a3520e7517092302 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Fri, 21 Jun 2019 09:57:52 -0500 Subject: [PATCH 580/648] Update to mdbook-linkcheck 0.3.0 --- ci/install.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install.sh b/ci/install.sh index d22d2c6c1..7c730cae3 100755 --- a/ci/install.sh +++ b/ci/install.sh @@ -21,4 +21,4 @@ function cargo_install() { } cargo_install mdbook 0.3.0 -cargo_install mdbook-linkcheck 0.2.4 +cargo_install mdbook-linkcheck 0.3.0 From bb872a6072981f1875c446ecf1c4b40734a9510d Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Tue, 25 Jun 2019 15:16:30 -0500 Subject: [PATCH 581/648] add bibligraphy appendix --- src/SUMMARY.md | 1 + src/appendix/bibliography.md | 89 ++++++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+) create mode 100644 src/appendix/bibliography.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 663357f00..45b05c0c9 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -92,4 +92,5 @@ [Appendix B: Background material](./appendix/background.md) [Appendix C: Glossary](./appendix/glossary.md) [Appendix D: Code Index](./appendix/code-index.md) +[Appendix E: Bibliography](./appendix/bibliography.md) [](./important-links.md) diff --git a/src/appendix/bibliography.md b/src/appendix/bibliography.md new file mode 100644 index 000000000..64e014f91 --- /dev/null +++ b/src/appendix/bibliography.md @@ -0,0 +1,89 @@ +# Rust Bibliography + +This is a reading list of material relevant to Rust. It includes prior +research that has - at one time or another - influenced the design of +Rust, as well as publications about Rust. + +## Type system + +* [Region based memory management in Cyclone](http://209.68.42.137/ucsd-pages/Courses/cse227.w03/handouts/cyclone-regions.pdf) +* [Safe manual memory management in Cyclone](http://www.cs.umd.edu/projects/PL/cyclone/scp.pdf) +* [Typeclasses: making ad-hoc polymorphism less ad hoc](http://www.ps.uni-sb.de/courses/typen-ws99/class.ps.gz) +* [Macros that work together](https://www.cs.utah.edu/plt/publications/jfp12-draft-fcdf.pdf) +* [Traits: composable units of behavior](http://scg.unibe.ch/archive/papers/Scha03aTraits.pdf) +* [Alias burying](http://www.cs.uwm.edu/faculty/boyland/papers/unique-preprint.ps) - We tried something similar and abandoned it. +* [External uniqueness is unique enough](http://www.cs.uu.nl/research/techreps/UU-CS-2002-048.html) +* [Uniqueness and Reference Immutability for Safe Parallelism](https://research.microsoft.com/pubs/170528/msr-tr-2012-79.pdf) +* [Region Based Memory Management](http://www.cs.ucla.edu/~palsberg/tba/papers/tofte-talpin-iandc97.pdf) + +## Concurrency + +* [Singularity: rethinking the software stack](https://research.microsoft.com/pubs/69431/osr2007_rethinkingsoftwarestack.pdf) +* [Language support for fast and reliable message passing in singularity OS](https://research.microsoft.com/pubs/67482/singsharp.pdf) +* [Scheduling multithreaded computations by work stealing](http://supertech.csail.mit.edu/papers/steal.pdf) +* [Thread scheduling for multiprogramming multiprocessors](http://www.eecis.udel.edu/%7Ecavazos/cisc879-spring2008/papers/arora98thread.pdf) +* [The data locality of work stealing](http://www.aladdin.cs.cmu.edu/papers/pdfs/y2000/locality_spaa00.pdf) +* [Dynamic circular work stealing deque](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.170.1097&rep=rep1&type=pdf) - The Chase/Lev deque +* [Work-first and help-first scheduling policies for async-finish task parallelism](http://www.cs.rice.edu/%7Eyguo/pubs/PID824943.pdf) - More general than fully-strict work stealing +* [A Java fork/join calamity](http://www.coopsoft.com/ar/CalamityArticle.html) - critique of Java's fork/join library, particularly its application of work stealing to non-strict computation +* [Scheduling techniques for concurrent systems](http://www.stanford.edu/~ouster/cgi-bin/papers/coscheduling.pdf) +* [Contention aware scheduling](http://www.blagodurov.net/files/a8-blagodurov.pdf) +* [Balanced work stealing for time-sharing multicores](http://www.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-12-1.pdf) +* [Three layer cake for shared-memory programming](http://dl.acm.org/citation.cfm?id=1953616&dl=ACM&coll=DL&CFID=524387192&CFTOKEN=44362705) +* [Non-blocking steal-half work queues](http://www.cs.bgu.ac.il/%7Ehendlerd/papers/p280-hendler.pdf) +* [Reagents: expressing and composing fine-grained concurrency](http://www.mpi-sws.org/~turon/reagents.pdf) +* [Algorithms for scalable synchronization of shared-memory multiprocessors](https://www.cs.rochester.edu/u/scott/papers/1991_TOCS_synch.pdf) +* [Epoch-based reclamation](https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-579.pdf). + +## Others + +* [Crash-only software](https://www.usenix.org/legacy/events/hotos03/tech/full_papers/candea/candea.pdf) +* [Composing High-Performance Memory Allocators](http://people.cs.umass.edu/~emery/pubs/berger-pldi2001.pdf) +* [Reconsidering Custom Memory Allocation](http://people.cs.umass.edu/~emery/pubs/berger-oopsla2002.pdf) + +## Papers *about* Rust + +* [GPU Programming in Rust: Implementing High Level Abstractions in a +Systems Level +Language](http://www.cs.indiana.edu/~eholk/papers/hips2013.pdf). Early GPU work by Eric Holk. +* [Parallel closures: a new twist on an old + idea](https://www.usenix.org/conference/hotpar12/parallel-closures-new-twist-old-idea) + - not exactly about Rust, but by nmatsakis +* [Patina: A Formalization of the Rust Programming + Language](ftp://ftp.cs.washington.edu/tr/2015/03/UW-CSE-15-03-02.pdf). Early + formalization of a subset of the type system, by Eric Reed. +* [Experience Report: Developing the Servo Web Browser Engine using + Rust](http://arxiv.org/abs/1505.07383). By Lars Bergstrom. +* [Implementing a Generic Radix Trie in + Rust](https://michaelsproul.github.io/rust_radix_paper/rust-radix-sproul.pdf). Undergrad + paper by Michael Sproul. +* [Reenix: Implementing a Unix-Like Operating System in + Rust](http://scialex.github.io/reenix.pdf). Undergrad paper by Alex + Light. +* [Evaluation of performance and productivity metrics of potential programming languages in the HPC environment](http://octarineparrot.com/assets/mrfloya-thesis-ba.pdf). + Bachelor's thesis by Florian Wilkens. Compares C, Go and Rust. +* [Nom, a byte oriented, streaming, zero copy, parser combinators library + in Rust](http://spw15.langsec.org/papers/couprie-nom.pdf). By + Geoffroy Couprie, research for VLC. +* [Graph-Based Higher-Order Intermediate + Representation](http://compilers.cs.uni-saarland.de/papers/lkh15_cgo.pdf). An + experimental IR implemented in Impala, a Rust-like language. +* [Code Refinement of Stencil + Codes](http://compilers.cs.uni-saarland.de/papers/ppl14_web.pdf). Another + paper using Impala. +* [Parallelization in Rust with fork-join and + friends](http://publications.lib.chalmers.se/records/fulltext/219016/219016.pdf). Linus + Farnstrand's master's thesis. +* [Session Types for + Rust](http://munksgaard.me/papers/laumann-munksgaard-larsen.pdf). Philip + Munksgaard's master's thesis. Research for Servo. +* [Ownership is Theft: Experiences Building an Embedded OS in Rust - Amit Levy, et. al.](http://amitlevy.com/papers/tock-plos2015.pdf) +* [You can't spell trust without Rust](https://raw.githubusercontent.com/Gankro/thesis/master/thesis.pdf). Alexis Beingessner's master's thesis. +* [Rust-Bio: a fast and safe bioinformatics library](http://bioinformatics.oxfordjournals.org/content/early/2015/10/06/bioinformatics.btv573). Johannes Köster +* [Safe, Correct, and Fast Low-Level Networking](https://octarineparrot.com/assets/msci_paper.pdf). Robert Clipsham's master's thesis. +* [Formalizing Rust traits](http://hdl.handle.net/2429/55609). Jonatan Milewski's master's thesis. +* [Rust as a Language for High Performance GC Implementation](http://users.cecs.anu.edu.au/~steveb/downloads/pdf/rust-ismm-2016.pdf) +* [Simple Verification of Rust Programs via Functional Purification](https://github.com/Kha/electrolysis). Sebastian Ullrich's master's thesis. +* [Writing parsers like it is 2017](http://spw17.langsec.org/papers/chifflier-parsing-in-2017.pdf) Pierre Chifflier and Geoffroy Couprie for the Langsec Workshop +* [The Case for Writing a Kernel in Rust](https://www.tockos.org/assets/papers/rust-kernel-apsys2017.pdf) +* [RustBelt: Securing the Foundations of the Rust Programming Language](https://plv.mpi-sws.org/rustbelt/popl18/) From 979387d4c23403dab9a9553557ecaf93045eb49d Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 26 Jun 2019 10:32:01 -0500 Subject: [PATCH 582/648] fix broken links --- src/appendix/bibliography.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/appendix/bibliography.md b/src/appendix/bibliography.md index 64e014f91..a5bd84b25 100644 --- a/src/appendix/bibliography.md +++ b/src/appendix/bibliography.md @@ -6,7 +6,7 @@ Rust, as well as publications about Rust. ## Type system -* [Region based memory management in Cyclone](http://209.68.42.137/ucsd-pages/Courses/cse227.w03/handouts/cyclone-regions.pdf) +* [Region based memory management in Cyclone](https://www.cs.umd.edu/projects/cyclone/papers/cyclone-regions.pdf) * [Safe manual memory management in Cyclone](http://www.cs.umd.edu/projects/PL/cyclone/scp.pdf) * [Typeclasses: making ad-hoc polymorphism less ad hoc](http://www.ps.uni-sb.de/courses/typen-ws99/class.ps.gz) * [Macros that work together](https://www.cs.utah.edu/plt/publications/jfp12-draft-fcdf.pdf) @@ -31,7 +31,7 @@ Rust, as well as publications about Rust. * [Balanced work stealing for time-sharing multicores](http://www.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-12-1.pdf) * [Three layer cake for shared-memory programming](http://dl.acm.org/citation.cfm?id=1953616&dl=ACM&coll=DL&CFID=524387192&CFTOKEN=44362705) * [Non-blocking steal-half work queues](http://www.cs.bgu.ac.il/%7Ehendlerd/papers/p280-hendler.pdf) -* [Reagents: expressing and composing fine-grained concurrency](http://www.mpi-sws.org/~turon/reagents.pdf) +* [Reagents: expressing and composing fine-grained concurrency](http://aturon.github.io/academic/reagents.pdf) * [Algorithms for scalable synchronization of shared-memory multiprocessors](https://www.cs.rochester.edu/u/scott/papers/1991_TOCS_synch.pdf) * [Epoch-based reclamation](https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-579.pdf). @@ -43,9 +43,10 @@ Rust, as well as publications about Rust. ## Papers *about* Rust -* [GPU Programming in Rust: Implementing High Level Abstractions in a -Systems Level -Language](http://www.cs.indiana.edu/~eholk/papers/hips2013.pdf). Early GPU work by Eric Holk. +* [GPU Programming in Rust: Implementing High Level Abstractions in a Systems + Level + Language](https://www.cs.indiana.edu/~achauhan/Publications/Pubs/2013-hips-holk-rust.pdf). + Early GPU work by Eric Holk. * [Parallel closures: a new twist on an old idea](https://www.usenix.org/conference/hotpar12/parallel-closures-new-twist-old-idea) - not exactly about Rust, but by nmatsakis From 85c06162882ba42696a76f653badf39e26d494e7 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 26 Jun 2019 10:41:07 -0500 Subject: [PATCH 583/648] avoid ftp links --- src/appendix/bibliography.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/appendix/bibliography.md b/src/appendix/bibliography.md index a5bd84b25..19a9217f7 100644 --- a/src/appendix/bibliography.md +++ b/src/appendix/bibliography.md @@ -51,8 +51,8 @@ Rust, as well as publications about Rust. idea](https://www.usenix.org/conference/hotpar12/parallel-closures-new-twist-old-idea) - not exactly about Rust, but by nmatsakis * [Patina: A Formalization of the Rust Programming - Language](ftp://ftp.cs.washington.edu/tr/2015/03/UW-CSE-15-03-02.pdf). Early - formalization of a subset of the type system, by Eric Reed. + Language](http://dada.cs.washington.edu/research/tr/2015/03/UW-CSE-15-03-02.pdf). + Early formalization of a subset of the type system, by Eric Reed. * [Experience Report: Developing the Servo Web Browser Engine using Rust](http://arxiv.org/abs/1505.07383). By Lars Bergstrom. * [Implementing a Generic Radix Trie in From 40e84619738ced662efc5aeaa076b4d8d415e989 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 18 Jun 2019 10:08:04 -0400 Subject: [PATCH 584/648] fix typo --- src/borrow_check/region_inference.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index 7fe50b870..fc3978436 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -48,7 +48,7 @@ The MIR-based region analysis consists of two major functions: ## Universal regions -The [`UnversalRegions`] type represents a collection of _universal_ regions +The [`UniversalRegions`] type represents a collection of _universal_ regions corresponding to some MIR `DefId`. It is constructed in [`replace_regions_in_mir`] when we replace all regions with fresh inference variables. [`UniversalRegions`] contains indices for all the free regions in From 83ab6e429644242e204b3de85d2dea7ba79509de Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 18 Jun 2019 10:17:36 -0400 Subject: [PATCH 585/648] break out parts of the region inference chapter into sub-chapters --- src/SUMMARY.md | 4 + src/borrow_check/region_inference.md | 457 ------------------ .../region_inference/closure_constraints.md | 10 + .../constraint_propagation.md | 3 + .../region_inference/error_reporting.md | 3 + .../placeholders_and_universes.md | 441 +++++++++++++++++ 6 files changed, 461 insertions(+), 457 deletions(-) create mode 100644 src/borrow_check/region_inference/closure_constraints.md create mode 100644 src/borrow_check/region_inference/constraint_propagation.md create mode 100644 src/borrow_check/region_inference/error_reporting.md create mode 100644 src/borrow_check/region_inference/placeholders_and_universes.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 45b05c0c9..7e9a9ee54 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -76,6 +76,10 @@ - [Move paths](./borrow_check/moves_and_initialization/move_paths.md) - [MIR type checker](./borrow_check/type_check.md) - [Region inference](./borrow_check/region_inference.md) + - [Constraint propagation](./borrow_check/region_inference/constraint_propagation.md) + - [Placeholders and universes](./borrow_check/region_inference/placeholders_and_universes.md) + - [Closure constraints](./borrow_check/region_inference/closure_constraints.md) + - [Errror reporting](./borrow_check/region_inference/error_reporting.md) - [Two-phase-borrows](./borrow_check/two_phase_borrows.md) - [Constant evaluation](./const-eval.md) - [miri const evaluator](./miri.md) diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index fc3978436..5279d210b 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -234,460 +234,3 @@ tests and universal regions, as discussed above. [`check_type_tests`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.check_type_tests [`check_universal_regions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.check_universal_regions -## Closures - -When we are checking the type tests and universal regions, we may come across a -constraint that we can't prove yet if we are in a closure body! However, the -necessary constraints may actually hold (we just don't know it yet). Thus, if -we are inside a closure, we just collect all the constraints we can't prove yet -and return them. Later, when we are borrow check the MIR node that created the -closure, we can also check that these constraints hold. At that time, if we -can't prove they hold, we report an error. - -## Placeholders and universes - -(This section describes ongoing work that hasn't landed yet.) - -From time to time we have to reason about regions that we can't -concretely know. For example, consider this program: - -```rust,ignore -// A function that needs a static reference -fn foo(x: &'static u32) { } - -fn bar(f: for<'a> fn(&'a u32)) { - // ^^^^^^^^^^^^^^^^^^^ a function that can accept **any** reference - let x = 22; - f(&x); -} - -fn main() { - bar(foo); -} -``` - -This program ought not to type-check: `foo` needs a static reference -for its argument, and `bar` wants to be given a function that that -accepts **any** reference (so it can call it with something on its -stack, for example). But *how* do we reject it and *why*? - -### Subtyping and Placeholders - -When we type-check `main`, and in particular the call `bar(foo)`, we -are going to wind up with a subtyping relationship like this one: - -```text -fn(&'static u32) <: for<'a> fn(&'a u32) ----------------- ------------------- -the type of `foo` the type `bar` expects -``` - -We handle this sort of subtyping by taking the variables that are -bound in the supertype and replacing them with -[universally quantified](../appendix/background.html#quantified) -representatives, written like `!1`. We call these regions "placeholder -regions" – they represent, basically, "some unknown region". - -Once we've done that replacement, we have the following relation: - -```text -fn(&'static u32) <: fn(&'!1 u32) -``` - -The key idea here is that this unknown region `'!1` is not related to -any other regions. So if we can prove that the subtyping relationship -is true for `'!1`, then it ought to be true for any region, which is -what we wanted. - -So let's work through what happens next. To check if two functions are -subtypes, we check if their arguments have the desired relationship -(fn arguments are [contravariant](../appendix/background.html#variance), so -we swap the left and right here): - -```text -&'!1 u32 <: &'static u32 -``` - -According to the basic subtyping rules for a reference, this will be -true if `'!1: 'static`. That is – if "some unknown region `!1`" lives -outlives `'static`. Now, this *might* be true – after all, `'!1` -could be `'static` – but we don't *know* that it's true. So this -should yield up an error (eventually). - -### What is a universe - -In the previous section, we introduced the idea of a placeholder -region, and we denoted it `!1`. We call this number `1` the **universe -index**. The idea of a "universe" is that it is a set of names that -are in scope within some type or at some point. Universes are formed -into a tree, where each child extends its parents with some new names. -So the **root universe** conceptually contains global names, such as -the the lifetime `'static` or the type `i32`. In the compiler, we also -put generic type parameters into this root universe (in this sense, -there is not just one root universe, but one per item). So consider -this function `bar`: - -```rust,ignore -struct Foo { } - -fn bar<'a, T>(t: &'a T) { - ... -} -``` - -Here, the root universe would consist of the lifetimes `'static` and -`'a`. In fact, although we're focused on lifetimes here, we can apply -the same concept to types, in which case the types `Foo` and `T` would -be in the root universe (along with other global types, like `i32`). -Basically, the root universe contains all the names that -[appear free](../appendix/background.html#free-vs-bound) in the body of `bar`. - -Now let's extend `bar` a bit by adding a variable `x`: - -```rust,ignore -fn bar<'a, T>(t: &'a T) { - let x: for<'b> fn(&'b u32) = ...; -} -``` - -Here, the name `'b` is not part of the root universe. Instead, when we -"enter" into this `for<'b>` (e.g., by replacing it with a placeholder), we will create -a child universe of the root, let's call it U1: - -```text -U0 (root universe) -│ -└─ U1 (child universe) -``` - -The idea is that this child universe U1 extends the root universe U0 -with a new name, which we are identifying by its universe number: -`!1`. - -Now let's extend `bar` a bit by adding one more variable, `y`: - -```rust,ignore -fn bar<'a, T>(t: &'a T) { - let x: for<'b> fn(&'b u32) = ...; - let y: for<'c> fn(&'b u32) = ...; -} -``` - -When we enter *this* type, we will again create a new universe, which -we'll call `U2`. Its parent will be the root universe, and U1 will be -its sibling: - -```text -U0 (root universe) -│ -├─ U1 (child universe) -│ -└─ U2 (child universe) -``` - -This implies that, while in U2, we can name things from U0 or U2, but -not U1. - -**Giving existential variables a universe.** Now that we have this -notion of universes, we can use it to extend our type-checker and -things to prevent illegal names from leaking out. The idea is that we -give each inference (existential) variable – whether it be a type or -a lifetime – a universe. That variable's value can then only -reference names visible from that universe. So for example is a -lifetime variable is created in U0, then it cannot be assigned a value -of `!1` or `!2`, because those names are not visible from the universe -U0. - -**Representing universes with just a counter.** You might be surprised -to see that the compiler doesn't keep track of a full tree of -universes. Instead, it just keeps a counter – and, to determine if -one universe can see another one, it just checks if the index is -greater. For example, U2 can see U0 because 2 >= 0. But U0 cannot see -U2, because 0 >= 2 is false. - -How can we get away with this? Doesn't this mean that we would allow -U2 to also see U1? The answer is that, yes, we would, **if that -question ever arose**. But because of the structure of our type -checker etc, there is no way for that to happen. In order for -something happening in the universe U1 to "communicate" with something -happening in U2, they would have to have a shared inference variable X -in common. And because everything in U1 is scoped to just U1 and its -children, that inference variable X would have to be in U0. And since -X is in U0, it cannot name anything from U1 (or U2). This is perhaps easiest -to see by using a kind of generic "logic" example: - -```text -exists { - forall { ... /* Y is in U1 ... */ } - forall { ... /* Z is in U2 ... */ } -} -``` - -Here, the only way for the two foralls to interact would be through X, -but neither Y nor Z are in scope when X is declared, so its value -cannot reference either of them. - -### Universes and placeholder region elements - -But where does that error come from? The way it happens is like this. -When we are constructing the region inference context, we can tell -from the type inference context how many placeholder variables exist -(the `InferCtxt` has an internal counter). For each of those, we -create a corresponding universal region variable `!n` and a "region -element" `placeholder(n)`. This corresponds to "some unknown set of other -elements". The value of `!n` is `{placeholder(n)}`. - -At the same time, we also give each existential variable a -**universe** (also taken from the `InferCtxt`). This universe -determines which placeholder elements may appear in its value: For -example, a variable in universe U3 may name `placeholder(1)`, `placeholder(2)`, and -`placeholder(3)`, but not `placeholder(4)`. Note that the universe of an inference -variable controls what region elements **can** appear in its value; it -does not say region elements **will** appear. - -### Placeholders and outlives constraints - -In the region inference engine, outlives constraints have the form: - -```text -V1: V2 @ P -``` - -where `V1` and `V2` are region indices, and hence map to some region -variable (which may be universally or existentially quantified). The -`P` here is a "point" in the control-flow graph; it's not important -for this section. This variable will have a universe, so let's call -those universes `U(V1)` and `U(V2)` respectively. (Actually, the only -one we are going to care about is `U(V1)`.) - -When we encounter this constraint, the ordinary procedure is to start -a DFS from `P`. We keep walking so long as the nodes we are walking -are present in `value(V2)` and we add those nodes to `value(V1)`. If -we reach a return point, we add in any `end(X)` elements. That part -remains unchanged. - -But then *after that* we want to iterate over the placeholder `placeholder(x)` -elements in V2 (each of those must be visible to `U(V2)`, but we -should be able to just assume that is true, we don't have to check -it). We have to ensure that `value(V1)` outlives each of those -placeholder elements. - -Now there are two ways that could happen. First, if `U(V1)` can see -the universe `x` (i.e., `x <= U(V1)`), then we can just add `placeholder(x)` -to `value(V1)` and be done. But if not, then we have to approximate: -we may not know what set of elements `placeholder(x)` represents, but we -should be able to compute some sort of **upper bound** B for it – -some region B that outlives `placeholder(x)`. For now, we'll just use -`'static` for that (since it outlives everything) – in the future, we -can sometimes be smarter here (and in fact we have code for doing this -already in other contexts). Moreover, since `'static` is in the root -universe U0, we know that all variables can see it – so basically if -we find that `value(V2)` contains `placeholder(x)` for some universe `x` -that `V1` can't see, then we force `V1` to `'static`. - -### Extending the "universal regions" check - -After all constraints have been propagated, the NLL region inference -has one final check, where it goes over the values that wound up being -computed for each universal region and checks that they did not get -'too large'. In our case, we will go through each placeholder region -and check that it contains *only* the `placeholder(u)` element it is known to -outlive. (Later, we might be able to know that there are relationships -between two placeholder regions and take those into account, as we do -for universal regions from the fn signature.) - -Put another way, the "universal regions" check can be considered to be -checking constraints like: - -```text -{placeholder(1)}: V1 -``` - -where `{placeholder(1)}` is like a constant set, and V1 is the variable we -made to represent the `!1` region. - -## Back to our example - -OK, so far so good. Now let's walk through what would happen with our -first example: - -```text -fn(&'static u32) <: fn(&'!1 u32) @ P // this point P is not imp't here -``` - -The region inference engine will create a region element domain like this: - -```text -{ CFG; end('static); placeholder(1) } - --- ------------ ------- from the universe `!1` - | 'static is always in scope - all points in the CFG; not especially relevant here -``` - -It will always create two universal variables, one representing -`'static` and one representing `'!1`. Let's call them Vs and V1. They -will have initial values like so: - -```text -Vs = { CFG; end('static) } // it is in U0, so can't name anything else -V1 = { placeholder(1) } -``` - -From the subtyping constraint above, we would have an outlives constraint like - -```text -'!1: 'static @ P -``` - -To process this, we would grow the value of V1 to include all of Vs: - -```text -Vs = { CFG; end('static) } -V1 = { CFG; end('static), placeholder(1) } -``` - -At that point, constraint propagation is complete, because all the -outlives relationships are satisfied. Then we would go to the "check -universal regions" portion of the code, which would test that no -universal region grew too large. - -In this case, `V1` *did* grow too large – it is not known to outlive -`end('static)`, nor any of the CFG – so we would report an error. - -## Another example - -What about this subtyping relationship? - -```text -for<'a> fn(&'a u32, &'a u32) - <: -for<'b, 'c> fn(&'b u32, &'c u32) -``` - -Here we would replace the bound region in the supertype with a placeholder, as before, yielding: - -```text -for<'a> fn(&'a u32, &'a u32) - <: -fn(&'!1 u32, &'!2 u32) -``` - -then we instantiate the variable on the left-hand side with an -existential in universe U2, yielding the following (`?n` is a notation -for an existential variable): - -```text -fn(&'?3 u32, &'?3 u32) - <: -fn(&'!1 u32, &'!2 u32) -``` - -Then we break this down further: - -```text -&'!1 u32 <: &'?3 u32 -&'!2 u32 <: &'?3 u32 -``` - -and even further, yield up our region constraints: - -```text -'!1: '?3 -'!2: '?3 -``` - -Note that, in this case, both `'!1` and `'!2` have to outlive the -variable `'?3`, but the variable `'?3` is not forced to outlive -anything else. Therefore, it simply starts and ends as the empty set -of elements, and hence the type-check succeeds here. - -(This should surprise you a little. It surprised me when I first realized it. -We are saying that if we are a fn that **needs both of its arguments to have -the same region**, we can accept being called with **arguments with two -distinct regions**. That seems intuitively unsound. But in fact, it's fine, as -I tried to explain in [this issue][ohdeargoditsallbroken] on the Rust issue -tracker long ago. The reason is that even if we get called with arguments of -two distinct lifetimes, those two lifetimes have some intersection (the call -itself), and that intersection can be our value of `'a` that we use as the -common lifetime of our arguments. -nmatsakis) - -[ohdeargoditsallbroken]: https://github.com/rust-lang/rust/issues/32330#issuecomment-202536977 - -## Final example - -Let's look at one last example. We'll extend the previous one to have -a return type: - -```text -for<'a> fn(&'a u32, &'a u32) -> &'a u32 - <: -for<'b, 'c> fn(&'b u32, &'c u32) -> &'b u32 -``` - -Despite seeming very similar to the previous example, this case is going to get -an error. That's good: the problem is that we've gone from a fn that promises -to return one of its two arguments, to a fn that is promising to return the -first one. That is unsound. Let's see how it plays out. - -First, we replace the bound region in the supertype with a placeholder: - -```text -for<'a> fn(&'a u32, &'a u32) -> &'a u32 - <: -fn(&'!1 u32, &'!2 u32) -> &'!1 u32 -``` - -Then we instantiate the subtype with existentials (in U2): - -```text -fn(&'?3 u32, &'?3 u32) -> &'?3 u32 - <: -fn(&'!1 u32, &'!2 u32) -> &'!1 u32 -``` - -And now we create the subtyping relationships: - -```text -&'!1 u32 <: &'?3 u32 // arg 1 -&'!2 u32 <: &'?3 u32 // arg 2 -&'?3 u32 <: &'!1 u32 // return type -``` - -And finally the outlives relationships. Here, let V1, V2, and V3 be the -variables we assign to `!1`, `!2`, and `?3` respectively: - -```text -V1: V3 -V2: V3 -V3: V1 -``` - -Those variables will have these initial values: - -```text -V1 in U1 = {placeholder(1)} -V2 in U2 = {placeholder(2)} -V3 in U2 = {} -``` - -Now because of the `V3: V1` constraint, we have to add `placeholder(1)` into `V3` (and -indeed it is visible from `V3`), so we get: - -```text -V3 in U2 = {placeholder(1)} -``` - -then we have this constraint `V2: V3`, so we wind up having to enlarge -`V2` to include `placeholder(1)` (which it can also see): - -```text -V2 in U2 = {placeholder(1), placeholder(2)} -``` - -Now constraint propagation is done, but when we check the outlives -relationships, we find that `V2` includes this new element `placeholder(1)`, -so we report an error. - -## Borrow Checker Errors - -TODO: we should discuss how to generate errors from the results of these analyses. diff --git a/src/borrow_check/region_inference/closure_constraints.md b/src/borrow_check/region_inference/closure_constraints.md new file mode 100644 index 000000000..13230d037 --- /dev/null +++ b/src/borrow_check/region_inference/closure_constraints.md @@ -0,0 +1,10 @@ +# Propagating closure constraints + +When we are checking the type tests and universal regions, we may come +across a constraint that we can't prove yet if we are in a closure +body! However, the necessary constraints may actually hold (we just +don't know it yet). Thus, if we are inside a closure, we just collect +all the constraints we can't prove yet and return them. Later, when we +are borrow check the MIR node that created the closure, we can also +check that these constraints hold. At that time, if we can't prove +they hold, we report an error. diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md new file mode 100644 index 000000000..2e69629b1 --- /dev/null +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -0,0 +1,3 @@ +# Constraint propagation + +The main work of the region inference is **constraint propagation**. diff --git a/src/borrow_check/region_inference/error_reporting.md b/src/borrow_check/region_inference/error_reporting.md new file mode 100644 index 000000000..79b3e077c --- /dev/null +++ b/src/borrow_check/region_inference/error_reporting.md @@ -0,0 +1,3 @@ +# Reporting region errors + +TODO: we should discuss how to generate errors from the results of these analyses. diff --git a/src/borrow_check/region_inference/placeholders_and_universes.md b/src/borrow_check/region_inference/placeholders_and_universes.md new file mode 100644 index 000000000..0d4068e00 --- /dev/null +++ b/src/borrow_check/region_inference/placeholders_and_universes.md @@ -0,0 +1,441 @@ +# Placeholders and universes + +From time to time we have to reason about regions that we can't +concretely know. For example, consider this program: + +```rust,ignore +// A function that needs a static reference +fn foo(x: &'static u32) { } + +fn bar(f: for<'a> fn(&'a u32)) { + // ^^^^^^^^^^^^^^^^^^^ a function that can accept **any** reference + let x = 22; + f(&x); +} + +fn main() { + bar(foo); +} +``` + +This program ought not to type-check: `foo` needs a static reference +for its argument, and `bar` wants to be given a function that that +accepts **any** reference (so it can call it with something on its +stack, for example). But *how* do we reject it and *why*? + +## Subtyping and Placeholders + +When we type-check `main`, and in particular the call `bar(foo)`, we +are going to wind up with a subtyping relationship like this one: + +```text +fn(&'static u32) <: for<'a> fn(&'a u32) +---------------- ------------------- +the type of `foo` the type `bar` expects +``` + +We handle this sort of subtyping by taking the variables that are +bound in the supertype and replacing them with +[universally quantified](../appendix/background.html#quantified) +representatives, written like `!1`. We call these regions "placeholder +regions" – they represent, basically, "some unknown region". + +Once we've done that replacement, we have the following relation: + +```text +fn(&'static u32) <: fn(&'!1 u32) +``` + +The key idea here is that this unknown region `'!1` is not related to +any other regions. So if we can prove that the subtyping relationship +is true for `'!1`, then it ought to be true for any region, which is +what we wanted. + +So let's work through what happens next. To check if two functions are +subtypes, we check if their arguments have the desired relationship +(fn arguments are [contravariant](../appendix/background.html#variance), so +we swap the left and right here): + +```text +&'!1 u32 <: &'static u32 +``` + +According to the basic subtyping rules for a reference, this will be +true if `'!1: 'static`. That is – if "some unknown region `!1`" lives +outlives `'static`. Now, this *might* be true – after all, `'!1` +could be `'static` – but we don't *know* that it's true. So this +should yield up an error (eventually). + +## What is a universe + +In the previous section, we introduced the idea of a placeholder +region, and we denoted it `!1`. We call this number `1` the **universe +index**. The idea of a "universe" is that it is a set of names that +are in scope within some type or at some point. Universes are formed +into a tree, where each child extends its parents with some new names. +So the **root universe** conceptually contains global names, such as +the the lifetime `'static` or the type `i32`. In the compiler, we also +put generic type parameters into this root universe (in this sense, +there is not just one root universe, but one per item). So consider +this function `bar`: + +```rust,ignore +struct Foo { } + +fn bar<'a, T>(t: &'a T) { + ... +} +``` + +Here, the root universe would consist of the lifetimes `'static` and +`'a`. In fact, although we're focused on lifetimes here, we can apply +the same concept to types, in which case the types `Foo` and `T` would +be in the root universe (along with other global types, like `i32`). +Basically, the root universe contains all the names that +[appear free](../appendix/background.html#free-vs-bound) in the body of `bar`. + +Now let's extend `bar` a bit by adding a variable `x`: + +```rust,ignore +fn bar<'a, T>(t: &'a T) { + let x: for<'b> fn(&'b u32) = ...; +} +``` + +Here, the name `'b` is not part of the root universe. Instead, when we +"enter" into this `for<'b>` (e.g., by replacing it with a placeholder), we will create +a child universe of the root, let's call it U1: + +```text +U0 (root universe) +│ +└─ U1 (child universe) +``` + +The idea is that this child universe U1 extends the root universe U0 +with a new name, which we are identifying by its universe number: +`!1`. + +Now let's extend `bar` a bit by adding one more variable, `y`: + +```rust,ignore +fn bar<'a, T>(t: &'a T) { + let x: for<'b> fn(&'b u32) = ...; + let y: for<'c> fn(&'b u32) = ...; +} +``` + +When we enter *this* type, we will again create a new universe, which +we'll call `U2`. Its parent will be the root universe, and U1 will be +its sibling: + +```text +U0 (root universe) +│ +├─ U1 (child universe) +│ +└─ U2 (child universe) +``` + +This implies that, while in U2, we can name things from U0 or U2, but +not U1. + +**Giving existential variables a universe.** Now that we have this +notion of universes, we can use it to extend our type-checker and +things to prevent illegal names from leaking out. The idea is that we +give each inference (existential) variable – whether it be a type or +a lifetime – a universe. That variable's value can then only +reference names visible from that universe. So for example is a +lifetime variable is created in U0, then it cannot be assigned a value +of `!1` or `!2`, because those names are not visible from the universe +U0. + +**Representing universes with just a counter.** You might be surprised +to see that the compiler doesn't keep track of a full tree of +universes. Instead, it just keeps a counter – and, to determine if +one universe can see another one, it just checks if the index is +greater. For example, U2 can see U0 because 2 >= 0. But U0 cannot see +U2, because 0 >= 2 is false. + +How can we get away with this? Doesn't this mean that we would allow +U2 to also see U1? The answer is that, yes, we would, **if that +question ever arose**. But because of the structure of our type +checker etc, there is no way for that to happen. In order for +something happening in the universe U1 to "communicate" with something +happening in U2, they would have to have a shared inference variable X +in common. And because everything in U1 is scoped to just U1 and its +children, that inference variable X would have to be in U0. And since +X is in U0, it cannot name anything from U1 (or U2). This is perhaps easiest +to see by using a kind of generic "logic" example: + +```text +exists { + forall { ... /* Y is in U1 ... */ } + forall { ... /* Z is in U2 ... */ } +} +``` + +Here, the only way for the two foralls to interact would be through X, +but neither Y nor Z are in scope when X is declared, so its value +cannot reference either of them. + +## Universes and placeholder region elements + +But where does that error come from? The way it happens is like this. +When we are constructing the region inference context, we can tell +from the type inference context how many placeholder variables exist +(the `InferCtxt` has an internal counter). For each of those, we +create a corresponding universal region variable `!n` and a "region +element" `placeholder(n)`. This corresponds to "some unknown set of other +elements". The value of `!n` is `{placeholder(n)}`. + +At the same time, we also give each existential variable a +**universe** (also taken from the `InferCtxt`). This universe +determines which placeholder elements may appear in its value: For +example, a variable in universe U3 may name `placeholder(1)`, `placeholder(2)`, and +`placeholder(3)`, but not `placeholder(4)`. Note that the universe of an inference +variable controls what region elements **can** appear in its value; it +does not say region elements **will** appear. + +## Placeholders and outlives constraints + +In the region inference engine, outlives constraints have the form: + +```text +V1: V2 @ P +``` + +where `V1` and `V2` are region indices, and hence map to some region +variable (which may be universally or existentially quantified). The +`P` here is a "point" in the control-flow graph; it's not important +for this section. This variable will have a universe, so let's call +those universes `U(V1)` and `U(V2)` respectively. (Actually, the only +one we are going to care about is `U(V1)`.) + +When we encounter this constraint, the ordinary procedure is to start +a DFS from `P`. We keep walking so long as the nodes we are walking +are present in `value(V2)` and we add those nodes to `value(V1)`. If +we reach a return point, we add in any `end(X)` elements. That part +remains unchanged. + +But then *after that* we want to iterate over the placeholder `placeholder(x)` +elements in V2 (each of those must be visible to `U(V2)`, but we +should be able to just assume that is true, we don't have to check +it). We have to ensure that `value(V1)` outlives each of those +placeholder elements. + +Now there are two ways that could happen. First, if `U(V1)` can see +the universe `x` (i.e., `x <= U(V1)`), then we can just add `placeholder(x)` +to `value(V1)` and be done. But if not, then we have to approximate: +we may not know what set of elements `placeholder(x)` represents, but we +should be able to compute some sort of **upper bound** B for it – +some region B that outlives `placeholder(x)`. For now, we'll just use +`'static` for that (since it outlives everything) – in the future, we +can sometimes be smarter here (and in fact we have code for doing this +already in other contexts). Moreover, since `'static` is in the root +universe U0, we know that all variables can see it – so basically if +we find that `value(V2)` contains `placeholder(x)` for some universe `x` +that `V1` can't see, then we force `V1` to `'static`. + +## Extending the "universal regions" check + +After all constraints have been propagated, the NLL region inference +has one final check, where it goes over the values that wound up being +computed for each universal region and checks that they did not get +'too large'. In our case, we will go through each placeholder region +and check that it contains *only* the `placeholder(u)` element it is known to +outlive. (Later, we might be able to know that there are relationships +between two placeholder regions and take those into account, as we do +for universal regions from the fn signature.) + +Put another way, the "universal regions" check can be considered to be +checking constraints like: + +```text +{placeholder(1)}: V1 +``` + +where `{placeholder(1)}` is like a constant set, and V1 is the variable we +made to represent the `!1` region. + +## Back to our example + +OK, so far so good. Now let's walk through what would happen with our +first example: + +```text +fn(&'static u32) <: fn(&'!1 u32) @ P // this point P is not imp't here +``` + +The region inference engine will create a region element domain like this: + +```text +{ CFG; end('static); placeholder(1) } + --- ------------ ------- from the universe `!1` + | 'static is always in scope + all points in the CFG; not especially relevant here +``` + +It will always create two universal variables, one representing +`'static` and one representing `'!1`. Let's call them Vs and V1. They +will have initial values like so: + +```text +Vs = { CFG; end('static) } // it is in U0, so can't name anything else +V1 = { placeholder(1) } +``` + +From the subtyping constraint above, we would have an outlives constraint like + +```text +'!1: 'static @ P +``` + +To process this, we would grow the value of V1 to include all of Vs: + +```text +Vs = { CFG; end('static) } +V1 = { CFG; end('static), placeholder(1) } +``` + +At that point, constraint propagation is complete, because all the +outlives relationships are satisfied. Then we would go to the "check +universal regions" portion of the code, which would test that no +universal region grew too large. + +In this case, `V1` *did* grow too large – it is not known to outlive +`end('static)`, nor any of the CFG – so we would report an error. + +## Another example + +What about this subtyping relationship? + +```text +for<'a> fn(&'a u32, &'a u32) + <: +for<'b, 'c> fn(&'b u32, &'c u32) +``` + +Here we would replace the bound region in the supertype with a placeholder, as before, yielding: + +```text +for<'a> fn(&'a u32, &'a u32) + <: +fn(&'!1 u32, &'!2 u32) +``` + +then we instantiate the variable on the left-hand side with an +existential in universe U2, yielding the following (`?n` is a notation +for an existential variable): + +```text +fn(&'?3 u32, &'?3 u32) + <: +fn(&'!1 u32, &'!2 u32) +``` + +Then we break this down further: + +```text +&'!1 u32 <: &'?3 u32 +&'!2 u32 <: &'?3 u32 +``` + +and even further, yield up our region constraints: + +```text +'!1: '?3 +'!2: '?3 +``` + +Note that, in this case, both `'!1` and `'!2` have to outlive the +variable `'?3`, but the variable `'?3` is not forced to outlive +anything else. Therefore, it simply starts and ends as the empty set +of elements, and hence the type-check succeeds here. + +(This should surprise you a little. It surprised me when I first realized it. +We are saying that if we are a fn that **needs both of its arguments to have +the same region**, we can accept being called with **arguments with two +distinct regions**. That seems intuitively unsound. But in fact, it's fine, as +I tried to explain in [this issue][ohdeargoditsallbroken] on the Rust issue +tracker long ago. The reason is that even if we get called with arguments of +two distinct lifetimes, those two lifetimes have some intersection (the call +itself), and that intersection can be our value of `'a` that we use as the +common lifetime of our arguments. -nmatsakis) + +[ohdeargoditsallbroken]: https://github.com/rust-lang/rust/issues/32330#issuecomment-202536977 + +## Final example + +Let's look at one last example. We'll extend the previous one to have +a return type: + +```text +for<'a> fn(&'a u32, &'a u32) -> &'a u32 + <: +for<'b, 'c> fn(&'b u32, &'c u32) -> &'b u32 +``` + +Despite seeming very similar to the previous example, this case is going to get +an error. That's good: the problem is that we've gone from a fn that promises +to return one of its two arguments, to a fn that is promising to return the +first one. That is unsound. Let's see how it plays out. + +First, we replace the bound region in the supertype with a placeholder: + +```text +for<'a> fn(&'a u32, &'a u32) -> &'a u32 + <: +fn(&'!1 u32, &'!2 u32) -> &'!1 u32 +``` + +Then we instantiate the subtype with existentials (in U2): + +```text +fn(&'?3 u32, &'?3 u32) -> &'?3 u32 + <: +fn(&'!1 u32, &'!2 u32) -> &'!1 u32 +``` + +And now we create the subtyping relationships: + +```text +&'!1 u32 <: &'?3 u32 // arg 1 +&'!2 u32 <: &'?3 u32 // arg 2 +&'?3 u32 <: &'!1 u32 // return type +``` + +And finally the outlives relationships. Here, let V1, V2, and V3 be the +variables we assign to `!1`, `!2`, and `?3` respectively: + +```text +V1: V3 +V2: V3 +V3: V1 +``` + +Those variables will have these initial values: + +```text +V1 in U1 = {placeholder(1)} +V2 in U2 = {placeholder(2)} +V3 in U2 = {} +``` + +Now because of the `V3: V1` constraint, we have to add `placeholder(1)` into `V3` (and +indeed it is visible from `V3`), so we get: + +```text +V3 in U2 = {placeholder(1)} +``` + +then we have this constraint `V2: V3`, so we wind up having to enlarge +`V2` to include `placeholder(1)` (which it can also see): + +```text +V2 in U2 = {placeholder(1), placeholder(2)} +``` + +Now constraint propagation is done, but when we check the outlives +relationships, we find that `V2` includes this new element `placeholder(1)`, +so we report an error. From 4615a9a1f7079d9ec6ce788a46967fb8666bfce1 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 18 Jun 2019 10:55:07 -0400 Subject: [PATCH 586/648] start filling out the constraint propagation chapter in more detail --- .../constraint_propagation.md | 145 +++++++++++++++++- 1 file changed, 144 insertions(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index 2e69629b1..ce01764c4 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -1,3 +1,146 @@ # Constraint propagation -The main work of the region inference is **constraint propagation**. +The main work of the region inference is **constraint +propagation**. This means processing the set of constraints to compute +the final values for all the region variables. + +## Kinds of constraints + +Each kind of constraint is handled somewhat differently by the region inferencer. + +### Liveness constraints + +A **liveness constraint** arises when some variable whose type +includes a region R is live at some point P. This simply means that +the value of R must include the point P. Liveness constraints are +computed by the MIR type checker. + +We represent them by keeping a (sparse) bitset for each region +variable, which is the field [`liveness_constraints`], of type +[`LivenessValues`] + +[`liveness_constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#structfield.liveness_constraints +[`LivenessValues`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/values/struct.LivenessValues.html + +### Outlives constraints + +An outlives constraint `'a: 'b` indicates that the value of `'a` must +be a **superset** of the value of `'b`. On creation, we are given a +set of outlives constraints in the form of a +[`ConstraintSet`]. However, to work more efficiently with outlives +constraints, they are [converted into the form of a graph][graph-fn], +where the nodes of the graph are region variables (`'a`, `'b`) and +each constraint `'a: 'b` induces an edge `'a -> 'b`. This conversion +happens in the [`RegionInferenceContext::new`] function that creates +the inference context. + +[`ConstraintSet`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/constraints/struct.ConstraintSet.html +[graph-fn]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/constraints/struct.ConstraintSet.html#method.graph +[`RegionInferenceContext::new`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.new + +### Member constraints + +A member constraint `'m member of ['c_1..'c_N]` expresses that the +region `'m` must be *equal* to some **choice regions** `'c_i` (for +some `i`). These constraints cannot be expressed by users, but they arise +from `impl Trait` due to its lifetime capture rules. Consinder a function +such as the following: + +```rust +fn make(a: &'a u32, b: &'b u32) -> impl Trait<'a, 'b> { .. } +``` + +Here, the true return type (often called the "hidden type") is only +permitted to capture the lifeimes `'a` or `'b`. You can kind of see +this more clearly by desugaring that `impl Trait` return type into its +more explicit form: + +```rust +type MakeReturn<'x, 'y> = impl Trait<'x, 'y>; +fn make(a: &'a u32, b: &'b u32) -> MakeReturn<'a, 'b> { .. } +``` + +Here, the idea is that the hidden type must be some type that could +have been written in place of the `impl Trait<'x, 'y>` -- but clearly +such a type can only reference the regions `'x` or `'y` (or +`'static`!), as those are the only names in scope. This limitation is +then translated into a restriction to only access `'a` or `'b` because +we are returning `MakeReturn<'a, 'b>`, where `'x` and `'y` have been +replaced with `'a` and `'b` respectively. + +## SCCs in the outlives constraint graph + +The most common sort of constraint in practice are outlives +constraints like `'a: 'b`. Such a cosntraint means that `'a` is a +superset of `'b`. So what happens if we have two regions `'a` and `'b` +that mutually outlive one another, like so? + +``` +'a: 'b +'b: 'a +``` + +In this case, we can conclude that `'a` and `'b` must be equal +sets. In fact, it doesn't have to be just two regions. We could create +an extended "chain" of outlives constraints: + +``` +'a: 'b +'b: 'c +'c: 'd +'d: 'a +``` + +Here, we know that `'a..'d` are all equal to one another. + +As mentioned above, an outlives constraint like `'a: 'b` can be viewed +as an edge in a graph `'a -> 'b`. Cycles in this graph indicate regions +that mutually outlive one another and hence must be equal. + +Therefore, one of the first things that we do in propagating region +values is to compute the **strongly connected components** (SCCs) in +the constraint graph. The result is stored in the [`constraint_sccs`] +field. You can then easily find the SCC that a region `r` is a part of +by invoking `constraint_sccs.scc(r)`. + +Working in terms of SCCs allows us to be more efficient: if we have a +set of regions `'a...'d` that are part of a single SCC, we don't have +to compute/store their values separarely. We can just store one value +**for the SCC**, since they must all be equal. + +If you look over the region inference code, you will see that a number +of fields are defined in terms of SCCs. For example, the +[`scc_values`] field stores the values of each SCC. To get the value +of a specific region `'a` then, we first figure out the SCC that the +region is a part of, and then find the value of that SCC. + +[`constraint_sccs`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#structfield.constraint_sccs +[`scc_values`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#structfield.scc_values + +When we compute SCCs, we not only figure out which regions are a +member of each SCC, we also figure out the edges between them. So for example +consider this set of outlives constraints: + +``` +'a: 'b +'b: 'a + +'a: 'c + +'c: 'd +'d: 'c +``` + +Here we have two SCCs: S0 contains `'a` and `'b`, and S1 contains `'c` +and `'d`. But these SCCs are not independent: because `'a: 'c`, that +means that `S0: S1` as well. That is -- the value of `S0` must be a +superset of the value of `S1`. One crucial thing is that this graph of +SCCs is always a DAG -- that is, it never has cycles. This is because +all the cycles have been removed to form the SCCs themselves. + +## How constraint propagation works + +The main work of constraint propagation is done in the +`propagation_constraints` function. + +[`propagate_constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.propagate_constraints From 472f4e83670fea495daa8cbc64e070cae70d3cc2 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Thu, 20 Jun 2019 09:43:48 -0400 Subject: [PATCH 587/648] describe region inference and member constraints in some detail --- src/SUMMARY.md | 2 + src/borrow_check/region_inference.md | 2 + .../constraint_propagation.md | 212 ++++++++++++------ .../region_inference/lifetime_parameters.md | 125 +++++++++++ .../region_inference/member_constraints.md | 193 ++++++++++++++++ 5 files changed, 466 insertions(+), 68 deletions(-) create mode 100644 src/borrow_check/region_inference/lifetime_parameters.md create mode 100644 src/borrow_check/region_inference/member_constraints.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 7e9a9ee54..69e46b796 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -77,6 +77,8 @@ - [MIR type checker](./borrow_check/type_check.md) - [Region inference](./borrow_check/region_inference.md) - [Constraint propagation](./borrow_check/region_inference/constraint_propagation.md) + - [Lifetime parameters](./borrow_check/region_inference/lifetime_parameters.md) + - [Member constraints](./borrow_check/region_inference/member_constraints.md) - [Placeholders and universes](./borrow_check/region_inference/placeholders_and_universes.md) - [Closure constraints](./borrow_check/region_inference/closure_constraints.md) - [Errror reporting](./borrow_check/region_inference/error_reporting.md) diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index 5279d210b..e9f0e35e3 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -71,6 +71,8 @@ TODO: write about _how_ these regions are computed. [`UniversalRegions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/universal_regions/struct.UniversalRegions.html + + ## Region variables The value of a region can be thought of as a **set**. This set contains all diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index ce01764c4..e2c36062f 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -1,88 +1,121 @@ # Constraint propagation -The main work of the region inference is **constraint -propagation**. This means processing the set of constraints to compute -the final values for all the region variables. +The main work of the region inference is **constraint propagation**, +which is done in the [`propagate_constraints`] function. There are +three sorts of constraints that are used in NLL, and we'll explain how +`propagate_constraints` works by "layering" those sorts of constraints +on one at a time (each of them is fairly independent from the others): -## Kinds of constraints +- liveness constraints (`R live at E`), which arise from liveness; +- outlives constraints (`R1: R2`), which arise from subtyping; +- [member constraints][m_c] (`member R_m of [R_c...]`), which arise from impl Trait. -Each kind of constraint is handled somewhat differently by the region inferencer. +[`propagate_constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.propagate_constraints +[m_c]: ./member_constraints.html + +In this chapter, we'll explain the "heart" of constraint propagation, +covering both liveness and outlives constraints. + +## Notation and high-level concepts + +Conceptually, region inference is a "fixed-point" computation. It is +given some set of constraints `{C}` and it computes a set of values +`Values: R -> {E}` that maps each region `R` to a set of elements +`{E}` (see [here][riv] for more notes on region elements): + +- Initially, each region is mapped to an empty set, so `Values(R) = + {}` for all regions `R`. +- Next, we process the constraints repeatedly until a fixed-point is reached: + - For each constraint C: + - Update `Values` as needed to satisfy the constraint -### Liveness constraints +[riv]: ../region-inference.html#region-variables + +As a simple example, if we have a liveness constraint `R live at E`, +then we can apply `Values(R) = Values(R) union {E}` to make the +constraint be satisfied. Similarly, if we have an outlives constraints +`R1: R2`, we can apply `Values(R1) = Values(R1) union Values(R2)`. +(Member constraints are more complex and we discuss them below.) + +In practice, however, we are a bit more clever. Instead of applying +the constraints in a loop, we can analyze the constraints and figure +out the correct order to apply them, so that we only have to apply +each constraint once in order to find the final result. + +Similarly, in the implementation, the `Values` set is stored in the +`scc_values` field, but they are indexed not by a *region* but by a +*strongly connected component* (SCC). SCCs are an optimization that +avoids a lot of redundant storage and computation. They are explained +in the section on outlives constraints. + +## Liveness constraints A **liveness constraint** arises when some variable whose type includes a region R is live at some point P. This simply means that the value of R must include the point P. Liveness constraints are computed by the MIR type checker. -We represent them by keeping a (sparse) bitset for each region -variable, which is the field [`liveness_constraints`], of type -[`LivenessValues`] +A liveness constraint `R live at E` is satisfied if `E` is a member of +`Values(R)`. So to "apply" such a constraint to `Values`, we just have +to compute `Values(R) = Values(R) union {E}`. + +The liveness values are computed in the type-check and passes to the +region inference upon creation in the `liveness_constraints` argument. +These are not represented as individual constraints like `R live at E` +though; instead, we store a (sparse) bitset per region variable (of +type [`LivenessValues`]). This way we only need a single bit for each +liveness constraint. [`liveness_constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#structfield.liveness_constraints [`LivenessValues`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/values/struct.LivenessValues.html -### Outlives constraints +One thing that is worth mentioning: All lifetime parameters are always +considered to be live over the entire function body. This is because +they correspond to some portion of the *caller's* execution, and that +execution clearly includes the time spent in this function, since the +caller is waiting for us to return. -An outlives constraint `'a: 'b` indicates that the value of `'a` must -be a **superset** of the value of `'b`. On creation, we are given a -set of outlives constraints in the form of a -[`ConstraintSet`]. However, to work more efficiently with outlives -constraints, they are [converted into the form of a graph][graph-fn], -where the nodes of the graph are region variables (`'a`, `'b`) and -each constraint `'a: 'b` induces an edge `'a -> 'b`. This conversion -happens in the [`RegionInferenceContext::new`] function that creates -the inference context. - -[`ConstraintSet`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/constraints/struct.ConstraintSet.html -[graph-fn]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/constraints/struct.ConstraintSet.html#method.graph -[`RegionInferenceContext::new`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.new +## Outlives constraints -### Member constraints +An outlives constraint `'a: 'b` indicates that the value of `'a` must +be a **superset** of the value of `'b`. That is, an outlives +constraint `R1: R2` is satisfied if `Values(R1)` is a superset of +`Values(R2)`. So to "apply" such a constraint to `Values`, we just +have to compute `Values(R1) = Values(R1) union Values(R2)`. -A member constraint `'m member of ['c_1..'c_N]` expresses that the -region `'m` must be *equal* to some **choice regions** `'c_i` (for -some `i`). These constraints cannot be expressed by users, but they arise -from `impl Trait` due to its lifetime capture rules. Consinder a function -such as the following: +One observation that follows from this is that if you have `R1: R2` +and `R2: R1`, then `R1 = R2` must be true. Similarly, if you have: -```rust -fn make(a: &'a u32, b: &'b u32) -> impl Trait<'a, 'b> { .. } ``` - -Here, the true return type (often called the "hidden type") is only -permitted to capture the lifeimes `'a` or `'b`. You can kind of see -this more clearly by desugaring that `impl Trait` return type into its -more explicit form: - -```rust -type MakeReturn<'x, 'y> = impl Trait<'x, 'y>; -fn make(a: &'a u32, b: &'b u32) -> MakeReturn<'a, 'b> { .. } +R1: R2 +R2: R3 +R3: R4 +R4: R1 ``` -Here, the idea is that the hidden type must be some type that could -have been written in place of the `impl Trait<'x, 'y>` -- but clearly -such a type can only reference the regions `'x` or `'y` (or -`'static`!), as those are the only names in scope. This limitation is -then translated into a restriction to only access `'a` or `'b` because -we are returning `MakeReturn<'a, 'b>`, where `'x` and `'y` have been -replaced with `'a` and `'b` respectively. +then `R1 = R2 = R3 = R4` follows. We take advantage of this to make things +much faster, as described shortly. -## SCCs in the outlives constraint graph +In the code, the set of outlives constraints is given to the region +inference context on creation in a parameter of type +[`ConstraintSet`]. The constraint set is basically just a list of `'a: +'b` constraints. -The most common sort of constraint in practice are outlives -constraints like `'a: 'b`. Such a cosntraint means that `'a` is a -superset of `'b`. So what happens if we have two regions `'a` and `'b` -that mutually outlive one another, like so? +### The outlives constraint graph and SCCs -``` -'a: 'b -'b: 'a -``` +In order to work more efficiently with outlives constraints, they are +[converted into the form of a graph][graph-fn], where the nodes of the +graph are region variables (`'a`, `'b`) and each constraint `'a: 'b` +induces an edge `'a -> 'b`. This conversion happens in the +[`RegionInferenceContext::new`] function that creates the inference +context. + +[`ConstraintSet`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/constraints/struct.ConstraintSet.html +[graph-fn]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/constraints/struct.ConstraintSet.html#method.graph +[`RegionInferenceContext::new`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.new -In this case, we can conclude that `'a` and `'b` must be equal -sets. In fact, it doesn't have to be just two regions. We could create -an extended "chain" of outlives constraints: +When using a graph representation, we can detect regions that must be equal +by looking for cycles. That is, if you have a constraint like ``` 'a: 'b @@ -91,11 +124,8 @@ an extended "chain" of outlives constraints: 'd: 'a ``` -Here, we know that `'a..'d` are all equal to one another. - -As mentioned above, an outlives constraint like `'a: 'b` can be viewed -as an edge in a graph `'a -> 'b`. Cycles in this graph indicate regions -that mutually outlive one another and hence must be equal. +then this will correspond to a cycle in the graph containing the +elements `'a...'d`. Therefore, one of the first things that we do in propagating region values is to compute the **strongly connected components** (SCCs) in @@ -138,9 +168,55 @@ superset of the value of `S1`. One crucial thing is that this graph of SCCs is always a DAG -- that is, it never has cycles. This is because all the cycles have been removed to form the SCCs themselves. -## How constraint propagation works +### Applying liveness constraints to SCCs + +The liveness constraints that come in from the type-checker are +expressed in terms of regions -- that is, we have a map like +`Liveness: R -> {E}`. But we want our final result to be expressed +in terms of SCCs -- we can integrate these liveness constraints very +easily just by taking the union: + +``` +for each region R: + let S by the SCC that contains R + Values(S) = Values(S) union Liveness(R) +``` + +In the region inferencer, this step is done in [`RegionInferenceContext::new`]. + +### Applying outlives constraints + +Once we have computed the DAG of SCCs, we use that to structure out +entire computation. If we have an edge `S1 -> S2` between two SCCs, +that means that `Values(S1) >= Values(S2)` must hold. So, to compute +the value of `S1`, we first compute the values of each successor `S2`. +Then we simply union all of those values together. To use a +quasi-iterator-like notation: + +``` +Values(S1) = + s1.successors() + .map(|s2| Values(s2)) + .union() +``` + +In the code, this work starts in the [`propagate_constraints`] +function, which iterates over all the SCCs. For each SCC S1, we +compute its value by first computing the value of its +successors. Since SCCs form a DAG, we don't have to be conecrned about +cycles, though we do need to keep a set around to track whether we +have already processed a given SCC or not. For each successor S2, once +we have computed S2's value, we can union those elements into the +value for S1. (Although we have to be careful in this process to +properly handle [higher-ranked +placeholders](./placeholders_and_universes.html). Note that the value +for S1 already contains the liveness constraints, since they were +added in [`RegionInferenceContext::new`]. + +Once that process is done, we now have the "minimal value" for S1, +taking into account all of the liveness and outlives +constraints. However, in order to complete the process, we must also +consider [member constraints][m_c], which are described in [a later +section][m_c]. -The main work of constraint propagation is done in the -`propagation_constraints` function. -[`propagate_constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.propagate_constraints diff --git a/src/borrow_check/region_inference/lifetime_parameters.md b/src/borrow_check/region_inference/lifetime_parameters.md new file mode 100644 index 000000000..10759ee56 --- /dev/null +++ b/src/borrow_check/region_inference/lifetime_parameters.md @@ -0,0 +1,125 @@ +# Universal regions + +"Universal regions" is the name that the code uses to refer to "named +lifetimes" -- e.g., lifetime parameters and `'static`. The name +derives from the fact that such lifetimes are "universally quantified" +(i.e., we must make sure the code is true for all values of those +lifetimes). It is worth spending a bit of discussing how lifetime +parameters are handled during region inference. Consider this example: + +```rust +fn foo<'a, 'b>(x: &'a u32, y: &'b u32) -> &'b u32 { + x +} +``` + +This example is intended not to compile, because we are returning `x`, +which has type `&'a u32`, but our signature promises that we will +return a `&'b u32` value. But how are lifetimes like `'a` and `'b +integrated into region inference, and how this error wind up being +detected? + +## Universal regions and their relationships to one another + +Early on in region inference, one of the first things we do is to +construct a [`UniversalRegions`] struct. This struct tracks the +various universal regions in scope on a particular function. We also +create a [`UniversalRegionRelations`] struct, which tracks their +relationships to one another. So if you have e.g. `where 'a: 'b`, then +the [`UniversalRegionRelations`] struct would track that `'a: 'b` is +known to hold (which could be tested with the [`outlives`] function. + +[`UniversalRegions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/universal_regions/struct.UniversalRegions.html +[`UniversalRegionsRelations`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/type_check/free_region_relations/struct.UniversalRegionRelations.html +[`outlives`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/type_check/free_region_relations/struct.UniversalRegionRelations.html#method.outlives + +## Everything is a region variable + +One important aspect of how NLL region inference works is that **all +lifetimes** are represented as numbered variables. This means that the +only variant of [`ty::RegionKind`] that we use is the [`ReVar`] +variant. These region variables are broken into two major categories, +based on their index: + +[`ty::RegionKind`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/enum.RegionKind.html +[`ReVar`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/enum.RegionKind.html#variant.ReVar + +- 0..N: universal regions -- the ones we are discussing here. In this + case, the code must be correct with respect to any value of those + variables that meets the declared relationships. +- N..M: existential regions -- inference variables where the region + inferencer is tasked with finding *some* suitable value. + +In fact, the universal regions can be further subdivided based on +where they were brought into scope (see the [`RegionClassification`] +type). These subdivions are not important for the topics discussed +here, but become important when we consider [closure constraint +propagation](./closure_constraints.html), so we discuss them there. + +[`RegionClassification`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/universal_regions/enum.RegionClassification.html#variant.Local + +## Universal lifetimes as the elements of a region's value + +As noted previously, the value that we infer for each region is a set +`{E}`. The elements of this set can be points in the control-flow +graph, but they can also be an element `end('a)` corresponding to each +universal lifetime `'a`. If the value for some region `R0` includes +`end('a`), then this implies that R0 must extend until the end of `'a` +in the caller. + +## The "value" of a universal region + +During region inference, we compute a value for each universal region +in the same way as we compute values for other regions. This value +represents, effectively, the **lower bound** on that universal region +-- the things that it must outlive. We now describe how we use this +value to check for errors. + +## Liveness and universal regions + +All universal regions have an initial liveness constraint that +includes the entire function body. This is because lifetime parameters +are defined in the caller and must include the entirety of the +function call that invokes this particular function. In addition, each +universal region `'a` includes itself (that is, `end('a)`) in its +liveness constraint (i.e., `'a` must extend until the end of +itself). In the code, these liveness constraints are setup in +[`init_free_and_bound_regions`]. + +[`init_free_and_bound_regions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.init_free_and_bound_regions + +## Propagating outlives constraints for universal regions + +So, consider the first example of this section: + +```rust +fn foo<'a, 'b>(x: &'a u32, y: &'b u32) -> &'b u32 { + x +} +``` + +Here, returning `x` requires that `&'a u32 <: &'b u32`, which gives +rise to an outlives constraint `'a: 'b`. Combined with our default liveness +constraints we get: + +``` +'a live at {B, end('a)} // B represents the "bunction body" +'b live at {B, end('b)} +'a: 'b +``` + +When we process the `'a: 'b` constraint, therefore, we will add +`end('b)` into the value for `'a`, resulting in a final value of `{B, +end('a), end('b)}`. + +## Detecting errors + +Once we have finished constraint propagation, we then enforce a +constraint that if some universal region `'a` includes an element +`end('b)`, then `'a: 'b` must be declared in the function's bounds. If +not, as in our example, that is an error. This check is done in the +[`check_universal_regions`] function, which simply iterates over all +universal regions, inspects their final value, and tests against the +declared [`UniversalRegionRelations`]. + +[`check_universal_regions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.check_universal_regions diff --git a/src/borrow_check/region_inference/member_constraints.md b/src/borrow_check/region_inference/member_constraints.md new file mode 100644 index 000000000..4d7312644 --- /dev/null +++ b/src/borrow_check/region_inference/member_constraints.md @@ -0,0 +1,193 @@ +# Member constraints + +A member constraint `'m member of ['c_1..'c_N]` expresses that the +region `'m` must be *equal* to some **choice regions** `'c_i` (for +some `i`). These constraints cannot be expressed by users, but they +arise from `impl Trait` due to its lifetime capture rules. Consinder a +function such as the following: + +```rust +fn make(a: &'a u32, b: &'b u32) -> impl Trait<'a, 'b> { .. } +``` + +Here, the true return type (often called the "hidden type") is only +permitted to capture the lifeimes `'a` or `'b`. You can kind of see +this more clearly by desugaring that `impl Trait` return type into its +more explicit form: + +```rust +type MakeReturn<'x, 'y> = impl Trait<'x, 'y>; +fn make(a: &'a u32, b: &'b u32) -> MakeReturn<'a, 'b> { .. } +``` + +Here, the idea is that the hidden type must be some type that could +have been written in place of the `impl Trait<'x, 'y>` -- but clearly +such a type can only reference the regions `'x` or `'y` (or +`'static`!), as those are the only names in scope. This limitation is +then translated into a restriction to only access `'a` or `'b` because +we are returning `MakeReturn<'a, 'b>`, where `'x` and `'y` have been +replaced with `'a` and `'b` respectively. + +## Detailed example + +To help us explain member constraints in more detail, let's spell out +the `make` example in a bit more detail. First off, let's assume that +you have some dummy trait: + +```rust +trait Trait<'a, 'b> { } +impl Trait<'_, '_> for T { } +``` + +and this is the `make` function (in desugared form): + +```rust +type MakeReturn<'x, 'y> = impl Trait<'x, 'y>; +fn make(a: &'a u32, b: &'b u32) -> MakeReturn<'a, 'b> { + (a, b) +} +``` + +What happens in this case is that the return type will be `(&'0 u32, &'1 u32)`, +where `'0` and `'1` are fresh region variables. We will have the following +region constraints: + +``` +'0 live at {L} +'1 live at {L} +'a: '0 +'b: '1 +'0 member of ['a, 'b, 'static] +'1 member of ['a, 'b, 'static] +``` + +Here the "liveness set" `{L}` corresponds to that subset of the body +where `'0` and `'1` are live -- basically the point from where the +return tuple is constructed to where it is returned (in fact, `'0` and +`'1` might have slightly different liveness sets, but that's not very +interesting to the point we are illustrating here). + +The `'a: '0` and `'b: '1` constraints arise from subtyping. When we +construct the `(a, b)` value, it will be assigned type `(&'0 u32, &'1 +u32)` -- the region variables reflect that the lifetime of this +references could be made smaller. For this value to be created from +`a` and `b`, however, we do require that: + +``` +(&'a u32, &'b u32) <: (&'0 u32, &'1 u32) +``` + +which means in turn that `&'a u32 <: &'0 u32` and hence that `'a: '0` +(and similarly that `&'b u32 <: &'1 u32`, `'b: '1`). + +Note that if we ignore member constraints, the value of `'0` would be +inferred to some subset of the function body (from the liveness +constraints, which we did not write explicitly). It would never become +`'a`, because there is no need for it too -- we have a constraint that +`'a: '0`, but that just puts a "cap" on how *large* `'0` can grow to +become. Since we compute the *minimal* value that we can, we are happy +to leave `'0` as being just equal to the liveness set. This is where +member constraints come in. + +## Choices are always lifetime parameters + +At present, the "choice" regions from a member constraint are always +lifetime parameters from the current function. This falls out from the +placement of impl Trait, though in the future it may not be the case. +We take some advantage of this fact, as it simplifies the current +code. In particular, we don't have to consider a case like `'0 member +of ['1, 'static]`, in which the value of both `'0` and `'1` are being +inferred and hence changing. See [rust-lang/rust#61773] for more +information. + +[#61773]: https://github.com/rust-lang/rust/issues/61773 + +## Applying member constraints + +Member constraints are a bit more complex than other forms of +constraints. This is because they have a "or" quality to them -- that +is, they describe multiple choices that we must select from. E.g., in +our example constraint `'0 member of ['a, 'b, 'static]`, it might be +that `'0` is equal to `'a`, `'b`, *or* `'static`. How can we pick the +correct one? What we currently do is to look for a *minimal choice* +-- if we find one, then we will grow `'0` to be equal to that minimal +choice. To find that minimal choice, we take two factors into +consideration: lower and upper bounds. + +### Lower bounds + +The *lower bounds* are those lifetimes that `'0` *must outlive* -- +i.e., that `'0` must be larger than. In fact, when it comes time to +apply member constraints, we've already *computed* the lower bounds of +`'0` because we computed its minimal value (or at least, the lower +bounds considering everything but member constraints). + +Let `LB` be the current value of `'0`. We know then that `'0: LB` must +hold, whatever the final value of `'0` is. Therefore, we can rule out +any choice where `'choice` where `'choice: LB` does not hold. + +Unfortunately, in our example, this is not very helpful. The lower +bound for `'0` will just be the liveness set `{L}`, and we know that +all the lifetime parameters outlive that set. So we are left with the +same set of choices here. (But in other examples, particularly those +with different variance, lower bound constraints may be relevant.) + +### Upper bounds + +The *upper bounds* are those lifetimes that *must outlive* `'0` -- +i.e., that `'0` must be *smaller* then. In our example, this would be +`'a`, because we have the constraint that `'a: '0`. In more complex +examples, the chain may be more indirect. + +We can use upper bounds to rule out members in a very similar way to +lower lower bounds. If UB is some upper bound, then we know that `UB: +'0` must hold, so we can rule out any choice `'choice` where `UB: +'choice` does not hold. + +In our example, we would be able to reduce our choice set from `['a, +'b, 'static]` to just `['a]`. This is because `'0` has an upper bound +of `'a`, and neither `'a: 'b` nor `'a: 'static` is known to hold. + +(For notes on how we collect upper bounds in the implementation, see +[the section below](#collecting).) + +### Minimal choice + +After applying lower and upper bounds, we can still sometimes have +multiple possibilities. For example, imagine a variant of our example +using types with the opposite variance. In that case, we would have +the constraint `'0: 'a` instead of `'a: '0`. Hence the current value +of `'0` would be `{L, 'a}`. Using this as a lower bound, we would be +able to narrow down the member choices to `['a, 'static]` because `'b: +'a` is not known to hold (but `'a: 'a` and `'static: 'a` do hold). We +would not have any upper bounds, so that would be our final set of choices. + +In that case, we apply the **minimal choice** rule -- basically, if +one of our choices if smaller than the others, we can use that. In +this case, we would opt for `'a` (and not `'static`). + +This choice is consistent with the general 'flow' of region +propagation, which always aims to compute a minimal value for the +region being inferred. However, it is somewhat arbitrary. + + + +### Collecting upper bounds in the implementation + +In practice, computing upper bounds is a bit inconvenient, because our +data structures are setup for the opposite. What we do is to compute +the **reverse SCC graph** (we do this lazilly and cache the result) -- +that is, a graph where `'a: 'b` induces an edge `SCC('b) -> +SCC('a)`. Like the normal SCC graph, this is a DAG. We can then do a +depth-first search starting from `SCC('0)` in this graph. This will +take us to all the SCCs that must outlive `'0`. + +One wrinkle is that, as we walk the "upper bound" SCCs, their values +will not yet have been fully computed. However, we **have** already +applied their liveness constraints, so we have some information about +their value. In particular, for any regions representing lifetime +parameters, their value will contain themselves (i.e., the initial +value for `'a` includes `'a` and the value for `'b` contains `'b`). So +we can collect all of the lifetime parameters that are reachable, +which is precisely what we are interested in. + From bd347df7514d8f0d5bf9631a4230c78940c4eede Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Thu, 20 Jun 2019 09:51:11 -0400 Subject: [PATCH 588/648] adjust overview slightly --- src/borrow_check/region_inference.md | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index e9f0e35e3..46caab567 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -24,20 +24,19 @@ The MIR-based region analysis consists of two major functions: - [`compute_regions`], invoked second: this is given as argument the results of move analysis. It has the job of computing values for all the inference variables that `replace_regions_in_mir` introduced. - - To do that, it first runs the [MIR type checker]. This - is basically a normal type-checker but specialized to MIR, which - is much simpler than full Rust, of course. Running the MIR type - checker will however create **outlives constraints** between - region variables (e.g., that one variable must outlive another - one) to reflect the subtyping relationships that arise. - - It also adds **liveness constraints** that arise from where variables - are used. - - After this, we create a [`RegionInferenceContext`] with the constraints we - have computed and the inference variables we introduced and use the - [`solve`] method to infer values for all region inference varaibles. + - To do that, it first runs the [MIR type checker]. This is + basically a normal type-checker but specialized to MIR, which is + much simpler than full Rust, of course. Running the MIR type + checker will however create various [constraints][cp] between region + variables, indicating their potential values and relationships to + one another. + - After this, we perform [constraint propagation][cp] by creating a + [`RegionInferenceContext`] and invoking its [`solve`] + method. - The [NLL RFC] also includes fairly thorough (and hopefully readable) coverage. +[cp]: ./region_inference/constraint_propagation.html [fvb]: ../appendix/background.html#free-vs-bound [`replace_regions_in_mir`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/fn.replace_regions_in_mir.html [`compute_regions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/fn.compute_regions.html @@ -105,8 +104,10 @@ The kinds of region elements are as follows: ## Constraints -Before we can infer the value of regions, we need to collect constraints on the -regions. There are two primary types of constraints. +Before we can infer the value of regions, we need to collect +constraints on the regions. The full set of constraints is described +in [the section on constraint propagation][cp], but the two most +common sorts of constraints are: 1. Outlives constraints. These are constraints that one region outlives another (e.g. `'a: 'b`). Outlives constraints are generated by the [MIR type From 5c92fc254ea793df7a79c746bad5913925c41d82 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:32:53 -0400 Subject: [PATCH 589/648] Update src/borrow_check/region_inference/constraint_propagation.md Co-Authored-By: lqd --- src/borrow_check/region_inference/constraint_propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index e2c36062f..385e04892 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -178,7 +178,7 @@ easily just by taking the union: ``` for each region R: - let S by the SCC that contains R + let S be the SCC that contains R Values(S) = Values(S) union Liveness(R) ``` From 989c943eb3e34ef34873c67eb2018cb37c4347a2 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:33:02 -0400 Subject: [PATCH 590/648] Update src/borrow_check/region_inference/lifetime_parameters.md Co-Authored-By: lqd --- src/borrow_check/region_inference/lifetime_parameters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/lifetime_parameters.md b/src/borrow_check/region_inference/lifetime_parameters.md index 10759ee56..1a50f45f9 100644 --- a/src/borrow_check/region_inference/lifetime_parameters.md +++ b/src/borrow_check/region_inference/lifetime_parameters.md @@ -15,7 +15,7 @@ fn foo<'a, 'b>(x: &'a u32, y: &'b u32) -> &'b u32 { This example is intended not to compile, because we are returning `x`, which has type `&'a u32`, but our signature promises that we will -return a `&'b u32` value. But how are lifetimes like `'a` and `'b +return a `&'b u32` value. But how are lifetimes like `'a` and `'b` integrated into region inference, and how this error wind up being detected? From 13691d6372a6f330f76b3d4525564e8a479b2df6 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:33:15 -0400 Subject: [PATCH 591/648] Update src/borrow_check/region_inference/lifetime_parameters.md Co-Authored-By: lqd --- src/borrow_check/region_inference/lifetime_parameters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/lifetime_parameters.md b/src/borrow_check/region_inference/lifetime_parameters.md index 1a50f45f9..798af1337 100644 --- a/src/borrow_check/region_inference/lifetime_parameters.md +++ b/src/borrow_check/region_inference/lifetime_parameters.md @@ -103,7 +103,7 @@ rise to an outlives constraint `'a: 'b`. Combined with our default liveness constraints we get: ``` -'a live at {B, end('a)} // B represents the "bunction body" +'a live at {B, end('a)} // B represents the "function body" 'b live at {B, end('b)} 'a: 'b ``` From 3435363ef36fc804553a97734197678ccbe6c991 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:33:24 -0400 Subject: [PATCH 592/648] Update src/borrow_check/region_inference/lifetime_parameters.md Co-Authored-By: lqd --- src/borrow_check/region_inference/lifetime_parameters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/lifetime_parameters.md b/src/borrow_check/region_inference/lifetime_parameters.md index 798af1337..aa36d7dbb 100644 --- a/src/borrow_check/region_inference/lifetime_parameters.md +++ b/src/borrow_check/region_inference/lifetime_parameters.md @@ -30,7 +30,7 @@ the [`UniversalRegionRelations`] struct would track that `'a: 'b` is known to hold (which could be tested with the [`outlives`] function. [`UniversalRegions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/universal_regions/struct.UniversalRegions.html -[`UniversalRegionsRelations`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/type_check/free_region_relations/struct.UniversalRegionRelations.html +[`UniversalRegionRelations`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/type_check/free_region_relations/struct.UniversalRegionRelations.html [`outlives`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/type_check/free_region_relations/struct.UniversalRegionRelations.html#method.outlives ## Everything is a region variable From 53f83a9b1ca0dad9642fa7d07fe275a196e592ea Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:33:33 -0400 Subject: [PATCH 593/648] Update src/borrow_check/region_inference/constraint_propagation.md Co-Authored-By: lqd --- src/borrow_check/region_inference/constraint_propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index 385e04892..88fb00a7d 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -201,7 +201,7 @@ Values(S1) = ``` In the code, this work starts in the [`propagate_constraints`] -function, which iterates over all the SCCs. For each SCC S1, we +function, which iterates over all the SCCs. For each SCC `S1`, we compute its value by first computing the value of its successors. Since SCCs form a DAG, we don't have to be conecrned about cycles, though we do need to keep a set around to track whether we From 6e64baba5e5e3900e01fcd80c3bd63cca1848eda Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:33:41 -0400 Subject: [PATCH 594/648] Update src/borrow_check/region_inference/constraint_propagation.md Co-Authored-By: lqd --- src/borrow_check/region_inference/constraint_propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index 88fb00a7d..87a72eb1d 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -205,7 +205,7 @@ function, which iterates over all the SCCs. For each SCC `S1`, we compute its value by first computing the value of its successors. Since SCCs form a DAG, we don't have to be conecrned about cycles, though we do need to keep a set around to track whether we -have already processed a given SCC or not. For each successor S2, once +have already processed a given SCC or not. For each successor `S2`, once we have computed S2's value, we can union those elements into the value for S1. (Although we have to be careful in this process to properly handle [higher-ranked From 8858bdf8d2cd0fb21e3a2ba42744871ba487e22f Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:33:49 -0400 Subject: [PATCH 595/648] Update src/borrow_check/region_inference/constraint_propagation.md Co-Authored-By: lqd --- src/borrow_check/region_inference/constraint_propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index 87a72eb1d..95266a486 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -203,7 +203,7 @@ Values(S1) = In the code, this work starts in the [`propagate_constraints`] function, which iterates over all the SCCs. For each SCC `S1`, we compute its value by first computing the value of its -successors. Since SCCs form a DAG, we don't have to be conecrned about +successors. Since SCCs form a DAG, we don't have to be concerned about cycles, though we do need to keep a set around to track whether we have already processed a given SCC or not. For each successor `S2`, once we have computed S2's value, we can union those elements into the From d4f8036b393c7d67c814bc0ebfef9cd1a7c17e75 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:33:57 -0400 Subject: [PATCH 596/648] Update src/borrow_check/region_inference/constraint_propagation.md Co-Authored-By: lqd --- src/borrow_check/region_inference/constraint_propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index 95266a486..3ec374b66 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -206,7 +206,7 @@ compute its value by first computing the value of its successors. Since SCCs form a DAG, we don't have to be concerned about cycles, though we do need to keep a set around to track whether we have already processed a given SCC or not. For each successor `S2`, once -we have computed S2's value, we can union those elements into the +we have computed `S2`'s value, we can union those elements into the value for S1. (Although we have to be careful in this process to properly handle [higher-ranked placeholders](./placeholders_and_universes.html). Note that the value From a90e3417f73e9ef1cf5b12b627e716c9edb14590 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:34:05 -0400 Subject: [PATCH 597/648] Update src/borrow_check/region_inference/constraint_propagation.md Co-Authored-By: lqd --- src/borrow_check/region_inference/constraint_propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index 3ec374b66..19af52ec4 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -207,7 +207,7 @@ successors. Since SCCs form a DAG, we don't have to be concerned about cycles, though we do need to keep a set around to track whether we have already processed a given SCC or not. For each successor `S2`, once we have computed `S2`'s value, we can union those elements into the -value for S1. (Although we have to be careful in this process to +value for `S1`. (Although we have to be careful in this process to properly handle [higher-ranked placeholders](./placeholders_and_universes.html). Note that the value for S1 already contains the liveness constraints, since they were From f540a0ba419131512858d0d05595030738c7c97c Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:34:15 -0400 Subject: [PATCH 598/648] Update src/borrow_check/region_inference/constraint_propagation.md Co-Authored-By: lqd --- src/borrow_check/region_inference/constraint_propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index 19af52ec4..06cee8ad8 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -213,7 +213,7 @@ placeholders](./placeholders_and_universes.html). Note that the value for S1 already contains the liveness constraints, since they were added in [`RegionInferenceContext::new`]. -Once that process is done, we now have the "minimal value" for S1, +Once that process is done, we now have the "minimal value" for `S1`, taking into account all of the liveness and outlives constraints. However, in order to complete the process, we must also consider [member constraints][m_c], which are described in [a later From ae217e5775f924bd790066f7ced935f0cb1ec2c1 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:34:22 -0400 Subject: [PATCH 599/648] Update src/borrow_check/region_inference/constraint_propagation.md Co-Authored-By: lqd --- src/borrow_check/region_inference/constraint_propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index 06cee8ad8..e86d850fa 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -210,7 +210,7 @@ we have computed `S2`'s value, we can union those elements into the value for `S1`. (Although we have to be careful in this process to properly handle [higher-ranked placeholders](./placeholders_and_universes.html). Note that the value -for S1 already contains the liveness constraints, since they were +for `S1` already contains the liveness constraints, since they were added in [`RegionInferenceContext::new`]. Once that process is done, we now have the "minimal value" for `S1`, From b71caff4420e99affa4769fdee53b032cf739ec5 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:34:32 -0400 Subject: [PATCH 600/648] Update src/borrow_check/region_inference/member_constraints.md Co-Authored-By: lqd --- src/borrow_check/region_inference/member_constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/member_constraints.md b/src/borrow_check/region_inference/member_constraints.md index 4d7312644..5e62d5125 100644 --- a/src/borrow_check/region_inference/member_constraints.md +++ b/src/borrow_check/region_inference/member_constraints.md @@ -3,7 +3,7 @@ A member constraint `'m member of ['c_1..'c_N]` expresses that the region `'m` must be *equal* to some **choice regions** `'c_i` (for some `i`). These constraints cannot be expressed by users, but they -arise from `impl Trait` due to its lifetime capture rules. Consinder a +arise from `impl Trait` due to its lifetime capture rules. Consider a function such as the following: ```rust From 32b57634c3d41821eb6e2a7d09c566cad60d91a4 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:34:40 -0400 Subject: [PATCH 601/648] Update src/borrow_check/region_inference/lifetime_parameters.md Co-Authored-By: lqd --- src/borrow_check/region_inference/lifetime_parameters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/lifetime_parameters.md b/src/borrow_check/region_inference/lifetime_parameters.md index aa36d7dbb..ecd3a19a5 100644 --- a/src/borrow_check/region_inference/lifetime_parameters.md +++ b/src/borrow_check/region_inference/lifetime_parameters.md @@ -64,7 +64,7 @@ As noted previously, the value that we infer for each region is a set `{E}`. The elements of this set can be points in the control-flow graph, but they can also be an element `end('a)` corresponding to each universal lifetime `'a`. If the value for some region `R0` includes -`end('a`), then this implies that R0 must extend until the end of `'a` +`end('a`), then this implies that `R0` must extend until the end of `'a` in the caller. ## The "value" of a universal region From 8c78b7dd810b249457a5c7f272fbe8dc49ce7fba Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:35:04 -0400 Subject: [PATCH 602/648] Update src/borrow_check/region_inference/member_constraints.md Co-Authored-By: lqd --- src/borrow_check/region_inference/member_constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/member_constraints.md b/src/borrow_check/region_inference/member_constraints.md index 5e62d5125..1d5fb1013 100644 --- a/src/borrow_check/region_inference/member_constraints.md +++ b/src/borrow_check/region_inference/member_constraints.md @@ -11,7 +11,7 @@ fn make(a: &'a u32, b: &'b u32) -> impl Trait<'a, 'b> { .. } ``` Here, the true return type (often called the "hidden type") is only -permitted to capture the lifeimes `'a` or `'b`. You can kind of see +permitted to capture the lifetimes `'a` or `'b`. You can kind of see this more clearly by desugaring that `impl Trait` return type into its more explicit form: From 16e49002c793e8fd18359b7e6eb169c0b0650af5 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:35:17 -0400 Subject: [PATCH 603/648] Update src/borrow_check/region_inference/member_constraints.md Co-Authored-By: lqd --- src/borrow_check/region_inference/member_constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/member_constraints.md b/src/borrow_check/region_inference/member_constraints.md index 1d5fb1013..219296d0d 100644 --- a/src/borrow_check/region_inference/member_constraints.md +++ b/src/borrow_check/region_inference/member_constraints.md @@ -124,7 +124,7 @@ bounds considering everything but member constraints). Let `LB` be the current value of `'0`. We know then that `'0: LB` must hold, whatever the final value of `'0` is. Therefore, we can rule out -any choice where `'choice` where `'choice: LB` does not hold. +any choice `'choice` where `'choice: LB` does not hold. Unfortunately, in our example, this is not very helpful. The lower bound for `'0` will just be the liveness set `{L}`, and we know that From 1a66e0c6a9266669c5e7779d99f58eda514eb99a Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:35:37 -0400 Subject: [PATCH 604/648] Update src/borrow_check/region_inference/constraint_propagation.md Co-Authored-By: Who? Me?! --- src/borrow_check/region_inference/constraint_propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index e86d850fa..c655df959 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -35,7 +35,7 @@ As a simple example, if we have a liveness constraint `R live at E`, then we can apply `Values(R) = Values(R) union {E}` to make the constraint be satisfied. Similarly, if we have an outlives constraints `R1: R2`, we can apply `Values(R1) = Values(R1) union Values(R2)`. -(Member constraints are more complex and we discuss them below.) +(Member constraints are more complex and we discuss them [in this section][m_c].) In practice, however, we are a bit more clever. Instead of applying the constraints in a loop, we can analyze the constraints and figure From 47dc406417759577f50d4a0433a47e8b6b48d6e0 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:35:48 -0400 Subject: [PATCH 605/648] Update src/borrow_check/region_inference/member_constraints.md Co-Authored-By: lqd --- src/borrow_check/region_inference/member_constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/member_constraints.md b/src/borrow_check/region_inference/member_constraints.md index 219296d0d..48b24d215 100644 --- a/src/borrow_check/region_inference/member_constraints.md +++ b/src/borrow_check/region_inference/member_constraints.md @@ -135,7 +135,7 @@ with different variance, lower bound constraints may be relevant.) ### Upper bounds The *upper bounds* are those lifetimes that *must outlive* `'0` -- -i.e., that `'0` must be *smaller* then. In our example, this would be +i.e., that `'0` must be *smaller* than. In our example, this would be `'a`, because we have the constraint that `'a: '0`. In more complex examples, the chain may be more indirect. From 986780beef728cfb060b55a31419f57d5d6b1b33 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:35:59 -0400 Subject: [PATCH 606/648] Update src/borrow_check/region_inference/constraint_propagation.md Co-Authored-By: Who? Me?! --- src/borrow_check/region_inference/constraint_propagation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index c655df959..54f2c4cfd 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -59,7 +59,7 @@ A liveness constraint `R live at E` is satisfied if `E` is a member of `Values(R)`. So to "apply" such a constraint to `Values`, we just have to compute `Values(R) = Values(R) union {E}`. -The liveness values are computed in the type-check and passes to the +The liveness values are computed in the type-check and passed to the region inference upon creation in the `liveness_constraints` argument. These are not represented as individual constraints like `R live at E` though; instead, we store a (sparse) bitset per region variable (of From 0c6f78907a1a2fe4b4e462ce098e0e30d2ed247c Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:36:59 -0400 Subject: [PATCH 607/648] Update src/borrow_check/region_inference/member_constraints.md Co-Authored-By: Who? Me?! --- src/borrow_check/region_inference/member_constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/member_constraints.md b/src/borrow_check/region_inference/member_constraints.md index 48b24d215..3f934e864 100644 --- a/src/borrow_check/region_inference/member_constraints.md +++ b/src/borrow_check/region_inference/member_constraints.md @@ -69,7 +69,7 @@ interesting to the point we are illustrating here). The `'a: '0` and `'b: '1` constraints arise from subtyping. When we construct the `(a, b)` value, it will be assigned type `(&'0 u32, &'1 -u32)` -- the region variables reflect that the lifetime of this +u32)` -- the region variables reflect that the lifetimes of these references could be made smaller. For this value to be created from `a` and `b`, however, we do require that: From edeb237444ad379c58c877fa1d97bc54213e0fac Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:37:12 -0400 Subject: [PATCH 608/648] Update src/borrow_check/region_inference/placeholders_and_universes.md Co-Authored-By: Who? Me?! --- src/borrow_check/region_inference/placeholders_and_universes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/placeholders_and_universes.md b/src/borrow_check/region_inference/placeholders_and_universes.md index 0d4068e00..cc262ad3e 100644 --- a/src/borrow_check/region_inference/placeholders_and_universes.md +++ b/src/borrow_check/region_inference/placeholders_and_universes.md @@ -66,7 +66,7 @@ outlives `'static`. Now, this *might* be true – after all, `'!1` could be `'static` – but we don't *know* that it's true. So this should yield up an error (eventually). -## What is a universe +## What is a universe? In the previous section, we introduced the idea of a placeholder region, and we denoted it `!1`. We call this number `1` the **universe From 5bfca5a8711fa442487d7118e17444e3dfcac180 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:37:22 -0400 Subject: [PATCH 609/648] Update src/borrow_check/region_inference/placeholders_and_universes.md Co-Authored-By: Who? Me?! --- src/borrow_check/region_inference/placeholders_and_universes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/placeholders_and_universes.md b/src/borrow_check/region_inference/placeholders_and_universes.md index cc262ad3e..57fae404b 100644 --- a/src/borrow_check/region_inference/placeholders_and_universes.md +++ b/src/borrow_check/region_inference/placeholders_and_universes.md @@ -37,7 +37,7 @@ the type of `foo` the type `bar` expects We handle this sort of subtyping by taking the variables that are bound in the supertype and replacing them with [universally quantified](../appendix/background.html#quantified) -representatives, written like `!1`. We call these regions "placeholder +representatives, denoted like `!1` here. We call these regions "placeholder regions" – they represent, basically, "some unknown region". Once we've done that replacement, we have the following relation: From 364f0e4f8ad1a9d4e810be45683db3d8c4eefc03 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:37:30 -0400 Subject: [PATCH 610/648] Update src/borrow_check/region_inference/placeholders_and_universes.md Co-Authored-By: Who? Me?! --- src/borrow_check/region_inference/placeholders_and_universes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/borrow_check/region_inference/placeholders_and_universes.md b/src/borrow_check/region_inference/placeholders_and_universes.md index 57fae404b..604c793d5 100644 --- a/src/borrow_check/region_inference/placeholders_and_universes.md +++ b/src/borrow_check/region_inference/placeholders_and_universes.md @@ -145,7 +145,7 @@ notion of universes, we can use it to extend our type-checker and things to prevent illegal names from leaking out. The idea is that we give each inference (existential) variable – whether it be a type or a lifetime – a universe. That variable's value can then only -reference names visible from that universe. So for example is a +reference names visible from that universe. So for example if a lifetime variable is created in U0, then it cannot be assigned a value of `!1` or `!2`, because those names are not visible from the universe U0. From a8de9a1a49d90c2616f72276d37e0c8a004a48ba Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:41:25 -0400 Subject: [PATCH 611/648] fix indentation --- .../region_inference/placeholders_and_universes.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/borrow_check/region_inference/placeholders_and_universes.md b/src/borrow_check/region_inference/placeholders_and_universes.md index 604c793d5..e40ee11b1 100644 --- a/src/borrow_check/region_inference/placeholders_and_universes.md +++ b/src/borrow_check/region_inference/placeholders_and_universes.md @@ -271,9 +271,9 @@ The region inference engine will create a region element domain like this: ```text { CFG; end('static); placeholder(1) } - --- ------------ ------- from the universe `!1` - | 'static is always in scope - all points in the CFG; not especially relevant here + --- ------------ ------- from the universe `!1` + | 'static is always in scope + all points in the CFG; not especially relevant here ``` It will always create two universal variables, one representing From ed9535686880eb1ce470d2fe4ae27b93f3bd59d6 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Mon, 24 Jun 2019 14:44:13 -0400 Subject: [PATCH 612/648] add `point` to the glossary and link a use of it --- src/appendix/glossary.md | 1 + src/borrow_check/region_inference/constraint_propagation.md | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 3d77def91..007b1bb4c 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -50,6 +50,7 @@ newtype | a "newtype" is a wrapper around some other type (e.g. NLL | [non-lexical lifetimes](../borrow_check/region_inference.html), an extension to Rust's borrowing system to make it be based on the control-flow graph. node-id or NodeId | an index identifying a particular node in the AST or HIR; gradually being phased out and replaced with `HirId`. obligation | something that must be proven by the trait system ([see more](../traits/resolution.html)) +point | used in the NLL analysis to refer to some particular location in the MIR; typically used to refer to a node in the control-flow graph. projection | a general term for a "relative path", e.g. `x.f` is a "field projection", and `T::Item` is an ["associated type projection"](../traits/goals-and-clauses.html#trait-ref) promoted constants | constants extracted from a function and lifted to static scope; see [this section](../mir/index.html#promoted) for more details. provider | the function that executes a query ([see more](../query.html)) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index 54f2c4cfd..7b9eeb13d 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -51,10 +51,12 @@ in the section on outlives constraints. ## Liveness constraints A **liveness constraint** arises when some variable whose type -includes a region R is live at some point P. This simply means that +includes a region R is live at some [point] P. This simply means that the value of R must include the point P. Liveness constraints are computed by the MIR type checker. +[point]: ../../appendix/glossary.html + A liveness constraint `R live at E` is satisfied if `E` is a member of `Values(R)`. So to "apply" such a constraint to `Values`, we just have to compute `Values(R) = Values(R) union {E}`. From abf512fc9cc969dcbea69aa15b44586bbeb13c2d Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 26 Jun 2019 11:03:43 -0500 Subject: [PATCH 613/648] fix long line --- src/SUMMARY.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 69e46b796..8c8890b79 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -79,7 +79,7 @@ - [Constraint propagation](./borrow_check/region_inference/constraint_propagation.md) - [Lifetime parameters](./borrow_check/region_inference/lifetime_parameters.md) - [Member constraints](./borrow_check/region_inference/member_constraints.md) - - [Placeholders and universes](./borrow_check/region_inference/placeholders_and_universes.md) + - [Placeholders and universes][pau] - [Closure constraints](./borrow_check/region_inference/closure_constraints.md) - [Errror reporting](./borrow_check/region_inference/error_reporting.md) - [Two-phase-borrows](./borrow_check/two_phase_borrows.md) @@ -100,3 +100,5 @@ [Appendix D: Code Index](./appendix/code-index.md) [Appendix E: Bibliography](./appendix/bibliography.md) [](./important-links.md) + +[pau]: ./borrow_check/region_inference/placeholders_and_universes.md From 87e355245c6b4d0e8b2877acf07627afc0d07493 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 26 Jun 2019 14:26:14 -0500 Subject: [PATCH 614/648] fix links --- src/borrow_check/region_inference/constraint_propagation.md | 6 +++--- .../region_inference/placeholders_and_universes.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index 7b9eeb13d..daa52393a 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -11,7 +11,7 @@ on one at a time (each of them is fairly independent from the others): - [member constraints][m_c] (`member R_m of [R_c...]`), which arise from impl Trait. [`propagate_constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.propagate_constraints -[m_c]: ./member_constraints.html +[m_c]: ./member_constraints.md In this chapter, we'll explain the "heart" of constraint propagation, covering both liveness and outlives constraints. @@ -29,7 +29,7 @@ given some set of constraints `{C}` and it computes a set of values - For each constraint C: - Update `Values` as needed to satisfy the constraint -[riv]: ../region-inference.html#region-variables +[riv]: ../region_inference.md#region-variables As a simple example, if we have a liveness constraint `R live at E`, then we can apply `Values(R) = Values(R) union {E}` to make the @@ -55,7 +55,7 @@ includes a region R is live at some [point] P. This simply means that the value of R must include the point P. Liveness constraints are computed by the MIR type checker. -[point]: ../../appendix/glossary.html +[point]: ../../appendix/glossary.md A liveness constraint `R live at E` is satisfied if `E` is a member of `Values(R)`. So to "apply" such a constraint to `Values`, we just have diff --git a/src/borrow_check/region_inference/placeholders_and_universes.md b/src/borrow_check/region_inference/placeholders_and_universes.md index e40ee11b1..eda8e6ff1 100644 --- a/src/borrow_check/region_inference/placeholders_and_universes.md +++ b/src/borrow_check/region_inference/placeholders_and_universes.md @@ -36,7 +36,7 @@ the type of `foo` the type `bar` expects We handle this sort of subtyping by taking the variables that are bound in the supertype and replacing them with -[universally quantified](../appendix/background.html#quantified) +[universally quantified](../../appendix/background.md#quantified) representatives, denoted like `!1` here. We call these regions "placeholder regions" – they represent, basically, "some unknown region". @@ -53,7 +53,7 @@ what we wanted. So let's work through what happens next. To check if two functions are subtypes, we check if their arguments have the desired relationship -(fn arguments are [contravariant](../appendix/background.html#variance), so +(fn arguments are [contravariant](../../appendix/background.md#variance), so we swap the left and right here): ```text @@ -92,7 +92,7 @@ Here, the root universe would consist of the lifetimes `'static` and the same concept to types, in which case the types `Foo` and `T` would be in the root universe (along with other global types, like `i32`). Basically, the root universe contains all the names that -[appear free](../appendix/background.html#free-vs-bound) in the body of `bar`. +[appear free](../../appendix/background.md#free-vs-bound) in the body of `bar`. Now let's extend `bar` a bit by adding a variable `x`: From c6d994370b604a0b0aa01a83d4ee7a49ac3e7cb4 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 26 Jun 2019 14:34:56 -0500 Subject: [PATCH 615/648] fix mdbook test --- .../region_inference/constraint_propagation.md | 10 +++++----- .../region_inference/lifetime_parameters.md | 6 +++--- .../region_inference/member_constraints.md | 12 ++++++------ 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index daa52393a..a8e44629e 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -88,7 +88,7 @@ have to compute `Values(R1) = Values(R1) union Values(R2)`. One observation that follows from this is that if you have `R1: R2` and `R2: R1`, then `R1 = R2` must be true. Similarly, if you have: -``` +```txt R1: R2 R2: R3 R3: R4 @@ -119,7 +119,7 @@ context. When using a graph representation, we can detect regions that must be equal by looking for cycles. That is, if you have a constraint like -``` +```txt 'a: 'b 'b: 'c 'c: 'd @@ -153,7 +153,7 @@ When we compute SCCs, we not only figure out which regions are a member of each SCC, we also figure out the edges between them. So for example consider this set of outlives constraints: -``` +```txt 'a: 'b 'b: 'a @@ -178,7 +178,7 @@ expressed in terms of regions -- that is, we have a map like in terms of SCCs -- we can integrate these liveness constraints very easily just by taking the union: -``` +```txt for each region R: let S be the SCC that contains R Values(S) = Values(S) union Liveness(R) @@ -195,7 +195,7 @@ the value of `S1`, we first compute the values of each successor `S2`. Then we simply union all of those values together. To use a quasi-iterator-like notation: -``` +```txt Values(S1) = s1.successors() .map(|s2| Values(s2)) diff --git a/src/borrow_check/region_inference/lifetime_parameters.md b/src/borrow_check/region_inference/lifetime_parameters.md index ecd3a19a5..87e9d53ed 100644 --- a/src/borrow_check/region_inference/lifetime_parameters.md +++ b/src/borrow_check/region_inference/lifetime_parameters.md @@ -7,7 +7,7 @@ derives from the fact that such lifetimes are "universally quantified" lifetimes). It is worth spending a bit of discussing how lifetime parameters are handled during region inference. Consider this example: -```rust +```rust,ignore fn foo<'a, 'b>(x: &'a u32, y: &'b u32) -> &'b u32 { x } @@ -92,7 +92,7 @@ itself). In the code, these liveness constraints are setup in So, consider the first example of this section: -```rust +```rust,ignore fn foo<'a, 'b>(x: &'a u32, y: &'b u32) -> &'b u32 { x } @@ -102,7 +102,7 @@ Here, returning `x` requires that `&'a u32 <: &'b u32`, which gives rise to an outlives constraint `'a: 'b`. Combined with our default liveness constraints we get: -``` +```txt 'a live at {B, end('a)} // B represents the "function body" 'b live at {B, end('b)} 'a: 'b diff --git a/src/borrow_check/region_inference/member_constraints.md b/src/borrow_check/region_inference/member_constraints.md index 3f934e864..d69c95511 100644 --- a/src/borrow_check/region_inference/member_constraints.md +++ b/src/borrow_check/region_inference/member_constraints.md @@ -6,7 +6,7 @@ some `i`). These constraints cannot be expressed by users, but they arise from `impl Trait` due to its lifetime capture rules. Consider a function such as the following: -```rust +```rust,ignore fn make(a: &'a u32, b: &'b u32) -> impl Trait<'a, 'b> { .. } ``` @@ -15,7 +15,7 @@ permitted to capture the lifetimes `'a` or `'b`. You can kind of see this more clearly by desugaring that `impl Trait` return type into its more explicit form: -```rust +```rust,ignore type MakeReturn<'x, 'y> = impl Trait<'x, 'y>; fn make(a: &'a u32, b: &'b u32) -> MakeReturn<'a, 'b> { .. } ``` @@ -34,14 +34,14 @@ To help us explain member constraints in more detail, let's spell out the `make` example in a bit more detail. First off, let's assume that you have some dummy trait: -```rust +```rust,ignore trait Trait<'a, 'b> { } impl Trait<'_, '_> for T { } ``` and this is the `make` function (in desugared form): -```rust +```rust,ignore type MakeReturn<'x, 'y> = impl Trait<'x, 'y>; fn make(a: &'a u32, b: &'b u32) -> MakeReturn<'a, 'b> { (a, b) @@ -52,7 +52,7 @@ What happens in this case is that the return type will be `(&'0 u32, &'1 u32)`, where `'0` and `'1` are fresh region variables. We will have the following region constraints: -``` +```txt '0 live at {L} '1 live at {L} 'a: '0 @@ -73,7 +73,7 @@ u32)` -- the region variables reflect that the lifetimes of these references could be made smaller. For this value to be created from `a` and `b`, however, we do require that: -``` +```txt (&'a u32, &'b u32) <: (&'0 u32, &'1 u32) ``` From 7f47c496c272a00c76a733bbf71c89e606210605 Mon Sep 17 00:00:00 2001 From: Yuki Okushi Date: Thu, 27 Jun 2019 05:45:00 +0900 Subject: [PATCH 616/648] Fix typo --- src/mir/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/mir/index.md b/src/mir/index.md index baaf23c36..9e9f933fe 100644 --- a/src/mir/index.md +++ b/src/mir/index.md @@ -138,7 +138,7 @@ than one successor – that is, control may flow to different places. Function calls like the call to `Vec::new` are always terminators because of the possibility of unwinding, although in the case of `Vec::new` we are able to see that indeed unwinding is not -possible, and hence we list only one succssor block, `bb2`. +possible, and hence we list only one successor block, `bb2`. If we look ahead to `bb2`, we will see it looks like this: From 92f3f8e3b8c0a2dd29cc4603b9cc7591bf6edfb4 Mon Sep 17 00:00:00 2001 From: Yuki Okushi Date: Thu, 4 Jul 2019 20:39:45 +0900 Subject: [PATCH 617/648] Update pass modes for ui tests (#380) --- src/tests/adding.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/src/tests/adding.md b/src/tests/adding.md index 5b8674814..d88b7cf57 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -171,8 +171,8 @@ source. argument is the first version to ignore. If no second argument is given, all subsequent versions are ignored; otherwise, the second argument is the last version to ignore. -* `compile-pass` for UI tests, indicates that the test is - supposed to compile, as opposed to the default where the test is +* `build-pass` for UI tests, indicates that the test is supposed to + successfully compile and link, as opposed to the default where the test is supposed to error out. * `compile-flags` passes extra command-line args to the compiler, e.g. `compile-flags -g` which forces debuginfo to be enabled. @@ -279,7 +279,9 @@ can also make UI tests where compilation is expected to succeed, and you can even run the resulting program. Just add one of the following [header commands](#header_commands): -- `// compile-pass` – compilation should succeed but do +- `// check-pass` - compilation should succeed but skip codegen + (which is expensive and isn't supposed to fail in most cases) +- `// build-pass` – compilation and linking should succeed but do not run the resulting binary - `// run-pass` – compilation should succeed and we should run the resulting binary From b5a2b9353c661000378415ecfeb757eb7df42d66 Mon Sep 17 00:00:00 2001 From: Yuki Okushi Date: Thu, 4 Jul 2019 20:41:50 +0900 Subject: [PATCH 618/648] Fix broken links (#382) --- src/borrow_check/region_inference/constraint_propagation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index a8e44629e..7ec46b90f 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -112,8 +112,8 @@ induces an edge `'a -> 'b`. This conversion happens in the [`RegionInferenceContext::new`] function that creates the inference context. -[`ConstraintSet`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/constraints/struct.ConstraintSet.html -[graph-fn]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/constraints/struct.ConstraintSet.html#method.graph +[`ConstraintSet`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/constraints/struct.OutlivesConstraintSet.html +[graph-fn]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/constraints/struct.OutlivesConstraintSet.html#method.graph [`RegionInferenceContext::new`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.new When using a graph representation, we can detect regions that must be equal From 4936b7cbd53b477a3389c5bb655e92c5f3a2ff0c Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 26 Jun 2019 10:58:05 -0500 Subject: [PATCH 619/648] add humor appendix; closes #163 --- src/SUMMARY.md | 5 +++++ src/appendix/humorust.md | 10 ++++++++++ 2 files changed, 15 insertions(+) create mode 100644 src/appendix/humorust.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 8c8890b79..e321f6633 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -99,6 +99,11 @@ [Appendix C: Glossary](./appendix/glossary.md) [Appendix D: Code Index](./appendix/code-index.md) [Appendix E: Bibliography](./appendix/bibliography.md) + +[Appendix Z HumorRust](./appendix/humorust.md) + +--- + [](./important-links.md) [pau]: ./borrow_check/region_inference/placeholders_and_universes.md diff --git a/src/appendix/humorust.md b/src/appendix/humorust.md new file mode 100644 index 000000000..e217ab65f --- /dev/null +++ b/src/appendix/humorust.md @@ -0,0 +1,10 @@ +# Humor in Rust + +What's a project without a sense of humor? And frankly some of these are +enlightening? + +- [Weird exprs test](https://github.com/rust-lang/rust/blob/master/src/test/run-pass/weird-exprs.rs) +- [Ferris Rap](http://fitzgeraldnick.com/2018/12/13/rust-raps.html) +- [The Genesis of Generic Germination](https://github.com/rust-lang/rust/pull/53645#issue-210543221) +- [The Bastion of the Turbofish test](https://github.com/rust-lang/rust/blob/79d8a0fcefa5134db2a94739b1d18daa01fc6e9f/src/test/ui/bastion-of-the-turbofish.rs) +- [Rust Koans](https://users.rust-lang.org/t/rust-koans/2408/4) From 06837ee64b2d4b9e08e1303b22f9aba7a9ff59cd Mon Sep 17 00:00:00 2001 From: Yuki Okushi Date: Wed, 17 Jul 2019 22:02:05 +0900 Subject: [PATCH 620/648] Update experts' link --- src/compiler-team.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/compiler-team.md b/src/compiler-team.md index 968e2f142..f01b4a8d5 100644 --- a/src/compiler-team.md +++ b/src/compiler-team.md @@ -18,10 +18,12 @@ Currently the compiler team chats in 2 places: If you're interested in figuring out who can answer questions about a particular part of the compiler, or you'd just like to know who works on what, -check out our [experts directory](https://github.com/rust-lang/compiler-team/blob/master/experts/MAP.md). +check out our [experts directory][experts]. It contains a listing of the various parts of the compiler and a list of people who are experts on each one. +[experts]: https://github.com/rust-lang/compiler-team/blob/master/content/docs/experts/map.toml + ## Rust compiler meeting The compiler team has a weekly meeting where we do triage and try to From 596e952f649e0aa8ef1f393d85b0df4bda4df4fe Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 17 Jul 2019 10:10:43 -0500 Subject: [PATCH 621/648] blacklist some commonly problematic links --- book.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book.toml b/book.toml index b9092a969..856ef2eba 100644 --- a/book.toml +++ b/book.toml @@ -9,4 +9,4 @@ description = "A guide to developing rustc" [output.linkcheck] follow-web-links = true -exclude = [ "crates\\.io", "gcc\\.godbolt\\.org" ] +exclude = [ "crates\\.io", "gcc\\.godbolt\\.org", "youtube\\.com", "dl\\.acm\\.org" ] From fc18a79ef688ee47ece5c736218d92537cf3bf63 Mon Sep 17 00:00:00 2001 From: Yuki Okushi Date: Fri, 12 Jul 2019 11:44:03 +0900 Subject: [PATCH 622/648] Document `--pass $mode` --- src/tests/running.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/tests/running.md b/src/tests/running.md index 5a05c156f..01d308f0b 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -108,6 +108,23 @@ Under the hood, the test runner invokes the standard rust test runner filtering for tests that include "issue-1234" in the name. (Thus `--test-args` is a good way to run a collection of related tests.) +## Passing `--pass $mode` + +Pass UI tests now have three modes, `check-pass`, `build-pass` and +`run-pass`. When `--pass $mode` is passed, these tests will be forced +to run under the given `$mode` unless the directive `// ignore-pass` +exists in the test file. For example, you can run all the tests in +`src/test/run-pass` as `check-pass`: + +```bash +> ./x.py test --stage 1 src/test/run-pass --pass check +``` + +By passing `--pass $mode`, you can reduce the testing time. For each +mode, please see [here][mode]. + +[mode]: https://rust-lang.github.io/rustc-guide/tests/adding.html#tests-that-do-not-result-in-compile-errors + ## Using incremental compilation You can further enable the `--incremental` flag to save additional From 1774649715ec84b80d8c887d913c91b83be56197 Mon Sep 17 00:00:00 2001 From: Yuki Okushi Date: Fri, 12 Jul 2019 11:02:45 +0900 Subject: [PATCH 623/648] Minor fix in borrow_check Fixes as follows: - fixes spelling - removes unnecessary white spaces and blank lines --- src/borrow_check/moves_and_initialization/move_paths.md | 5 ++--- src/borrow_check/region_inference.md | 1 - src/borrow_check/region_inference/constraint_propagation.md | 4 +--- src/borrow_check/region_inference/lifetime_parameters.md | 2 +- src/borrow_check/region_inference/member_constraints.md | 5 ++--- 5 files changed, 6 insertions(+), 11 deletions(-) diff --git a/src/borrow_check/moves_and_initialization/move_paths.md b/src/borrow_check/moves_and_initialization/move_paths.md index 4d4db4116..d733e8ac7 100644 --- a/src/borrow_check/moves_and_initialization/move_paths.md +++ b/src/borrow_check/moves_and_initialization/move_paths.md @@ -12,7 +12,7 @@ fn foo() { let b = a.0; // moves a.0 - // a.0 is not initializd, but a.1 still is + // a.0 is not initialized, but a.1 still is let c = a.0; // ERROR let d = a.1; // OK @@ -84,7 +84,7 @@ initialized (which lowers overhead). ## Looking up a move-path -If you have a [`Place`] and you would like to convert it to a [`MovePathIndex`], you +If you have a [`Place`] and you would like to convert it to a [`MovePathIndex`], you can do that using the [`MovePathLookup`] structure found in the [`rev_lookup`] field of [`MoveData`]. There are two different methods: @@ -124,4 +124,3 @@ given move-path (e.g., `a.b`) or any child of that move-path (e.g., [`Place`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/enum.Place.html [`has_any_child_of`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/at_location/struct.FlowAtLocation.html#method.has_any_child_of - diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index 46caab567..52659e93c 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -236,4 +236,3 @@ tests and universal regions, as discussed above. [`propagate_constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.propagate_constraints [`check_type_tests`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.check_type_tests [`check_universal_regions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#method.check_universal_regions - diff --git a/src/borrow_check/region_inference/constraint_propagation.md b/src/borrow_check/region_inference/constraint_propagation.md index 7ec46b90f..5f3b29c6d 100644 --- a/src/borrow_check/region_inference/constraint_propagation.md +++ b/src/borrow_check/region_inference/constraint_propagation.md @@ -137,7 +137,7 @@ by invoking `constraint_sccs.scc(r)`. Working in terms of SCCs allows us to be more efficient: if we have a set of regions `'a...'d` that are part of a single SCC, we don't have -to compute/store their values separarely. We can just store one value +to compute/store their values separately. We can just store one value **for the SCC**, since they must all be equal. If you look over the region inference code, you will see that a number @@ -220,5 +220,3 @@ taking into account all of the liveness and outlives constraints. However, in order to complete the process, we must also consider [member constraints][m_c], which are described in [a later section][m_c]. - - diff --git a/src/borrow_check/region_inference/lifetime_parameters.md b/src/borrow_check/region_inference/lifetime_parameters.md index 87e9d53ed..0d5219aa9 100644 --- a/src/borrow_check/region_inference/lifetime_parameters.md +++ b/src/borrow_check/region_inference/lifetime_parameters.md @@ -52,7 +52,7 @@ based on their index: In fact, the universal regions can be further subdivided based on where they were brought into scope (see the [`RegionClassification`] -type). These subdivions are not important for the topics discussed +type). These subdivisions are not important for the topics discussed here, but become important when we consider [closure constraint propagation](./closure_constraints.html), so we discuss them there. diff --git a/src/borrow_check/region_inference/member_constraints.md b/src/borrow_check/region_inference/member_constraints.md index d69c95511..7e254f0a2 100644 --- a/src/borrow_check/region_inference/member_constraints.md +++ b/src/borrow_check/region_inference/member_constraints.md @@ -176,11 +176,11 @@ region being inferred. However, it is somewhat arbitrary. In practice, computing upper bounds is a bit inconvenient, because our data structures are setup for the opposite. What we do is to compute -the **reverse SCC graph** (we do this lazilly and cache the result) -- +the **reverse SCC graph** (we do this lazily and cache the result) -- that is, a graph where `'a: 'b` induces an edge `SCC('b) -> SCC('a)`. Like the normal SCC graph, this is a DAG. We can then do a depth-first search starting from `SCC('0)` in this graph. This will -take us to all the SCCs that must outlive `'0`. +take us to all the SCCs that must outlive `'0`. One wrinkle is that, as we walk the "upper bound" SCCs, their values will not yet have been fully computed. However, we **have** already @@ -190,4 +190,3 @@ parameters, their value will contain themselves (i.e., the initial value for `'a` includes `'a` and the value for `'b` contains `'b`). So we can collect all of the lifetime parameters that are reachable, which is precisely what we are interested in. - From fb3bf19590bf7dd24af507180bca6f5bf0c27d80 Mon Sep 17 00:00:00 2001 From: Yuki Okushi Date: Fri, 12 Jul 2019 11:58:41 +0900 Subject: [PATCH 624/648] Move into rust-lang --- README.md | 2 +- src/traits/index.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e6230439e..064f80c6e 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ works. The aim of the guide is to help new contributors get oriented to rustc, as well as to help more experienced folks in figuring out some new part of the compiler that they haven't worked on before. -[You can read the latest version of the guide here.](https://rust-lang-nursery.github.io/rustc-guide/) +[You can read the latest version of the guide here.](https://rust-lang.github.io/rustc-guide/) You may also find the rustdocs [for the compiler itself][rustdocs] useful. diff --git a/src/traits/index.md b/src/traits/index.md index 84f812394..5544ce513 100644 --- a/src/traits/index.md +++ b/src/traits/index.md @@ -59,6 +59,6 @@ and designs for the trait system. It primarily consists of two parts: implementing them in rustc. This mainly happens in [`librustc_traits`][librustc_traits]. -[chalk]: https://github.com/rust-lang-nursery/chalk -[chalk_engine]: https://github.com/rust-lang-nursery/chalk/tree/master/chalk-engine +[chalk]: https://github.com/rust-lang/chalk +[chalk_engine]: https://github.com/rust-lang/chalk/tree/master/chalk-engine [librustc_traits]: https://github.com/rust-lang/rust/tree/master/src/librustc_traits From 7b106f95466f1040e0c8b4af512ace2d9f2cd7b4 Mon Sep 17 00:00:00 2001 From: Yuki Okushi Date: Fri, 12 Jul 2019 12:00:12 +0900 Subject: [PATCH 625/648] Add colon --- src/SUMMARY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index e321f6633..60aa8766d 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -100,7 +100,7 @@ [Appendix D: Code Index](./appendix/code-index.md) [Appendix E: Bibliography](./appendix/bibliography.md) -[Appendix Z HumorRust](./appendix/humorust.md) +[Appendix Z: HumorRust](./appendix/humorust.md) --- From ef9bc9d610cad6dbe903a23bccf0447dd2a411c5 Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 17 Jul 2019 12:01:32 -0500 Subject: [PATCH 626/648] break rust --- src/appendix/humorust.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/appendix/humorust.md b/src/appendix/humorust.md index e217ab65f..f044a50c0 100644 --- a/src/appendix/humorust.md +++ b/src/appendix/humorust.md @@ -8,3 +8,4 @@ enlightening? - [The Genesis of Generic Germination](https://github.com/rust-lang/rust/pull/53645#issue-210543221) - [The Bastion of the Turbofish test](https://github.com/rust-lang/rust/blob/79d8a0fcefa5134db2a94739b1d18daa01fc6e9f/src/test/ui/bastion-of-the-turbofish.rs) - [Rust Koans](https://users.rust-lang.org/t/rust-koans/2408/4) +- [`break rust;`](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=0ab2bd6a9d722e0f05a95e2a5dcf89cc) From 248d8af35e9423b92760d7fc2603d4df8a78345c Mon Sep 17 00:00:00 2001 From: Mark Mansi Date: Wed, 17 Jul 2019 12:04:18 -0500 Subject: [PATCH 627/648] add nomicon intro too --- src/appendix/humorust.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/appendix/humorust.md b/src/appendix/humorust.md index f044a50c0..36c25c890 100644 --- a/src/appendix/humorust.md +++ b/src/appendix/humorust.md @@ -9,3 +9,4 @@ enlightening? - [The Bastion of the Turbofish test](https://github.com/rust-lang/rust/blob/79d8a0fcefa5134db2a94739b1d18daa01fc6e9f/src/test/ui/bastion-of-the-turbofish.rs) - [Rust Koans](https://users.rust-lang.org/t/rust-koans/2408/4) - [`break rust;`](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=0ab2bd6a9d722e0f05a95e2a5dcf89cc) +- [The Nomicon Intro](https://doc.rust-lang.org/stable/nomicon/) From 382f1c4867da6791a0b1cb28bbc54b3d22cf56cd Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Fri, 19 Jul 2019 11:09:23 -0500 Subject: [PATCH 628/648] Update mdbook --- ci/install.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install.sh b/ci/install.sh index 7c730cae3..be64d1ee8 100755 --- a/ci/install.sh +++ b/ci/install.sh @@ -21,4 +21,4 @@ function cargo_install() { } cargo_install mdbook 0.3.0 -cargo_install mdbook-linkcheck 0.3.0 +cargo_install mdbook-linkcheck 0.3.1 From 7f7d1d8eaebeed349a8796441897271c84fa4b90 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Fri, 19 Jul 2019 11:23:56 -0500 Subject: [PATCH 629/648] Update install.sh --- ci/install.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install.sh b/ci/install.sh index be64d1ee8..48df957f9 100755 --- a/ci/install.sh +++ b/ci/install.sh @@ -20,5 +20,5 @@ function cargo_install() { fi } -cargo_install mdbook 0.3.0 +cargo_install mdbook 0.3.1 cargo_install mdbook-linkcheck 0.3.1 From c20647e97819a8000129b9ef395658493d74278c Mon Sep 17 00:00:00 2001 From: Ilija Tovilo Date: Sat, 20 Jul 2019 20:12:03 +0200 Subject: [PATCH 630/648] Fix typo --- src/queries/query-evaluation-model-in-detail.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/queries/query-evaluation-model-in-detail.md b/src/queries/query-evaluation-model-in-detail.md index 7772f36be..a1fbed2e5 100644 --- a/src/queries/query-evaluation-model-in-detail.md +++ b/src/queries/query-evaluation-model-in-detail.md @@ -73,7 +73,7 @@ not memoize the result. When the query context is created, it is still empty: No queries have been executed, no results are cached. But the context already provides access to -"input" data, i.e. pieces of immutable data that where computed before the +"input" data, i.e. pieces of immutable data that were computed before the context was created and that queries can access to do their computations. Currently this input data consists mainly of the HIR map and the command-line options the compiler was invoked with. In the future, inputs will just consist From 18c466252e3b321ca5bc35ff92440e65c476e12b Mon Sep 17 00:00:00 2001 From: Ilija Tovilo Date: Sun, 21 Jul 2019 23:47:05 +0200 Subject: [PATCH 631/648] Fix typo --- src/closure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/closure.md b/src/closure.md index 77f2b42c3..e73f2e168 100644 --- a/src/closure.md +++ b/src/closure.md @@ -140,7 +140,7 @@ declared in the file [`src/librustc/ty/mod.rs`][ty]. [upvar]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_typeck/check/upvar/index.html [ty]:https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/index.html -Before we go any further, let's discuss how we can examine the flow of coontrol through the rustc +Before we go any further, let's discuss how we can examine the flow of control through the rustc codebase. For closures specifically, set the `RUST_LOG` env variable as below and collect the output in a file: From 831a469050c23aaa1e578db5356dd46063f04f11 Mon Sep 17 00:00:00 2001 From: Ilija Tovilo Date: Sun, 21 Jul 2019 12:37:42 +0200 Subject: [PATCH 632/648] Fix macro syntax error --- src/macro-expansion.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/macro-expansion.md b/src/macro-expansion.md index c62301247..bb429282f 100644 --- a/src/macro-expansion.md +++ b/src/macro-expansion.md @@ -19,11 +19,11 @@ whenever we refer to the "example _definition_", we mean the following: macro_rules! printer { (print $mvar:ident) => { println!("{}", $mvar); - } + }; (print twice $mvar:ident) => { println!("{}", $mvar); println!("{}", $mvar); - } + }; } ``` From d8af1f7fff3881eef5b2e4e29995073a85b67b5d Mon Sep 17 00:00:00 2001 From: Aleksey Kladov Date: Sun, 21 Jul 2019 14:33:57 +0300 Subject: [PATCH 633/648] write a short paragraph about the new lexer --- src/the-parser.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/src/the-parser.md b/src/the-parser.md index ac902d915..5796ae40e 100644 --- a/src/the-parser.md +++ b/src/the-parser.md @@ -32,6 +32,19 @@ To minimise the amount of copying that is done, both the `StringReader` and `Parser` have lifetimes which bind them to the parent `ParseSess`. This contains all the information needed while parsing, as well as the `SourceMap` itself. +## More on Lexical Analysis + +Code for lexical analysis is split between two crates: + +- `rustc_lexer` crate is responsible for breaking a `&str` into chunks + constituting tokens. Although it is popular to implement lexers as generated + finite state machines, the lexer in `rustc_lexer` is hand-written. + +- [`StringReader`] from [libsyntax] integrates `rustc_lexer` with `rustc` + specific data structures. Specifically, it adds `Span` information to tokens + returned by `rustc_lexer` and interns identifiers. + + [libsyntax]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/index.html [rustc_errors]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html [ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree From 093e1f0169855043f3cf2bc45dfba066dd9e0ffe Mon Sep 17 00:00:00 2001 From: Santiago Pastorino Date: Wed, 24 Jul 2019 18:03:40 -0300 Subject: [PATCH 634/648] Place is now a struct on nightly --- src/borrow_check/moves_and_initialization/move_paths.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/borrow_check/moves_and_initialization/move_paths.md b/src/borrow_check/moves_and_initialization/move_paths.md index d733e8ac7..157a9eaf3 100644 --- a/src/borrow_check/moves_and_initialization/move_paths.md +++ b/src/borrow_check/moves_and_initialization/move_paths.md @@ -28,7 +28,7 @@ they are indexed in ways that enable us to do move analysis more efficiently. [`MovePath`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/move_paths/struct.MovePath.html -[`Place`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/enum.Place.html +[`Place`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/struct.Place.html ## Move path indices @@ -122,5 +122,5 @@ which checks whether the dataflow results contain a value for the given move-path (e.g., `a.b`) or any child of that move-path (e.g., `a.b.c`). -[`Place`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/enum.Place.html +[`Place`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/mir/struct.Place.html [`has_any_child_of`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/dataflow/at_location/struct.FlowAtLocation.html#method.has_any_child_of From b4b7dca0d244f4496256d13566d972dd69e92e5f Mon Sep 17 00:00:00 2001 From: Brian Foley Date: Thu, 25 Jul 2019 15:25:26 -0700 Subject: [PATCH 635/648] Fix miscellaneous spelling typos. (#397) --- src/SUMMARY.md | 2 +- src/appendix/glossary.md | 2 +- src/borrow_check/region_inference.md | 2 +- src/closure.md | 2 +- src/codegen/updating-llvm.md | 2 +- src/how-to-build-and-run.md | 4 ++-- src/kinds.md | 2 +- src/traits/goals-and-clauses.md | 2 +- src/traits/implied-bounds.md | 2 +- 9 files changed, 10 insertions(+), 10 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 60aa8766d..75e1fcdc2 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -81,7 +81,7 @@ - [Member constraints](./borrow_check/region_inference/member_constraints.md) - [Placeholders and universes][pau] - [Closure constraints](./borrow_check/region_inference/closure_constraints.md) - - [Errror reporting](./borrow_check/region_inference/error_reporting.md) + - [Error reporting](./borrow_check/region_inference/error_reporting.md) - [Two-phase-borrows](./borrow_check/two_phase_borrows.md) - [Constant evaluation](./const-eval.md) - [miri const evaluator](./miri.md) diff --git a/src/appendix/glossary.md b/src/appendix/glossary.md index 007b1bb4c..3375d035f 100644 --- a/src/appendix/glossary.md +++ b/src/appendix/glossary.md @@ -8,7 +8,7 @@ Term | Meaning ------------------------|-------- AST | the abstract syntax tree produced by the syntax crate; reflects user syntax very closely. binder | a "binder" is a place where a variable or type is declared; for example, the `` is a binder for the generic type parameter `T` in `fn foo(..)`, and \|`a`\|` ...` is a binder for the parameter `a`. See [the background chapter for more](./background.html#free-vs-bound) -bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expession \|`a`\|` a * 2`. See [the background chapter for more](./background.html#free-vs-bound) +bound variable | a "bound variable" is one that is declared within an expression/term. For example, the variable `a` is bound within the closure expression \|`a`\|` a * 2`. See [the background chapter for more](./background.html#free-vs-bound) codegen | the code to translate MIR into LLVM IR. codegen unit | when we produce LLVM IR, we group the Rust code into a number of codegen units. Each of these units is processed by LLVM independently from one another, enabling parallelism. They are also the unit of incremental re-use. completeness | completeness is a technical term in type theory. Completeness means that every type-safe program also type-checks. Having both soundness and completeness is very hard, and usually soundness is more important. (see "soundness"). diff --git a/src/borrow_check/region_inference.md b/src/borrow_check/region_inference.md index 52659e93c..216128ffb 100644 --- a/src/borrow_check/region_inference.md +++ b/src/borrow_check/region_inference.md @@ -217,7 +217,7 @@ Here are some of the fields of the struct: - [`type_tests`]: contains some constraints on types that we must check after inference (e.g. `T: 'a`). - [`closure_bounds_mapping`]: used for propagating region constraints from - closures back out to the creater of the closure. + closures back out to the creator of the closure. [`constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#structfield.constraints [`liveness_constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/region_infer/struct.RegionInferenceContext.html#structfield.liveness_constraints diff --git a/src/closure.md b/src/closure.md index e73f2e168..c47ad9bc8 100644 --- a/src/closure.md +++ b/src/closure.md @@ -121,7 +121,7 @@ the *free variables* meaning they are not bound to the context of the closure. Other than lazy invocation, one other thing that the distinguishes a closure from a normal function is that it can use the upvars. It borrows these upvars from its surrounding -context; therfore the compiler has to determine the upvar's borrow type. The compiler starts with +context; therefore the compiler has to determine the upvar's borrow type. The compiler starts with assigning an immutable borrow type and lowers the restriction (that is, changes it from **immutable** to **mutable** to **move**) as needed, based on the usage. In the Example 1 above, the closure only uses the variable for printing but does not modify it in any way and therefore, in the diff --git a/src/codegen/updating-llvm.md b/src/codegen/updating-llvm.md index b26c475f7..50dc94962 100644 --- a/src/codegen/updating-llvm.md +++ b/src/codegen/updating-llvm.md @@ -24,7 +24,7 @@ another: them upstream in LLVM. We'll want to pull fixes back to the compiler itself as they're merged upstream. -* Second, a new feature may be avaiable in LLVM that we want to use in rustc, +* Second, a new feature may be available in LLVM that we want to use in rustc, but we don't want to wait for a full LLVM release to test it out. Each of these reasons has a different strategy for updating LLVM, and we'll go diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 84efffc84..c3968b591 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -495,12 +495,12 @@ One of the challenges with rustc is that the RLS can't handle it, since it's a bootstrapping compiler. This makes code navigation difficult. One solution is to use `ctags`. -`ctags` has a long history and several variants. Exhuberant CTags seems to be +`ctags` has a long history and several variants. Exuberant Ctags seems to be quite commonly distributed but it does not have out-of-box Rust support. Some distributions seem to use [Universal Ctags][utags], which is a maintained fork and does have built-in Rust support. -The following script can be used to set up Exhuberant Ctags: +The following script can be used to set up Exuberant Ctags: [https://github.com/nikomatsakis/rust-etags][etags]. `ctags` integrates into emacs and vim quite easily. The following can then be diff --git a/src/kinds.md b/src/kinds.md index ac6a2d35b..e4ba72995 100644 --- a/src/kinds.md +++ b/src/kinds.md @@ -30,7 +30,7 @@ make use of the safe [`UnpackedKind`](#unpackedkind) abstraction. As `Kind` itself is not type-safe, the `UnpackedKind` enum provides a more convenient and safe interface for dealing with kinds. An `UnpackedKind` can be converted to a raw `Kind` using `Kind::from()` (or simply `.into()` when -the context is clear). As mentioned earlier, substition lists store raw +the context is clear). As mentioned earlier, substitution lists store raw `Kind`s, so before dealing with them, it is preferable to convert them to `UnpackedKind`s first. This is done by calling the `.unpack()` method. diff --git a/src/traits/goals-and-clauses.md b/src/traits/goals-and-clauses.md index 8f1ffa488..a653528f4 100644 --- a/src/traits/goals-and-clauses.md +++ b/src/traits/goals-and-clauses.md @@ -53,7 +53,7 @@ In terms of code, these types are defined in definitions given above, general goals basically consist in a combination of domain goals. -Moreover, flattenning a bit the definition of clauses given previously, one can +Moreover, flattening a bit the definition of clauses given previously, one can see that clauses are always of the form: ```text forall { DomainGoal :- Goal } diff --git a/src/traits/implied-bounds.md b/src/traits/implied-bounds.md index f32c9d0cb..5876f3b62 100644 --- a/src/traits/implied-bounds.md +++ b/src/traits/implied-bounds.md @@ -81,7 +81,7 @@ The rationale for implied bounds for traits is that if a type implements `Copy`, that is, if there exists an `impl Copy` for that type, there *ought* to exist an `impl Clone` for that type, otherwise the compiler would have reported an error in the first place. So again, if we were forced to repeat the -additionnal `where SomeType: Clone` everywhere whereas we already know that +additional `where SomeType: Clone` everywhere whereas we already know that `SomeType: Copy` hold, we would kind of duplicate the verification work. Implied bounds are not yet completely enforced in rustc, at the moment it only From 588af16689f4b54c0160b24ee31ebd6e94ed5267 Mon Sep 17 00:00:00 2001 From: Santiago Pastorino Date: Thu, 1 Aug 2019 15:30:45 -0300 Subject: [PATCH 636/648] Fix experts map link --- src/compiler-team.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/compiler-team.md b/src/compiler-team.md index f01b4a8d5..1dbeeb422 100644 --- a/src/compiler-team.md +++ b/src/compiler-team.md @@ -22,7 +22,7 @@ check out our [experts directory][experts]. It contains a listing of the various parts of the compiler and a list of people who are experts on each one. -[experts]: https://github.com/rust-lang/compiler-team/blob/master/content/docs/experts/map.toml +[experts]: https://github.com/rust-lang/compiler-team/blob/master/content/experts/map.toml ## Rust compiler meeting From 69c17d9a5edbfb2362bdc1ed1c9829f76a00097e Mon Sep 17 00:00:00 2001 From: Bruce Mitchener Date: Fri, 2 Aug 2019 01:47:43 +0700 Subject: [PATCH 637/648] Fix broken links. --- src/appendix/humorust.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/appendix/humorust.md b/src/appendix/humorust.md index 36c25c890..ef926c1c6 100644 --- a/src/appendix/humorust.md +++ b/src/appendix/humorust.md @@ -3,7 +3,7 @@ What's a project without a sense of humor? And frankly some of these are enlightening? -- [Weird exprs test](https://github.com/rust-lang/rust/blob/master/src/test/run-pass/weird-exprs.rs) +- [Weird exprs test](https://github.com/rust-lang/rust/blob/master/src/test/ui/weird-exprs.rs) - [Ferris Rap](http://fitzgeraldnick.com/2018/12/13/rust-raps.html) - [The Genesis of Generic Germination](https://github.com/rust-lang/rust/pull/53645#issue-210543221) - [The Bastion of the Turbofish test](https://github.com/rust-lang/rust/blob/79d8a0fcefa5134db2a94739b1d18daa01fc6e9f/src/test/ui/bastion-of-the-turbofish.rs) From 442aed9afe80511685cab6259b5a7aad43f78a49 Mon Sep 17 00:00:00 2001 From: Bruce Mitchener Date: Fri, 2 Aug 2019 01:16:56 +0700 Subject: [PATCH 638/648] Fix typos. --- src/debugging-support-in-rustc.md | 2 +- src/kinds.md | 2 +- src/miri.md | 2 +- src/stabilization_guide.md | 6 +++--- src/ty.md | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/src/debugging-support-in-rustc.md b/src/debugging-support-in-rustc.md index 1775b07af..3d08cc825 100644 --- a/src/debugging-support-in-rustc.md +++ b/src/debugging-support-in-rustc.md @@ -1,7 +1,7 @@ # Debugging support in the Rust compiler This document explains the state of debugging tools support in the Rust compiler (rustc). -The document gives an overview of debugging tools like GDB, LLDB etc. and infrastrcture +The document gives an overview of debugging tools like GDB, LLDB etc. and infrastructure around Rust compiler to debug Rust code. If you want to learn how to debug the Rust compiler itself, then you must see [Debugging the Compiler] page. diff --git a/src/kinds.md b/src/kinds.md index e4ba72995..194d2ee2f 100644 --- a/src/kinds.md +++ b/src/kinds.md @@ -15,7 +15,7 @@ For example, given a `HashMap` with two type parameters, `K` and `V`, an instantiation of the parameters, for example `HashMap`, would be represented by the substitution `&'tcx [tcx.types.i32, tcx.types.u32]`. -`Subst` provides various convenience methods to instantiant substitutions +`Subst` provides various convenience methods to instantiate substitutions given item definitions, which should generally be used rather than explicitly constructing such substitution slices. diff --git a/src/miri.md b/src/miri.md index d361103f2..aee2376f1 100644 --- a/src/miri.md +++ b/src/miri.md @@ -131,7 +131,7 @@ evaluation of another constant simply calls `tcx.const_eval`, which produces an entirely new and independent stack frame. The frames are just a `Vec`, there's no way to actually refer to a -`Frame`'s memory even if horrible shenigans are done via unsafe code. The only +`Frame`'s memory even if horrible shenanigans are done via unsafe code. The only memory that can be referred to are `Allocation`s. Miri now calls the `step` method (in diff --git a/src/stabilization_guide.md b/src/stabilization_guide.md index 76446150c..302be41d0 100644 --- a/src/stabilization_guide.md +++ b/src/stabilization_guide.md @@ -34,8 +34,8 @@ Places that may need updated documentation: to the language, updating examples is important. - [Rust by Example]: As needed. -Prepare PRs to update documentations invovling this new feature -for repositories mentioned above. Maintainers of these repositories +Prepare PRs to update documentation involving this new feature +for repositories mentioned above. Maintainers of these repositories will keep these PRs open until the whole stabilization process has completed. Meanwhile, we can proceed to the next step. @@ -44,7 +44,7 @@ has completed. Meanwhile, we can proceed to the next step. Find the tracking issue of the feature, and create a short stabilization report. Essentially this would be a brief summary of the feature plus some links to test cases showing it works -as expected, along with a list of edge cases that came up and +as expected, along with a list of edge cases that came up and were considered. This is a minimal "due diligence" that we do before stabilizing. diff --git a/src/ty.md b/src/ty.md index 72405d297..811f473c9 100644 --- a/src/ty.md +++ b/src/ty.md @@ -64,7 +64,7 @@ defining all of the different kinds of types in the compiler. > known later.). To allocate a new type, you can use the various `mk_` methods defined -on the `tcx`. These have names that correpond mostly to the various kinds +on the `tcx`. These have names that correspond mostly to the various kinds of type variants. For example: ```rust,ignore From bfe0a08ea4648a08cbc9cf61458d920c90826424 Mon Sep 17 00:00:00 2001 From: Tshepang Lekhonkhobe Date: Fri, 2 Aug 2019 04:25:07 +0200 Subject: [PATCH 639/648] run-pass dir is gone (#409) Fixes #405 --- src/compiletest.md | 4 ++-- src/how-to-build-and-run.md | 2 +- src/tests/adding.md | 15 ++++++--------- src/tests/running.md | 8 ++++---- 4 files changed, 13 insertions(+), 16 deletions(-) diff --git a/src/compiletest.md b/src/compiletest.md index 3cfb943b0..1b424e1a2 100644 --- a/src/compiletest.md +++ b/src/compiletest.md @@ -16,8 +16,8 @@ expect, and more. If you are unfamiliar with the compiler testing framework, see [this chapter](./tests/intro.html) for additional background. The tests themselves are typically (but not always) organized into -"suites" – for example, `run-pass`, a folder representing tests that should -succeed, `run-fail`, a folder holding tests that should compile successfully, +"suites" – for example, `run-fail`, +a folder holding tests that should compile successfully, but return a failure (non-zero status), `compile-fail`, a folder holding tests that should fail to compile, and many more. The various suites are defined in [src/tools/compiletest/src/common.rs][common] in the `pub struct Config` diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index c3968b591..0a746177b 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -483,7 +483,7 @@ in other sections: - Running tests (see the [section on running tests](./tests/running.html) for more details): - `./x.py test --stage 1 src/libstd` – runs the `#[test]` tests from libstd - - `./x.py test --stage 1 src/test/run-pass` – runs the `run-pass` test suite + - `./x.py test --stage 1 src/test/ui` – runs the `ui` test suite - `./x.py test --stage 1 src/test/ui/const-generics` - runs all the tests in the `const-generics/` subdirectory of the `ui` test suite - `./x.py test --stage 1 src/test/ui/const-generics/const-types.rs` - runs diff --git a/src/tests/adding.md b/src/tests/adding.md index d88b7cf57..a660d26b5 100644 --- a/src/tests/adding.md +++ b/src/tests/adding.md @@ -43,14 +43,14 @@ rough heuristics: We have not traditionally had a lot of structure in the names of tests. Moreover, for a long time, the rustc test runner did not support subdirectories (it now does), so test suites like -[`src/test/run-pass`] have a huge mess of files in them. This is not +[`src/test/ui`] have a huge mess of files in them. This is not considered an ideal setup. -[`src/test/run-pass`]: https://github.com/rust-lang/rust/tree/master/src/test/run-pass/ +[`src/test/ui`]: https://github.com/rust-lang/rust/tree/master/src/test/ui/ For regression tests – basically, some random snippet of code that came in from the internet – we often just name the test after the -issue. For example, `src/test/run-pass/issue-12345.rs`. If possible, +issue. For example, `src/test/ui/issue-12345.rs`. If possible, though, it is better if you can put the test into a directory that helps identify what piece of code is being tested here (e.g., `borrowck/issue-12345.rs` is much better), or perhaps give it a more @@ -58,11 +58,8 @@ meaningful name. Still, **do include the issue number somewhere**. When writing a new feature, **create a subdirectory to store your tests**. For example, if you are implementing RFC 1234 ("Widgets"), -then it might make sense to put the tests in directories like: - -- `src/test/ui/rfc1234-widgets/` -- `src/test/run-pass/rfc1234-widgets/` -- etc +then it might make sense to put the tests in a directory like +`src/test/ui/rfc1234-widgets/`. In other cases, there may already be a suitable directory. (The proper directory structure to use is actually an area of active debate.) @@ -216,7 +213,7 @@ The error levels that you can have are: ## Revisions Certain classes of tests support "revisions" (as of the time of this -writing, this includes run-pass, compile-fail, run-fail, and +writing, this includes compile-fail, run-fail, and incremental, though incremental tests are somewhat different). Revisions allow a single test file to be used for multiple tests. This is done by adding a special header at the top of the file: diff --git a/src/tests/running.md b/src/tests/running.md index 01d308f0b..ef3227f9a 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -34,10 +34,10 @@ test" that can be used after modifying rustc to see if things are generally working correctly would be the following: ```bash -> ./x.py test --stage 1 src/test/{ui,compile-fail,run-pass} +> ./x.py test --stage 1 src/test/{ui,compile-fail} ``` -This will run the `ui`, `compile-fail`, and `run-pass` test suites, +This will run the `ui` and `compile-fail` test suites, and only with the stage 1 build. Of course, the choice of test suites is somewhat arbitrary, and may not suit the task you are doing. For example, if you are hacking on debuginfo, you may be better off with @@ -114,10 +114,10 @@ Pass UI tests now have three modes, `check-pass`, `build-pass` and `run-pass`. When `--pass $mode` is passed, these tests will be forced to run under the given `$mode` unless the directive `// ignore-pass` exists in the test file. For example, you can run all the tests in -`src/test/run-pass` as `check-pass`: +`src/test/ui` as `check-pass`: ```bash -> ./x.py test --stage 1 src/test/run-pass --pass check +> ./x.py test --stage 1 src/test/ui --pass check ``` By passing `--pass $mode`, you can reduce the testing time. For each From d420542602d623497778880ec63561b477dc0756 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Mon, 5 Aug 2019 15:57:25 -0500 Subject: [PATCH 640/648] hack around timeouts (#416) --- .travis.yml | 4 ++-- ci/build-ignore-timeouts.sh | 23 +++++++++++++++++++++++ 2 files changed, 25 insertions(+), 2 deletions(-) create mode 100644 ci/build-ignore-timeouts.sh diff --git a/.travis.yml b/.travis.yml index fc6f4f05e..bd72a03fc 100644 --- a/.travis.yml +++ b/.travis.yml @@ -8,9 +8,9 @@ before_install: - MAX_LINE_LENGTH=100 bash ci/check_line_lengths.sh src/**/*.md install: - source ~/.cargo/env || true -- bash ci/install.sh +- bash -x ci/install.sh script: -- mdbook build +- bash -x ci/build-ignore-timeouts.sh - mdbook test notifications: email: diff --git a/ci/build-ignore-timeouts.sh b/ci/build-ignore-timeouts.sh new file mode 100644 index 000000000..b21ce24e8 --- /dev/null +++ b/ci/build-ignore-timeouts.sh @@ -0,0 +1,23 @@ + +output=$(mktemp) + +RUST_LOG=mdbook_linkcheck=debug mdbook build 2>&1 | tee $output + +result=${PIPESTATUS[0]} + +# if passed, great! +if [ "$result" -eq "0" ] ; then + exit 0 ; +fi + +errors=$(cat $output | sed -n 's/There \(was\|were\) \([0-9]\+\).*$/\2/p') +timeouts=$(cat $output | grep "error while fetching" | wc -l) + +# if all errors are timeouts, ignore them... +if [ "$errors" -eq "$timeouts" ] ; then + echo "Ignoring $timeouts timeouts"; + exit 0; +else + echo "Non-timeout errors found"; + exit 1; +fi From 84aa3362e6271b996fd557d5a2755b371e38b6cf Mon Sep 17 00:00:00 2001 From: Tshepang Lekhonkhobe Date: Mon, 5 Aug 2019 23:03:34 +0200 Subject: [PATCH 641/648] a colon there is unusual (#415) --- src/how-to-build-and-run.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 0a746177b..eb14e8fad 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -434,7 +434,7 @@ whether the compiler builds is not enough. A common example is that you need to add a `debug!` statement to inspect the value of some state or better understand the problem. In that case, you really need a full build. By leveraging incremental, though, you can often get -these builds to complete very fast (e.g., around 30 seconds): the only +these builds to complete very fast (e.g., around 30 seconds). The only catch is this requires a bit of fudging and may produce compilers that don't work (but that is easily detected and fixed). From b94deb1afea33a15d79cf2944ba7458ce9c42926 Mon Sep 17 00:00:00 2001 From: Tshepang Lekhonkhobe Date: Mon, 5 Aug 2019 23:06:51 +0200 Subject: [PATCH 642/648] fix typo (#412) --- src/how-to-build-and-run.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index eb14e8fad..8a803e66c 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -160,7 +160,7 @@ The following tables indicate the outputs of various stage actions: | `stage0` builds `rustc` with `stage0-sysroot` | `build/HOST/stage0-rustc/HOST` | | copy `stage0-rustc (except executable)` | `build/HOST/stage0-sysroot/lib/rustlib/HOST` | | build `llvm` | `build/HOST/llvm` | -| `stage0` builds `codegen` with `stage0-sysroot` | `build/HOST/stage0-codgen/HOST` | +| `stage0` builds `codegen` with `stage0-sysroot` | `build/HOST/stage0-codegen/HOST` | | `stage0` builds `rustdoc` with `stage0-sysroot` | `build/HOST/stage0-tools/HOST` | `--stage=0` stops here. From 4ee7cdaea47119f7d53f10df7a4be273f51dea2c Mon Sep 17 00:00:00 2001 From: Tshepang Lekhonkhobe Date: Mon, 5 Aug 2019 23:07:05 +0200 Subject: [PATCH 643/648] update link (#411) --- src/compiler-team.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/compiler-team.md b/src/compiler-team.md index 1dbeeb422..d61137682 100644 --- a/src/compiler-team.md +++ b/src/compiler-team.md @@ -5,7 +5,7 @@ this team collectively work to track regressions and implement new features. Members of the Rust compiler team are people who have made significant contributions to rustc and its design. -[team]: https://www.rust-lang.org/governance/teams/language-and-compiler +[team]: https://www.rust-lang.org/governance/teams/compiler ## Discussion From 07dbcdec1888b2a5e9e78f4828c52417c49e0398 Mon Sep 17 00:00:00 2001 From: Tshepang Lekhonkhobe Date: Mon, 5 Aug 2019 23:07:17 +0200 Subject: [PATCH 644/648] fix awkward sentence structure (#410) --- src/about-this-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/about-this-guide.md b/src/about-this-guide.md index df060a208..48ea4dce6 100644 --- a/src/about-this-guide.md +++ b/src/about-this-guide.md @@ -3,7 +3,7 @@ This guide is meant to help document how rustc – the Rust compiler – works, as well as to help new contributors get involved in rustc development. It is not meant to replace code documentation – each -chapter gives only high-level details – the kinds of things that +chapter gives only high-level details, the kinds of things that (ideally) don't change frequently. There are three parts to this guide. Part 1 contains information that should From 6f4ba673ff9d4613e98415bc095347a6a0031e9c Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Tue, 6 Aug 2019 16:49:01 -0500 Subject: [PATCH 645/648] Remove leading new line --- ci/build-ignore-timeouts.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/ci/build-ignore-timeouts.sh b/ci/build-ignore-timeouts.sh index b21ce24e8..e777f349a 100644 --- a/ci/build-ignore-timeouts.sh +++ b/ci/build-ignore-timeouts.sh @@ -1,3 +1,4 @@ +#!/bin/bash -x output=$(mktemp) From b7f24940c06fff812c89198844335d6d9a65eabc Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Tue, 13 Aug 2019 09:56:02 -0500 Subject: [PATCH 646/648] fix broken link (#421) --- ci/build-ignore-timeouts.sh | 0 src/appendix/code-index.md | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) mode change 100644 => 100755 ci/build-ignore-timeouts.sh diff --git a/ci/build-ignore-timeouts.sh b/ci/build-ignore-timeouts.sh old mode 100644 new mode 100755 diff --git a/src/appendix/code-index.md b/src/appendix/code-index.md index 52a7f42d6..267a3a3ea 100644 --- a/src/appendix/code-index.md +++ b/src/appendix/code-index.md @@ -19,7 +19,7 @@ Item | Kind | Short description | Chapter | `ParamEnv` | struct | Information about generic parameters or `Self`, useful for working with associated or generic items | [Parameter Environment] | [src/librustc/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.ParamEnv.html) `ParseSess` | struct | This struct contains information about a parsing session | [The parser] | [src/libsyntax/parse/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/parse/struct.ParseSess.html) `Query` | struct | Represents the result of query to the `Compiler` interface and allows stealing, borrowing, and returning the results of compiler passes. | [The Rustc Driver and Interface] | [src/librustc_interface/queries.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/queries/struct.Query.html) -`Rib` | struct | Represents a single scope of names | [Name resolution] | [src/librustc_resolve/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/struct.Rib.html) +`Rib` | struct | Represents a single scope of names | [Name resolution] | [src/librustc_resolve/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/late/struct.Rib.html) `Session` | struct | The data associated with a compilation session | [The parser], [The Rustc Driver and Interface] | [src/librustc/session/mod.html](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/session/struct.Session.html) `SourceFile` | struct | Part of the `SourceMap`. Maps AST nodes to their source code for a single source file. Was previously called FileMap | [The parser] | [src/libsyntax_pos/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceFile.html) `SourceMap` | struct | Maps AST nodes to their source code. It is composed of `SourceFile`s. Was previously called CodeMap | [The parser] | [src/libsyntax/source_map.rs](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/source_map/struct.SourceMap.html) From 6b6e6cc3bfb312e39661db026343ee0be03cf26f Mon Sep 17 00:00:00 2001 From: Aleksey Kladov Date: Tue, 13 Aug 2019 18:20:22 +0300 Subject: [PATCH 647/648] document how to use system llvm --- src/how-to-build-and-run.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 8a803e66c..2c4d2a979 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -469,6 +469,18 @@ You can also use `--keep-stage 1` when running tests. Something like this: - Initial test run: `./x.py test -i --stage 1 src/test/ui` - Subsequent test run: `./x.py test -i --stage 1 src/test/ui --keep-stage 1` +### Building with system LLVM + +By default, LLVM is built from source, and that can take significant amount of time. +An alternative is to use LLVM already installed on your computer. + +This is specified in the `target` section of `config.toml`: + +```toml +[target.x86_64-unknown-linux-gnu] +llvm-config = "/path/to/llvm/llvm-7.0.1/bin/llvm-config" +``` + ### Other `x.py` commands Here are a few other useful `x.py` commands. We'll cover some of them in detail From 5919642a096a474f426214efaf65ee996df4f08a Mon Sep 17 00:00:00 2001 From: Paul Daniel Faria Date: Sat, 17 Aug 2019 13:19:10 -0400 Subject: [PATCH 648/648] Fix a typo in the implied bounds section of traits/lowering-rules --- src/traits/lowering-rules.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/traits/lowering-rules.md b/src/traits/lowering-rules.md index bbebf1450..c780e7cf5 100644 --- a/src/traits/lowering-rules.md +++ b/src/traits/lowering-rules.md @@ -114,7 +114,7 @@ cover). For each trait, we produce two clauses: // // For each where clause WC: forall { - FromEnv(WC) :- FromEnv(Self: Trait) } ```