Consolidate staging for `rustc_private` tools #144303

Kobzol · 2025-07-22T12:05:51Z

This PR continues bootstrap refactoring, this time by consolidating staging for Mode::ToolRustc tools. This refactoring was in the critical path of refactoring test/dist/clippy/doc steps, and getting rid of the rmeta/rlib sysroot copy, because tools are pervasive and they are being used for a lot of things in bootstrap.

The main idea is to explicitly model the fact that a stage N Mode::ToolRustc tool always works with two different compilers:

Stage N-1 rustc (build_compiler) builds stage N rustc (target_compiler)
Rlib artifacts from stage N rustc are copied to the sysroot of stage N-1 rustc
Stage N-1 rustc builds the (stage N) tool itself, the tool links to the rlib artifacts of the stage N rustc

Before, the code often used compiler, which meant sometimes the build compiler, sometimes the target compiler, and sometimes neither (looking at you, download-rustc). This is especially annoying when you get to a situation where you have an install step that invokes a dist step that invokes a tool build step, where some compiler is being propagated through, without it being clear what does that compiler represent. This refactoring hopefully makes that clearer and more explicit. It also gets rid of a few builder.ensure(Rustc(...)) calls within bootstrap, which is always nice.

Rustdoc needs to be handled a bit specially, because it acts as a compiler itself, I documented that in the changes.

It wasn't practical to do these refactorings in multiple PRs, so I did it all in one PR. The meat of the change is 9ee6d1c.

I tested manually that x build rustdoc and x build miri still works even with download-rustc, although I cannot promise any extra support for download-rustc, IMO we will just have to reimplement it from scratch in a different way.

As usually, I did some drive-by refactorings to bootstrap, trying to document and clarify things, add more step metadata and tests.

Since these changes broke Cargo, which was incorrectly using Mode::ToolRustc, I also changed cargo to ToolTarget in this PR.

Best reviewed commit-by-commit (note that I renamed link_compiler to target_compiler, in accordance to the rest of bootstrap, in the last commit).

r? @jieyouxu

try-job: x86_64-gnu-aux
try-job: x86_64-msvc-ext1

rustbot · 2025-07-22T12:05:57Z

jieyouxu is not on the review rotation at the moment.
They may take a while to respond.

bors · 2025-07-23T14:01:35Z

☔ The latest upstream changes (presumably #143412) made this pull request unmergeable. Please resolve the merge conflicts.

jieyouxu · 2025-07-25T09:03:03Z

Will start looking at this today, might take me some time to get through.

jieyouxu

Thanks, looks good overall. You can r=me after comment nit.

src/bootstrap/src/core/build_steps/tool.rs

src/bootstrap/src/core/builder/tests.rs

jieyouxu · 2025-07-28T09:59:32Z

@rustbot author

Kobzol · 2025-07-28T10:08:24Z

@bors r=jieyouxu

bors · 2025-07-28T10:08:26Z

📌 Commit 7bf4a1a has been approved by jieyouxu

It is now in the queue for this repository.

…ieyouxu Consolidate staging for `rustc_private` tools This PR continues bootstrap refactoring, this time by consolidating staging for `Mode::ToolRustc` tools. This refactoring was in the critical path of refactoring `test`/`dist`/`clippy`/`doc` steps, and getting rid of the rmeta/rlib sysroot copy, because tools are pervasive and they are being used for a lot of things in bootstrap. The main idea is to explicitly model the fact that a stage N `Mode::ToolRustc` tool always works with two different compilers: - Stage N-1 rustc (`build_compiler`) builds stage N rustc (`target_compiler`) - Rlib artifacts from stage N rustc are copied to the sysroot of stage N-1 rustc - Stage N-1 rustc builds the (stage N) tool itself, the tool links to the rlib artifacts of the stage N rustc Before, the code often used `compiler`, which meant sometimes the build compiler, sometimes the target compiler, and sometimes neither (looking at you, `download-rustc`). This is especially annoying when you get to a situation where you have an install step that invokes a dist step that invokes a tool build step, where *some* compiler is being propagated through, without it being clear what does that compiler represent. This refactoring hopefully makes that clearer and more explicit. It also gets rid of a few `builder.ensure(Rustc(...))` calls within bootstrap, which is always nice. `Rustdoc` needs to be handled a bit specially, because it acts as a compiler itself, I documented that in the changes. It wasn't practical to do these refactorings in multiple PRs, so I did it all in one PR. The meat of the change is 9ee6d1c. I tested manually that `x build rustdoc` and `x build miri` still works even with `download-rustc`, although I cannot promise any extra support for `download-rustc`, IMO we will just have to reimplement it from scratch in a different way. As usually, I did some drive-by refactorings to bootstrap, trying to document and clarify things, add more step metadata and tests. Best reviewed commit-by-commit (note that I renamed `link_compiler` to `target_compiler`, in accordance to the rest of bootstrap, in the last commit). r? `@jieyouxu`

Rollup of 11 pull requests Successful merges: - #143289 (Remove `[T]::array_chunks(_mut)`) - #143883 (Add `--link-targets-dir` argument to linkchecker) - #144034 (tests: Test line number in debuginfo for diverging function calls) - #144236 (Add `core::mem::DropGuard`) - #144268 (Add method `find_ancestor_not_from_macro` and `find_ancestor_not_from_extern_macro` to supersede `find_oldest_ancestor_in_same_ctxt`) - #144303 (Consolidate staging for `rustc_private` tools) - #144539 (constify with_exposed_provenance) - #144569 (rustc-dev-guide subtree update) - #144573 (Raw Pointers are Constant PatKinds too) - #144578 (Ensure correct aligement of rustc_hir::Lifetime on platforms with lower default alignments.) - #144582 (fix `Atomic*::as_ptr` wording) r? `@ghost` `@rustbot` modify labels: rollup

…ieyouxu Consolidate staging for `rustc_private` tools This PR continues bootstrap refactoring, this time by consolidating staging for `Mode::ToolRustc` tools. This refactoring was in the critical path of refactoring `test`/`dist`/`clippy`/`doc` steps, and getting rid of the rmeta/rlib sysroot copy, because tools are pervasive and they are being used for a lot of things in bootstrap. The main idea is to explicitly model the fact that a stage N `Mode::ToolRustc` tool always works with two different compilers: - Stage N-1 rustc (`build_compiler`) builds stage N rustc (`target_compiler`) - Rlib artifacts from stage N rustc are copied to the sysroot of stage N-1 rustc - Stage N-1 rustc builds the (stage N) tool itself, the tool links to the rlib artifacts of the stage N rustc Before, the code often used `compiler`, which meant sometimes the build compiler, sometimes the target compiler, and sometimes neither (looking at you, `download-rustc`). This is especially annoying when you get to a situation where you have an install step that invokes a dist step that invokes a tool build step, where *some* compiler is being propagated through, without it being clear what does that compiler represent. This refactoring hopefully makes that clearer and more explicit. It also gets rid of a few `builder.ensure(Rustc(...))` calls within bootstrap, which is always nice. `Rustdoc` needs to be handled a bit specially, because it acts as a compiler itself, I documented that in the changes. It wasn't practical to do these refactorings in multiple PRs, so I did it all in one PR. The meat of the change is 9ee6d1c. I tested manually that `x build rustdoc` and `x build miri` still works even with `download-rustc`, although I cannot promise any extra support for `download-rustc`, IMO we will just have to reimplement it from scratch in a different way. As usually, I did some drive-by refactorings to bootstrap, trying to document and clarify things, add more step metadata and tests. Best reviewed commit-by-commit (note that I renamed `link_compiler` to `target_compiler`, in accordance to the rest of bootstrap, in the last commit). r? ``@jieyouxu``

…ieyouxu Consolidate staging for `rustc_private` tools This PR continues bootstrap refactoring, this time by consolidating staging for `Mode::ToolRustc` tools. This refactoring was in the critical path of refactoring `test`/`dist`/`clippy`/`doc` steps, and getting rid of the rmeta/rlib sysroot copy, because tools are pervasive and they are being used for a lot of things in bootstrap. The main idea is to explicitly model the fact that a stage N `Mode::ToolRustc` tool always works with two different compilers: - Stage N-1 rustc (`build_compiler`) builds stage N rustc (`target_compiler`) - Rlib artifacts from stage N rustc are copied to the sysroot of stage N-1 rustc - Stage N-1 rustc builds the (stage N) tool itself, the tool links to the rlib artifacts of the stage N rustc Before, the code often used `compiler`, which meant sometimes the build compiler, sometimes the target compiler, and sometimes neither (looking at you, `download-rustc`). This is especially annoying when you get to a situation where you have an install step that invokes a dist step that invokes a tool build step, where *some* compiler is being propagated through, without it being clear what does that compiler represent. This refactoring hopefully makes that clearer and more explicit. It also gets rid of a few `builder.ensure(Rustc(...))` calls within bootstrap, which is always nice. `Rustdoc` needs to be handled a bit specially, because it acts as a compiler itself, I documented that in the changes. It wasn't practical to do these refactorings in multiple PRs, so I did it all in one PR. The meat of the change is 9ee6d1c. I tested manually that `x build rustdoc` and `x build miri` still works even with `download-rustc`, although I cannot promise any extra support for `download-rustc`, IMO we will just have to reimplement it from scratch in a different way. As usually, I did some drive-by refactorings to bootstrap, trying to document and clarify things, add more step metadata and tests. Best reviewed commit-by-commit (note that I renamed `link_compiler` to `target_compiler`, in accordance to the rest of bootstrap, in the last commit). r? ```@jieyouxu```

…ieyouxu Consolidate staging for `rustc_private` tools This PR continues bootstrap refactoring, this time by consolidating staging for `Mode::ToolRustc` tools. This refactoring was in the critical path of refactoring `test`/`dist`/`clippy`/`doc` steps, and getting rid of the rmeta/rlib sysroot copy, because tools are pervasive and they are being used for a lot of things in bootstrap. The main idea is to explicitly model the fact that a stage N `Mode::ToolRustc` tool always works with two different compilers: - Stage N-1 rustc (`build_compiler`) builds stage N rustc (`target_compiler`) - Rlib artifacts from stage N rustc are copied to the sysroot of stage N-1 rustc - Stage N-1 rustc builds the (stage N) tool itself, the tool links to the rlib artifacts of the stage N rustc Before, the code often used `compiler`, which meant sometimes the build compiler, sometimes the target compiler, and sometimes neither (looking at you, `download-rustc`). This is especially annoying when you get to a situation where you have an install step that invokes a dist step that invokes a tool build step, where *some* compiler is being propagated through, without it being clear what does that compiler represent. This refactoring hopefully makes that clearer and more explicit. It also gets rid of a few `builder.ensure(Rustc(...))` calls within bootstrap, which is always nice. `Rustdoc` needs to be handled a bit specially, because it acts as a compiler itself, I documented that in the changes. It wasn't practical to do these refactorings in multiple PRs, so I did it all in one PR. The meat of the change is 9ee6d1c. I tested manually that `x build rustdoc` and `x build miri` still works even with `download-rustc`, although I cannot promise any extra support for `download-rustc`, IMO we will just have to reimplement it from scratch in a different way. As usually, I did some drive-by refactorings to bootstrap, trying to document and clarify things, add more step metadata and tests. Best reviewed commit-by-commit (note that I renamed `link_compiler` to `target_compiler`, in accordance to the rest of bootstrap, in the last commit). r? ````@jieyouxu````

Rollup of 11 pull requests Successful merges: - #143883 (Add `--link-targets-dir` argument to linkchecker) - #144236 (Add `core::mem::DropGuard`) - #144303 (Consolidate staging for `rustc_private` tools) - #144367 (Move dist-apple-various from x86_64 to aarch64) - #144539 (constify with_exposed_provenance) - #144569 (rustc-dev-guide subtree update) - #144573 (Raw Pointers are Constant PatKinds too) - #144575 (fixed typo chunks->as_chunks) - #144578 (Ensure correct aligement of rustc_hir::Lifetime on platforms with lower default alignments.) - #144582 (fix `Atomic*::as_ptr` wording) - #144616 (coverage: Regression test for "function name is empty" bug) r? `@ghost` `@rustbot` modify labels: rollup

Zalathar · 2025-07-29T06:14:17Z

Possibly failed in rollup? #144622 (comment)

@bors r-
@bors try jobs=x86_64-gnu-aux

jieyouxu · 2025-08-01T13:52:00Z

@bors r+

bors · 2025-08-01T13:52:02Z

📌 Commit c46f42d has been approved by jieyouxu

It is now in the queue for this repository.

bors · 2025-08-01T17:48:15Z

⌛ Testing commit c46f42d with merge e4677ed...

Consolidate staging for `rustc_private` tools This PR continues bootstrap refactoring, this time by consolidating staging for `Mode::ToolRustc` tools. This refactoring was in the critical path of refactoring `test`/`dist`/`clippy`/`doc` steps, and getting rid of the rmeta/rlib sysroot copy, because tools are pervasive and they are being used for a lot of things in bootstrap. The main idea is to explicitly model the fact that a stage N `Mode::ToolRustc` tool always works with two different compilers: - Stage N-1 rustc (`build_compiler`) builds stage N rustc (`target_compiler`) - Rlib artifacts from stage N rustc are copied to the sysroot of stage N-1 rustc - Stage N-1 rustc builds the (stage N) tool itself, the tool links to the rlib artifacts of the stage N rustc Before, the code often used `compiler`, which meant sometimes the build compiler, sometimes the target compiler, and sometimes neither (looking at you, `download-rustc`). This is especially annoying when you get to a situation where you have an install step that invokes a dist step that invokes a tool build step, where *some* compiler is being propagated through, without it being clear what does that compiler represent. This refactoring hopefully makes that clearer and more explicit. It also gets rid of a few `builder.ensure(Rustc(...))` calls within bootstrap, which is always nice. `Rustdoc` needs to be handled a bit specially, because it acts as a compiler itself, I documented that in the changes. It wasn't practical to do these refactorings in multiple PRs, so I did it all in one PR. The meat of the change is 9ee6d1c. I tested manually that `x build rustdoc` and `x build miri` still works even with `download-rustc`, although I cannot promise any extra support for `download-rustc`, IMO we will just have to reimplement it from scratch in a different way. As usually, I did some drive-by refactorings to bootstrap, trying to document and clarify things, add more step metadata and tests. Since these changes broke Cargo, which was incorrectly using `Mode::ToolRustc`, I also changed cargo to `ToolTarget` in this PR. Best reviewed commit-by-commit (note that I renamed `link_compiler` to `target_compiler`, in accordance to the rest of bootstrap, in the last commit). r? `@jieyouxu` try-job: x86_64-gnu-aux

bors · 2025-08-01T20:13:27Z

💔 Test failed - checks-actions

Kobzol · 2025-08-02T06:53:55Z

@bors try

Consolidate staging for `rustc_private` tools try-job: x86_64-gnu-aux try-job: x86_64-msvc-ext1

rust-bors · 2025-08-02T06:53:59Z

⌛ Trying commit c46f42d with merge ea4e57d…

To cancel the try build, run the command @bors try cancel.

jieyouxu · 2025-08-02T07:19:53Z

These are some... interesting failures

Caused by:
   Unable to update file:///C:/a/rust/rust/build/x86_64-pc-windows-msvc/stage1-tools/x86_64-pc-windows-msvc/tmp/cit/t2429/bar?branch=master#837a694d

rust-bors · 2025-08-02T09:08:44Z

💔 Test failed (CI). Failed jobs:

bors build finished (web logs, extended logs)
try - x86_64-msvc-ext1 (web logs, extended logs)

Kobzol · 2025-08-04T07:55:17Z

@bors try

Consolidate staging for `rustc_private` tools try-job: x86_64-gnu-aux try-job: x86_64-msvc-ext1

rust-bors · 2025-08-04T07:55:22Z

⌛ Trying commit 6380972 with merge 1e033b9…

To cancel the try build, run the command @bors try cancel.

rust-bors · 2025-08-04T10:14:06Z

☀️ Try build successful (CI)
Build commit: 1e033b9 (1e033b99400f76013b04ad3c4fcafa1de5a6a25d, parent: 07b7dc90ee4df5815dbb91ef8e98cb93571230f5)

Kobzol · 2025-08-04T10:22:08Z

Hmm, maybe I botched the environment path joining somehow before.

@bors r=jieyouxu

bors · 2025-08-04T10:22:11Z

📌 Commit ac28b5b has been approved by jieyouxu

It is now in the queue for this repository.

rustbot assigned jieyouxu Jul 22, 2025

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jul 22, 2025

This comment has been minimized.

Sign in to view

Kobzol force-pushed the bootstrap-tool-cleanup branch from 2e63ce8 to 7bf4a1a Compare July 23, 2025 14:14

Kobzol mentioned this pull request Jul 26, 2025

incorrect comment in bootstrap: stage 1 rustdoc *can* be used #142745

Open

jieyouxu approved these changes Jul 28, 2025

View reviewed changes

src/bootstrap/src/core/build_steps/tool.rs Outdated Show resolved Hide resolved

src/bootstrap/src/core/builder/tests.rs Outdated Show resolved Hide resolved

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 28, 2025

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 28, 2025

samueltardieu mentioned this pull request Jul 28, 2025

Rollup of 11 pull requests #144611

Closed

Zalathar mentioned this pull request Jul 29, 2025

Rollup of 12 pull requests #144618

Closed

Zalathar mentioned this pull request Jul 29, 2025

Rollup of 12 pull requests #144619

Closed

Zalathar mentioned this pull request Jul 29, 2025

Rollup of 11 pull requests #144622

Closed

bors removed the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Jul 29, 2025

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Aug 1, 2025

jieyouxu approved these changes Aug 1, 2025

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 1, 2025

Kobzol mentioned this pull request Aug 1, 2025

Refactor codegen backends in bootstrap #144787

Open

This comment has been minimized.

Sign in to view

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Aug 1, 2025

rust-bors bot added a commit that referenced this pull request Aug 2, 2025

Auto merge of #144303 - Kobzol:bootstrap-tool-cleanup, r=<try>

ea4e57d

Consolidate staging for `rustc_private` tools try-job: x86_64-gnu-aux try-job: x86_64-msvc-ext1

This comment has been minimized.

Sign in to view

rust-bors bot added a commit that referenced this pull request Aug 4, 2025

Auto merge of #144303 - Kobzol:bootstrap-tool-cleanup, r=<try>

1e033b9

Consolidate staging for `rustc_private` tools try-job: x86_64-gnu-aux try-job: x86_64-msvc-ext1

Fix splitting dylib paths

ac28b5b

Kobzol force-pushed the bootstrap-tool-cleanup branch from 6380972 to ac28b5b Compare August 4, 2025 10:21

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 4, 2025

Consolidate staging for rustc_private tools #144303

Are you sure you want to change the base?

Consolidate staging for rustc_private tools #144303

Uh oh!

Conversation

Kobzol commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Jul 22, 2025

Uh oh!

This comment has been minimized.

bors commented Jul 23, 2025

Uh oh!

jieyouxu commented Jul 25, 2025

Uh oh!

jieyouxu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jieyouxu commented Jul 28, 2025

Uh oh!

Kobzol commented Jul 28, 2025

Uh oh!

bors commented Jul 28, 2025

Uh oh!

Zalathar commented Jul 29, 2025

Uh oh!

jieyouxu commented Aug 1, 2025

Uh oh!

bors commented Aug 1, 2025

Uh oh!

bors commented Aug 1, 2025

Uh oh!

This comment has been minimized.

bors commented Aug 1, 2025

Uh oh!

Kobzol commented Aug 2, 2025

Uh oh!

rust-bors bot commented Aug 2, 2025

Uh oh!

jieyouxu commented Aug 2, 2025

Uh oh!

rust-bors bot commented Aug 2, 2025

Uh oh!

This comment has been minimized.

Kobzol commented Aug 4, 2025

Uh oh!

rust-bors bot commented Aug 4, 2025

Uh oh!

rust-bors bot commented Aug 4, 2025

Uh oh!

Kobzol commented Aug 4, 2025

Uh oh!

bors commented Aug 4, 2025

Uh oh!

Uh oh!

Consolidate staging for `rustc_private` tools #144303

Consolidate staging for `rustc_private` tools #144303

Kobzol commented Jul 22, 2025 •

edited

Loading

jieyouxu left a comment •

edited

Loading