Skip to content

Refactor codegen backends in bootstrap #144787

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 24 commits into
base: master
Choose a base branch
from

Conversation

Kobzol
Copy link
Member

@Kobzol Kobzol commented Aug 1, 2025

This PR refactors the codegen backend steps, in preparation to make more progress on the integration of the GCC codegen backend in bootstrap. It does several things:

  1. Splits the CodegenBackend step into two, one for clif and another one for gcc. Even though their code is mostly similar, that's IMO mostly fake similarity, and they do (or will) ultimately require different handling. This was already visible in the requirement of building GCC for cg_gcc, of course.
  2. It is now possible to build both backends (and dist cranelift) even if they are not specified in rust.codegen-backends. It was quite weird that it wasn't possible to even invoke the corresponding codegen backend if the backend wasn't specified in that array, as that array should ideally only change defaults (see later below).
  3. Changes the path specification of these steps to an alias. In other words, instead of compiler/rustc_codegen_cranelift, the step is now built only using rustc_codegen_cranelift or cg_clif. This is done to avoid an annoying clash with x build compiler, which would otherwise build both codegen backends after the 2) change.
  4. Made the copying of codegen backend artifacts more explicit, in particular in the Assemble step.
  5. Codifies the semantics of rust.codegen-backends, which now only affects the defaults of whether a codegen backend will be included in rustc's sysroot and whether it will be disted in x dist by default. We can change the behavior later, e.g. to dist cranelift by default in x dist once it becomes stabilized. Currently I left the existing behavior that we use on CI, I just tried to document it better.

I don't think that this requires a change tracker entry, because the defaults should work the same as before. It is just now possible to do x build/dist rustc_codegen_cranelift even if CLIF is not in the codegen-backends array. It is no longer possible to do ./x build compiler/rustc_codegen_cranelift though, not sure if that requires a change tracker entry.

There is one thing that I didn't touch yet, and that is the fact that rust.codegen-backends not only affects the default behavior of x dist w.r.t. Cranelift, but also of x test. In other words, x test rustc_codegen_cranelift still does not hing if cranelift isn't in rust.codegen-backends. I plan to take a look at this once I get to refactoring the test steps.

Based on #144303, which fixed the staging for codegen backends and introduced the RustcPrivateCompilers abstraction (last three commits are new).

r? @jieyouxu

@rustbot
Copy link
Collaborator

rustbot commented Aug 1, 2025

jieyouxu is currently at their maximum review capacity.
They may take a while to respond.

@Kobzol Kobzol changed the title Refactor codegen backends Refactor codegen backends in bootstrap Aug 1, 2025
@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Aug 1, 2025
@rustbot
Copy link
Collaborator

rustbot commented Aug 1, 2025

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

This PR modifies bootstrap.example.toml.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

@rust-log-analyzer
Copy link
Collaborator

The job pr-check-2 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)
   Compiling rustc_codegen_cranelift v0.1.0 (/checkout/compiler/rustc_codegen_cranelift)
[RUSTC-TIMING] rustc_codegen_cranelift test:false 9.731
    Finished `release` profile [optimized] target(s) in 1m 06s
##[endgroup]
[TIMING] core::build_steps::compile::CraneliftCodegenBackend { compilers: RustcPrivateCompilers { build_compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target_compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu, forced_compiler: false } } } -- 66.891
Creating a sysroot for stage1 compiler (use `rustup toolchain link 'name' build/host/stage1`)
[TIMING] core::build_steps::llvm::Lld { target: x86_64-unknown-linux-gnu } -- 0.000
##[group]Building stage0 tool lld-wrapper (x86_64-unknown-linux-gnu)
[RUSTC-TIMING] lld_wrapper test:false 0.241
    Finished `release` profile [optimized] target(s) in 0.12s
---
[RUSTC-TIMING] object test:false 2.459
error: this expression creates a reference which is immediately dereferenced by the compiler
    --> src/bootstrap/src/core/build_steps/dist.rs:1418:49
     |
1418 |         let mut tarball = Tarball::new(builder, &"rustc-codegen-cranelift", &target.triple);
     |                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^ help: change this to: `"rustc-codegen-cranelift"`
     |
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
     = note: `-D clippy::needless-borrow` implied by `-D warnings`
     = help: to override `-D warnings` add `#[allow(clippy::needless_borrow)]`

#
# Note that the LLVM codegen backend is special and will always be built and distributed.
#
# Currently, the only standard options supported here are `"llvm"`, `"cranelift"` and `"gcc"`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General comment: this is exactly the behaviour I wanted, thanks so much for implementing it!

Copy link
Member Author

@Kobzol Kobzol Aug 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the existing behavior, I didn't change it in this PR, just documented it and refactored it :D The only change is that it is now possible to build a backend even if it isn't in the array in the config file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants