Fix parallel rustc not being reproducible due to unstable sorts of items #144722

ywxt · 2025-07-31T08:11:42Z

Currently, A tuple (DefId, SymbolName) is used to determine the order of items in the final binary. However DefId is expected as non-deterministic, which leads to some not reproducible issues under parallel compilation. (See #140425 (comment))

Theoretically, we don't need the sorting because the order of these items is already deterministic.

However, codegen tests reply on the same order of items between in binary and source.

So here we added a new option codegen-source-order to indicate whether sorting based on the order in source. For codegen tests, items are sorted according to the order in the source code, whereas in the normal path, no sorting is performed.

Specially, for codegen tests, in preparation for parallel compilation potentially being enabled by default in the future, we use Span replacing DefId to make the order deterministic.

This PR is purposed to fix #140425, but seemly works on #140413 too.

This behavior hasn't added into any test until we have a test suit for the parallel frontend. (See #143953)

Related discussion: Zulip #144576

Update #113349

r? @oli-obk
cc @lqd @cramertj @matthiaskrgr @Zoxc @SparrowLii @bjorn3 @cjgillot @joshtriplett

rustbot · 2025-07-31T08:11:46Z

oli-obk is not on the review rotation at the moment.
They may take a while to respond.

rustbot · 2025-07-31T08:11:49Z

Some changes occurred in src/tools/compiletest

cc @jieyouxu

ywxt · 2025-07-31T09:37:37Z

I don't think the changes of diagnostic orders are expected.😅

compiler/rustc_middle/src/mir/mono.rs

cjgillot · 2025-08-03T20:29:28Z

compiler/rustc_middle/src/mir/mono.rs

-                    MonoItem::Static(def_id) => def_id.as_local().map(Idx::index),
-                    MonoItem::GlobalAsm(item_id) => Some(item_id.owner_id.def_id.index()),
-                },
+                local_item_query(item, |def_id| tcx.def_span(def_id)),


Should we try to use def_ident_span or find_ancestor_not_from_extern_macro to avoid having to shuffle tests?

Thanks for your advice. I'll try it. :)

~~def_ident_span failed on tests/assembly-llvm/emit-intel-att-syntax.rs about naked and global asm.~~

find_ancestor_not_from_macro works here.

ywxt · 2025-08-04T06:24:51Z

If we don't sort items again, it would broke existing tests.

How about (def_span().find_ancestor_not_form_macro(), SymbolName) to replace (DefId, SymbolName) instead of using a new option?

This reverts commit 9f47e26.

rust-log-analyzer · 2025-08-05T02:19:05Z

The job aarch64-gnu-llvm-19-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

---- [ui] tests/ui/linkage-attr/linkage-detect-local-generated-name-collision.rs stdout ----

error: Error: expected failure status (Some(1)) but received status Some(101).
status: exit status: 101
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/tests/ui/linkage-attr/linkage-detect-local-generated-name-collision.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/checkout/vendor" "--sysroot" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2" "--target=aarch64-unknown-linux-gnu" "--check-cfg" "cfg(test,FALSE)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/linkage-attr/linkage-detect-local-generated-name-collision" "-A" "unused" "-A" "internal_features" "-A" "unused_parens" "-A" "unused_braces" "-Crpath" "-Cdebuginfo=0" "-Lnative=/checkout/obj/build/aarch64-unknown-linux-gnu/native/rust-test-helpers" "-Ccodegen-units=16"
stdout: none
--- stderr -------------------------------

thread 'rustc' panicked at compiler/rustc_codegen_llvm/src/consts.rs:399:9:
assertion failed: llvm::LLVMGetInitializer(self.instances.borrow().get(&Instance::mono(self.tcx,
                        def_id)).unwrap()).is_none()
stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic
   3: <rustc_codegen_llvm::context::GenericCx<rustc_codegen_llvm::context::FullCx> as rustc_codegen_ssa::traits::statics::StaticCodegenMethods>::codegen_static
   4: <rustc_middle::mir::mono::MonoItem as rustc_codegen_ssa::mono_item::MonoItemExt>::define::<rustc_codegen_llvm::builder::GenericBuilder<rustc_codegen_llvm::context::FullCx>>
   5: rustc_codegen_llvm::base::compile_codegen_unit::module_codegen
   6: rustc_codegen_llvm::base::compile_codegen_unit
   7: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate
   8: rustc_interface::passes::start_codegen
   9: <rustc_interface::queries::Linker>::codegen_and_build_linker
  10: <std::thread::local::LocalKey<core::cell::Cell<*const ()>>>::with::<rustc_middle::ty::context::tls::enter_context<<rustc_middle::ty::context::GlobalCtxt>::enter<rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}, core::option::Option<rustc_interface::queries::Linker>>::{closure#1}, core::option::Option<rustc_interface::queries::Linker>>::{closure#0}, core::option::Option<rustc_interface::queries::Linker>>
  11: <rustc_middle::ty::context::TyCtxt>::create_global_ctxt::<core::option::Option<rustc_interface::queries::Linker>, rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}>
  12: <rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2} as core::ops::function::FnOnce<(&rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2})>>::call_once::{shim:vtable#0}
  13: <alloc::boxed::Box<dyn for<'a> core::ops::function::FnOnce<(&'a rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &'a std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt<'a>>, &'a rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena<'a>>, &'a rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena<'a>>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}), Output = core::option::Option<rustc_interface::queries::Linker>>> as core::ops::function::FnOnce<(&rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2})>>::call_once
  14: rustc_interface::passes::create_and_enter_global_ctxt::<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>
  15: <scoped_tls::ScopedKey<rustc_span::SessionGlobals>>::set::<rustc_interface::util::run_in_thread_with_globals<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}::{closure#0}, ()>
  16: rustc_span::create_session_globals_then::<(), rustc_interface::util::run_in_thread_with_globals<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}::{closure#0}>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---
note: please make sure that you have updated to the latest nightly

note: rustc 1.91.0-nightly (866f0240c 2025-08-05) running on aarch64-unknown-linux-gnu

note: compiler flags: -Z threads=1 -Z simulate-remapped-rust-src-base=/rustc/FAKE_PREFIX -Z translate-remapped-path-to-local-path=no -Z ignore-directory-in-diagnostics-source-blocks=/cargo -Z ignore-directory-in-diagnostics-source-blocks=/checkout/vendor -C codegen-units=1 -Z ui-testing -Z deduplicate-diagnostics=no -Z write-long-types-to-disk=no -C strip=debuginfo -C prefer-dynamic -C rpath -C debuginfo=0 -C codegen-units=16

query stack during panic:
end of query stack
------------------------------------------


---- [ui] tests/ui/linkage-attr/linkage-detect-extern-generated-name-collision.rs stdout ----

error: Error: expected failure status (Some(1)) but received status Some(101).
status: exit status: 101
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/tests/ui/linkage-attr/linkage-detect-extern-generated-name-collision.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/checkout/vendor" "--sysroot" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2" "--target=aarch64-unknown-linux-gnu" "--check-cfg" "cfg(test,FALSE)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/linkage-attr/linkage-detect-extern-generated-name-collision" "-A" "unused" "-A" "internal_features" "-A" "unused_parens" "-A" "unused_braces" "-Crpath" "-Cdebuginfo=0" "-Lnative=/checkout/obj/build/aarch64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/linkage-attr/linkage-detect-extern-generated-name-collision/auxiliary" "-Ccodegen-units=16"
stdout: none
--- stderr -------------------------------

thread 'rustc' panicked at compiler/rustc_codegen_llvm/src/consts.rs:399:9:
assertion failed: llvm::LLVMGetInitializer(self.instances.borrow().get(&Instance::mono(self.tcx,
                        def_id)).unwrap()).is_none()
stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic
   3: <rustc_codegen_llvm::context::GenericCx<rustc_codegen_llvm::context::FullCx> as rustc_codegen_ssa::traits::statics::StaticCodegenMethods>::codegen_static
   4: <rustc_middle::mir::mono::MonoItem as rustc_codegen_ssa::mono_item::MonoItemExt>::define::<rustc_codegen_llvm::builder::GenericBuilder<rustc_codegen_llvm::context::FullCx>>
   5: rustc_codegen_llvm::base::compile_codegen_unit::module_codegen
   6: rustc_codegen_llvm::base::compile_codegen_unit
   7: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate
   8: rustc_interface::passes::start_codegen
   9: <rustc_interface::queries::Linker>::codegen_and_build_linker
  10: <std::thread::local::LocalKey<core::cell::Cell<*const ()>>>::with::<rustc_middle::ty::context::tls::enter_context<<rustc_middle::ty::context::GlobalCtxt>::enter<rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}, core::option::Option<rustc_interface::queries::Linker>>::{closure#1}, core::option::Option<rustc_interface::queries::Linker>>::{closure#0}, core::option::Option<rustc_interface::queries::Linker>>
  11: <rustc_middle::ty::context::TyCtxt>::create_global_ctxt::<core::option::Option<rustc_interface::queries::Linker>, rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}>
  12: <rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2} as core::ops::function::FnOnce<(&rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2})>>::call_once::{shim:vtable#0}
  13: <alloc::boxed::Box<dyn for<'a> core::ops::function::FnOnce<(&'a rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &'a std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt<'a>>, &'a rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena<'a>>, &'a rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena<'a>>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}), Output = core::option::Option<rustc_interface::queries::Linker>>> as core::ops::function::FnOnce<(&rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2})>>::call_once
  14: rustc_interface::passes::create_and_enter_global_ctxt::<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>
  15: <scoped_tls::ScopedKey<rustc_span::SessionGlobals>>::set::<rustc_interface::util::run_in_thread_with_globals<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}::{closure#0}, ()>
  16: rustc_span::create_session_globals_then::<(), rustc_interface::util::run_in_thread_with_globals<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}::{closure#0}>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---
note: please make sure that you have updated to the latest nightly

note: rustc 1.91.0-nightly (866f0240c 2025-08-05) running on aarch64-unknown-linux-gnu

note: compiler flags: -Z threads=1 -Z simulate-remapped-rust-src-base=/rustc/FAKE_PREFIX -Z translate-remapped-path-to-local-path=no -Z ignore-directory-in-diagnostics-source-blocks=/cargo -Z ignore-directory-in-diagnostics-source-blocks=/checkout/vendor -C codegen-units=1 -Z ui-testing -Z deduplicate-diagnostics=no -Z write-long-types-to-disk=no -C strip=debuginfo -C prefer-dynamic -C rpath -C debuginfo=0 -C codegen-units=16

query stack during panic:
end of query stack
------------------------------------------

rustbot assigned oli-obk Jul 31, 2025

ywxt mentioned this pull request Jul 31, 2025

Fix parallel rustc not being reproducible due to unstable sorts of items #144576

Closed

This comment has been minimized.

Sign in to view

ywxt force-pushed the parallel-reproducibile branch from b7e0813 to a03edc6 Compare August 1, 2025 01:24

This comment has been minimized.

Sign in to view

cjgillot reviewed Aug 3, 2025

View reviewed changes

ywxt force-pushed the parallel-reproducibile branch from a03edc6 to d6294cc Compare August 4, 2025 03:19

This comment has been minimized.

Sign in to view

ywxt added 11 commits August 5, 2025 08:48

Add the codegen-source-order option

3336824

Sort items only within codegen tests

264f999

Fix tests.

5993fc8

Add the test for the option codegen_source_order.

8414a96

Improve comments.

ac191a0

Put the condition into the function items_in_deterministic_order

5cbb56d

Use find_ancestor_not_from_macro

32bdf8b

Revert part of "Fix tests."

5c87faf

This reverts commit 9f47e26.

Fix fmt.

db7e549

Remove DefPath

f0bf8e4

Bless to intrinsics tests.

989e39f

ywxt force-pushed the parallel-reproducibile branch from a9ba3d9 to 989e39f Compare August 5, 2025 01:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix parallel rustc not being reproducible due to unstable sorts of items #144722

Fix parallel rustc not being reproducible due to unstable sorts of items #144722

Uh oh!

ywxt commented Jul 31, 2025 •

edited

Loading

Uh oh!

rustbot commented Jul 31, 2025

Uh oh!

rustbot commented Jul 31, 2025

Uh oh!

This comment has been minimized.

ywxt commented Jul 31, 2025

Uh oh!

This comment has been minimized.

Uh oh!

cjgillot Aug 3, 2025

Uh oh!

ywxt Aug 4, 2025

Uh oh!

ywxt Aug 4, 2025 •

edited

Loading

Uh oh!

This comment has been minimized.

ywxt commented Aug 4, 2025 •

edited

Loading

Uh oh!

rust-log-analyzer commented Aug 5, 2025

Uh oh!

Uh oh!

Fix parallel rustc not being reproducible due to unstable sorts of items #144722

Are you sure you want to change the base?

Fix parallel rustc not being reproducible due to unstable sorts of items #144722

Uh oh!

Conversation

ywxt commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Jul 31, 2025

Uh oh!

rustbot commented Jul 31, 2025

Uh oh!

This comment has been minimized.

ywxt commented Jul 31, 2025

Uh oh!

This comment has been minimized.

Uh oh!

cjgillot Aug 3, 2025

Choose a reason for hiding this comment

Uh oh!

ywxt Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

ywxt Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

ywxt commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rust-log-analyzer commented Aug 5, 2025

Uh oh!

Uh oh!

ywxt commented Jul 31, 2025 •

edited

Loading

ywxt Aug 4, 2025 •

edited

Loading

ywxt commented Aug 4, 2025 •

edited

Loading