Building LLVM from source doesn't forward `-j x` to cmake

[sassy-crick]

I am building Rust on a large HPC cluster using EasyBuild but Rust is not behaving as it should:

3450751 pts/1    S+     0:00  |                   \_ /bin/bash /rds/general/user/jsassman/home/.local/bin/eb --robot --download-timeout=100 --modules-tool=EnvironmentModules --module-syntax=Tcl --allow-modules-tool-mismatch -dr --parallel=64 Rust-1.54.0-GCCcore-11.2.0.eb
3450771 pts/1    Sl+    0:04  |                       \_ python3 -m easybuild.main --robot --download-timeout=100 --modules-tool=EnvironmentModules --module-syntax=Tcl --allow-modules-tool-mismatch -dr --parallel=64 Rust-1.54.0-GCCcore-11.2.0.eb

3450859 pts/1    S+     0:00  |                           \_ make -j 64
3450865 pts/1    S+     0:11  |                               \_ /rds/general/user/jsassman/home/apps/easybuild/amd/zen2/software/Python/3.9.6-GCCcore-11.2.0-bare/bin/python3 /dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/src/bootstrap/bootstrap.py build --stage 2
3452315 pts/1    S+     0:00  |                                   \_ /dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/build/bootstrap/debug/bootstrap build --stage 2
3487991 pts/1    S+     0:00  |                                       \_ cmake --build . --target install --config Release -- -j 256
3487994 pts/1    S+     0:00  |                                           \_ /usr/bin/gmake -f Makefile -j 256 install

3450751 pts/1    S+     0:00  |                   \_ /bin/bash /rds/general/user/jsassman/home/.local/bin/eb --robot --download-timeout=100 --modules-tool=EnvironmentModules --module-syntax=Tcl --allow-modules-tool-mismatch -dr --parallel=64 Rust-1.54.0-GCCcore-11.2.0.eb
3450771 pts/1    Sl+    0:04  |                       \_ python3 -m easybuild.main --robot --download-timeout=100 --modules-tool=EnvironmentModules --module-syntax=Tcl --allow-modules-tool-mismatch -dr --parallel=64 Rust-1.54.0-GCCcore-11.2.0.eb
--> 3450859 pts/1    S+     0:00  |                           \_ make -j 64
3450865 pts/1    S+     0:11  |                               \_ /rds/general/user/jsassman/home/apps/easybuild/amd/zen2/software/Python/3.9.6-GCCcore-11.2.0-bare/bin/python3 /dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/src/bootstrap/bootstrap.py build --stage 2
3452315 pts/1    S+     0:00  |                                   \_ /dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/build/bootstrap/debug/bootstrap build --stage 2
--> 3487991 pts/1    S+     0:00  |                                       \_ cmake --build . --target install --config Release -- -j 256
--> 3487994 pts/1    S+     0:00  |                                           \_ /usr/bin/gmake -f Makefile -j 256 install
3488033 pts/1    S+     0:00  |                                               \_ /usr/bin/gmake -s -f CMakeFiles/Makefile2 all
3488574 pts/1    S+     0:00  |                                                   \_ /usr/bin/gmake -s -f utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/build.make utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/build
3488695 pts/1    S+     0:00  |                                                   |   \_ /rds/general/user/jsassman/home/apps/easybuild/amd/zen2/software/GCCcore/11.2.0/bin/g++ -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/build/x86_64-unknown-linux-gnu/llvm/build/utils/TableGen -I/dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/src/llvm-project/llvm/utils/TableGen -I/dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/build/x86_64-unknown-linux-gnu/llvm/build/include -I/dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/src/llvm-project/llvm/include -ffunction-sections -fdata-sections -fPIC -m64 -ftree-vectorize -march=native -fno-math-errno -fPIC -fvisibility-inlines-hidden -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -ffunction-sections -fdata-sections -O3 -DNDEBUG -fno-exceptions -fno-rtti -std=c++14 -MD -MT utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmWriterEmitter.cpp.o -MF CMakeFiles/obj.llvm-tblgen.dir/AsmWriterEmitter.cpp.o.d -o CMakeFiles/obj.llvm-tblgen.dir/AsmWriterEmitter.cpp.o -c /dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/src/llvm-project/llvm/utils/TableGen/AsmWriterEmitter.cpp

I expected to see only 64 threads being used but Rust is ignoring my request to use only 64 cores and uses the max-number of available treads. This leads to a very poor performance as each thread spends more time waiting than actually doing. specially when there are 128 threads available. This seems to be a bug in Rust and not in EasyBuild, as above.

Meta

 Rust 1.54.0

[krishchanDr]

Hello,
we tried to use PARALLEL_MAKE and tried to achieve the faster build.
Here are my observations,
8 Core  - Build completed successfully in 0:22:52
16 Core - Build completed successfully in 0:15:11
32 Core - Build completed successfully in 0:12:37
64 Core - Build completed successfully in 0:12:14
96 Core - Build completed successfully in 0:12:33
128 Core - Build completed successfully in 0:12:11
192 Core - Build completed successfully in 0:12:09
When the PARALLEL_MAKE is greater than or equal to 32, the time remains same...
Seems that its run as 32 parallel at the most

Regards
Chandra

[sassy-crick]

To add to this: this is roughly what I observe as well: building with cores is not significantly slower than with 64 cores.
That said: the problem is that right now it is ignoring the make -j 8 command and uses all available cores, real and virtual ones. That should not happen in my opinion.

[jyn514]

@sassy-crick it looks like this issue is specifically about cmake not being passed the correct -j flag, is that right? Not cargo in general?

[jyn514]

Mentoring instructions: Change the code around

rust/src/bootstrap/native.rs

Lines 315 to 317 in 9ef8407

	cfg.out_dir(&out_dir)
	.profile(profile)
	.define("LLVM_ENABLE_ASSERTIONS", assertions)

to pass build_arg(format!("-j{}", builder.config.jobs)) (or configure_arg, I don't know cmake well enough).

[rperier]

If the issue is still relevant, can I work on it ?

[Kobzol]

It would be useful to find out if it is still relevant :)

[rperier]

Mentoring instructions: Change the code around

rust/src/bootstrap/native.rs

Lines 315 to 317 in 9ef8407
cfg.out_dir(&out_dir)
.profile(profile)
.define("LLVM_ENABLE_ASSERTIONS", assertions)
to pass build_arg(format!("-j{}", builder.config.jobs)) (or configure_arg, I don't know cmake well enough).

The code for handling the LLVM Step has moved to

rust/src/bootstrap/src/core/build_steps/llvm.rs

Lines 337 to 339 in 3014e79

	cfg.out_dir(&out_dir)
	.profile(profile)
	.define("LLVM_ENABLE_ASSERTIONS", assertions)

, it seems.

[sassy-crick]

Apologies, somehow the email got snowed under.
Yes, the problem was that make -j X was being ignored (I am building with make and not cmake) and thus all available cores are being used.
Given I opened that around 3 years ago, I think the issue has been addressed in newer builds as I did not observe that behaviour any more.
Thus, I leave it up to the maintainers to mark that as resolved as I am sure quite some changes in the code have happened since.
Thanks for looking into that.