-
Notifications
You must be signed in to change notification settings - Fork 14.7k
[llvm] Proofread TestingGuide.rst #152089
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
kazutakahirata
wants to merge
2
commits into
llvm:main
Choose a base branch
from
kazutakahirata:cleanup_20250804_proofread_llvm_TestingGuide
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
[llvm] Proofread TestingGuide.rst #152089
kazutakahirata
wants to merge
2
commits into
llvm:main
from
kazutakahirata:cleanup_20250804_proofread_llvm_TestingGuide
+55
−55
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@llvm/pr-subscribers-testing-tools Author: Kazu Hirata (kazutakahirata) ChangesFull diff: https://github.com/llvm/llvm-project/pull/152089.diff 1 Files Affected:
diff --git a/llvm/docs/TestingGuide.rst b/llvm/docs/TestingGuide.rst
index 76b6b4e2cab30..2d006d509b076 100644
--- a/llvm/docs/TestingGuide.rst
+++ b/llvm/docs/TestingGuide.rst
@@ -30,9 +30,9 @@ LLVM Testing Infrastructure Organization
========================================
The LLVM testing infrastructure contains three major categories of tests:
-unit tests, regression tests and whole programs. The unit tests and regression
+unit tests, regression tests, and whole programs. The unit tests and regression
tests are contained inside the LLVM repository itself under ``llvm/unittests``
-and ``llvm/test`` respectively and are expected to always pass -- they should be
+and ``llvm/test`` respectively and are expected to always pass. They should be
run before every commit.
The whole programs tests are referred to as the "LLVM test suite" (or
@@ -48,7 +48,7 @@ Unit tests
Unit tests are written using `Google Test <https://github.com/google/googletest/blob/master/docs/primer.md>`_
and `Google Mock <https://github.com/google/googletest/blob/master/docs/gmock_for_dummies.md>`_
and are located in the ``llvm/unittests`` directory.
-In general unit tests are reserved for targeting the support library and other
+In general, unit tests are reserved for targeting the support library and other
generic data structure, we prefer relying on regression tests for testing
transformations and analysis on the IR.
@@ -61,7 +61,7 @@ written in depends on the part of LLVM being tested. These tests are driven by
the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and
are located in the ``llvm/test`` directory.
-Typically when a bug is found in LLVM, a regression test containing just
+Typically, when a bug is found in LLVM, a regression test containing just
enough code to reproduce the problem should be written and placed
somewhere underneath this directory. For example, it can be a small
piece of LLVM IR distilled from an actual application or benchmark.
@@ -82,10 +82,10 @@ for an example of such test.
The test suite contains whole programs, which are pieces of code which
can be compiled and linked into a stand-alone program that can be
-executed. These programs are generally written in high level languages
-such as C or C++.
+executed. These programs are generally written in high-level languages,
+such as C and C++.
-These programs are compiled using a user specified compiler and set of
+These programs are compiled using a user-specified compiler and set of
flags, and then executed to capture the program output and timing
information. The output of these programs is compared to a reference
output to ensure that the program is being compiled correctly.
@@ -103,11 +103,11 @@ See the :doc:`TestSuiteGuide` for details.
Debugging Information tests
---------------------------
-The test suite contains tests to check quality of debugging information.
-The test are written in C based languages or in LLVM assembly language.
+The test suite contains tests to check the quality of debugging information.
+The tests are written in C based languages or in LLVM assembly language.
These tests are compiled and run under a debugger. The debugger output
-is checked to validate of debugging information. See README.txt in the
+is checked to validate the debugging information. See ``README.txt`` in the
test suite for more information. This test suite is located in the
``cross-project-tests/debuginfo-tests`` directory.
@@ -126,13 +126,13 @@ and C++ programs. See the :doc:`TestSuiteGuide` for details.
Unit and Regression tests
-------------------------
-To run all of the LLVM unit tests use the check-llvm-unit target:
+To run all of the LLVM unit tests, use the ``check-llvm-unit`` target:
.. code-block:: bash
% make check-llvm-unit
-To run all of the LLVM regression tests use the check-llvm target:
+To run all of the LLVM regression tests, use the ``check-llvm`` target:
.. code-block:: bash
@@ -163,7 +163,7 @@ to enable testing with valgrind and with leak checking enabled.
To run individual tests or subsets of tests, you can use the ``llvm-lit``
script which is built as part of LLVM. For example, to run the
-``Integer/BitPacked.ll`` test by itself you can run:
+``Integer/BitPacked.ll`` test by itself, you can run:
.. code-block:: bash
@@ -224,35 +224,35 @@ only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit
documentation <CommandGuide/lit>` for more information.
Each test file must contain lines starting with "RUN:" that tell :program:`lit`
-how to run it. If there are no RUN lines, :program:`lit` will issue an error
+how to run it. If there are no ``RUN`` lines, :program:`lit` will issue an error
while running a test.
-RUN lines are specified in the comments of the test program using the
+``RUN`` lines are specified in the comments of the test program using the
keyword ``RUN`` followed by a colon, and lastly the command (pipeline)
to execute. Together, these lines form the "script" that :program:`lit`
-executes to run the test case. The syntax of the RUN lines is similar to a
+executes to run the test case. The syntax of the ``RUN`` lines is similar to a
shell's syntax for pipelines including I/O redirection and variable
substitution. However, even though these lines may *look* like a shell
-script, they are not. RUN lines are interpreted by :program:`lit`.
+script, they are not. ``RUN`` lines are interpreted by :program:`lit`.
Consequently, the syntax differs from shell in a few ways. You can specify
-as many RUN lines as needed.
+as many ``RUN`` lines as needed.
-:program:`lit` performs substitution on each RUN line to replace LLVM tool names
+:program:`lit` performs substitution on each ``RUN`` line to replace LLVM tool names
with the full paths to the executable built for each tool (in
``$(LLVM_OBJ_ROOT)/bin``). This ensures that :program:`lit` does
not invoke any stray LLVM tools in the user's path during testing.
-Each RUN line is executed on its own, distinct from other lines unless
-its last character is ``\``. This continuation character causes the RUN
-line to be concatenated with the next one. In this way you can build up
+Each ``RUN`` line is executed on its own, distinct from other lines unless
+its last character is ``\``. This continuation character causes the ``RUN``
+line to be concatenated with the next one. In this way, you can build up
long pipelines of commands without making huge line lengths. The lines
-ending in ``\`` are concatenated until a RUN line that doesn't end in
-``\`` is found. This concatenated set of RUN lines then constitutes one
+ending in ``\`` are concatenated until a ``RUN`` line that doesn't end in
+``\`` is found. This concatenated set of ``RUN`` lines then constitutes one
execution. :program:`lit` will substitute variables and arrange for the pipeline
to be executed. If any process in the pipeline fails, the entire line (and
test case) fails too.
-Below is an example of legal RUN lines in a ``.ll`` file:
+Below is an example of legal ``RUN`` lines in a ``.ll`` file:
.. code-block:: llvm
@@ -260,19 +260,19 @@ Below is an example of legal RUN lines in a ``.ll`` file:
; RUN: llvm-dis < %s.bc-13 > %t2
; RUN: diff %t1 %t2
-As with a Unix shell, the RUN lines permit pipelines and I/O
+As with a Unix shell, the ``RUN`` lines permit pipelines and I/O
redirection to be used.
There are some quoting rules that you must pay attention to when writing
-your RUN lines. In general nothing needs to be quoted. :program:`lit` won't
+your ``RUN`` lines. In general, nothing needs to be quoted. :program:`lit` won't
strip off any quote characters so they will get passed to the invoked program.
To avoid this use curly braces to tell :program:`lit` that it should treat
everything enclosed as one value.
-In general, you should strive to keep your RUN lines as simple as possible,
+In general, you should strive to keep your ``RUN`` lines as simple as possible,
using them only to run tools that generate textual output you can then examine.
The recommended way to examine output to figure out if the test passes is using
-the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in RUN
+the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in ``RUN``
lines is deprecated - please do not send or commit patches that use it.]*
Put related tests into a single file rather than having a separate file per
@@ -283,11 +283,11 @@ Generating assertions in regression tests
-----------------------------------------
Some regression test cases are very large and complex to write/update by hand.
-In that case to reduce the human work we can use the scripts available in
-llvm/utils/ to generate the assertions.
+In that case, to reduce the manual work, we can use the scripts available in
+``llvm/utils/`` to generate the assertions.
-For example to generate assertions in an :program:`llc`-based test, after
-adding one or more RUN lines use:
+For example, to generate assertions in an :program:`llc`-based test, after
+adding one or more ``RUN`` lines, use:
.. code-block:: bash
@@ -368,7 +368,7 @@ Best practices for regression tests
Extra files
-----------
-If your test requires extra files besides the file containing the ``RUN:`` lines
+If your test requires extra files besides the file containing the ``RUN:`` lines,
and the extra files are small, consider specifying them in the same file and
using ``split-file`` to extract them. For example,
@@ -442,7 +442,7 @@ Elaborated tests
Generally, IR and assembly test files benefit from being cleaned to remove
unnecessary details. However, for tests requiring elaborate IR or assembly
-files where cleanup is less practical (e.g., large amount of debug information
+files where cleanup is less practical (e.g., a large amount of debug information
output from Clang), you can include generation instructions within
``split-file`` part called ``gen``. Then, run
``llvm/utils/update_test_body.py`` on the test file to generate the needed
@@ -472,7 +472,7 @@ then rewrite the part after ``gen`` with its stdout.
For convenience, if the test needs one single assembly file, you can also wrap
``gen`` and its required files with ``.ifdef`` and ``.endif``. Then you can
-skip ``split-file`` in RUN lines.
+skip ``split-file`` in ``RUN`` lines.
.. code-block:: none
@@ -521,7 +521,7 @@ utilize ``split-file`` in ``RUN`` lines.
Fragile tests
-------------
-It is easy to write a fragile test that would fail spuriously if the tool being
+It is easy to write a fragile test that could fail spuriously if the tool being
tested outputs a full path to the input file. For example, :program:`opt` by
default outputs a ``ModuleID``:
@@ -552,7 +552,7 @@ default outputs a ``ModuleID``:
This test will fail if placed into a ``download`` directory.
-To make your tests robust, always use ``opt ... < %s`` in the RUN line.
+To make your tests robust, always use ``opt ... < %s`` in the ``RUN`` line.
:program:`opt` does not output a ``ModuleID`` when input comes from stdin.
Platform-Specific Tests
@@ -560,21 +560,21 @@ Platform-Specific Tests
Whenever adding tests that require the knowledge of a specific platform,
either related to code generated, specific output or back-end features,
-you must make sure to isolate the features, so that buildbots that
+you must isolate the features, so that buildbots that
run on different architectures (and don't even compile all back-ends),
don't fail.
The first problem is to check for target-specific output, for example sizes
of structures, paths and architecture names, for example:
-* Tests containing Windows paths will fail on Linux and vice-versa.
+* Tests containing Windows paths will fail on Linux and vice versa.
* Tests that check for ``x86_64`` somewhere in the text will fail anywhere else.
* Tests where the debug information calculates the size of types and structures.
-Also, if the test rely on any behaviour that is coded in any back-end, it must
+Also, if the test relies on any behaviour that is coded in any back-end, it must
go in its own directory. So, for instance, code generator tests for ARM go
into ``test/CodeGen/ARM`` and so on. Those directories contain a special
-``lit`` configuration file that ensure all tests in that directory will
+``lit`` configuration file that ensures all tests in that directory will
only run if a specific back-end is compiled and available.
For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is:
@@ -622,7 +622,7 @@ with debug builds or on particular platforms. Use ``REQUIRES``
and ``UNSUPPORTED`` to control when the test is enabled.
Some tests are expected to fail. For example, there may be a known bug
-that the test detect. Use ``XFAIL`` to mark a test as an expected failure.
+that the test detects. Use ``XFAIL`` to mark a test as an expected failure.
An ``XFAIL`` test will be successful if its execution fails, and
will be a failure if its execution succeeds.
@@ -645,7 +645,7 @@ list of boolean expressions. The values in each expression may be:
expressions can appear inside an identifier, so for example ``he{{l+}}o`` would match
``helo``, ``hello``, ``helllo``, and so on.
- The default target triple, preceded by the string ``target=`` (for example,
- ``target=x86_64-pc-windows-msvc``). Typically regular expressions are used
+ ``target=x86_64-pc-windows-msvc``). Typically, regular expressions are used
to match parts of the triple (for example, ``target={{.*}}-windows{{.*}}``
to match any Windows target triple).
@@ -684,7 +684,7 @@ have different effects. ``UNSUPPORTED`` causes the test to be skipped;
this saves execution time, but then you'll never know whether the test
actually would start working. Conversely, ``XFAIL`` actually runs the test
but expects a failure output, taking extra execution time but alerting you
-if/when the test begins to behave correctly (an XPASS test result). You
+if/when the test begins to behave correctly (an ``XPASS`` test result). You
need to decide which is more appropriate in each case.
**Using ``target=...``**
@@ -698,7 +698,7 @@ and it's generally a good idea to use a trailing wildcard to allow for
unexpected suffixes.
Also, it's generally better to write regular expressions that use entire
-triple components, than to do something clever to shorten them. For
+triple components than to do something clever to shorten them. For
example, to match both freebsd and netbsd in an expression, you could write
``target={{.*(free|net)bsd.*}}`` and that would work. However, it would
prevent a ``grep freebsd`` from finding this test. Better to use:
@@ -708,8 +708,8 @@ prevent a ``grep freebsd`` from finding this test. Better to use:
Substitutions
-------------
-Besides replacing LLVM tool names the following substitutions are performed in
-RUN lines:
+Besides replacing LLVM tool names, the following substitutions are performed in
+``RUN`` lines:
``%%``
Replaced by a single ``%``. This allows escaping other substitutions.
@@ -726,7 +726,7 @@ RUN lines:
Example: ``/home/user/llvm/test/MC/ELF``
``%t``
- File path to a temporary file name that could be used for this test case.
+ File path to a temporary file name that can be used for this test case.
The file name won't conflict with other test cases. You can append to it
if you need multiple temporaries. This is useful as the destination of
some redirected output.
@@ -811,7 +811,7 @@ RUN lines:
optional integer offset. These expand only if they appear
immediately in ``RUN:``, ``DEFINE:``, and ``REDEFINE:`` directives.
Occurrences in substitutions defined elsewhere are never expanded.
- For example, this can be used in tests with multiple RUN lines,
+ For example, this can be used in tests with multiple ``RUN`` lines,
which reference the test file's line numbers.
**LLVM-specific substitutions:**
@@ -988,7 +988,7 @@ directives:
- **Substitution value**: The value includes all text from the first
non-whitespace character after ``=`` to the last non-whitespace character. If
there is no non-whitespace character after ``=``, the value is the empty
- string. Escape sequences that can appear in python ``re.sub`` replacement
+ string. Escape sequences that can appear in Python ``re.sub`` replacement
strings are treated as plain text in the value.
- **Line continuations**: If the last non-whitespace character on the line after
``:`` is ``\``, then the next directive must use the same directive keyword
@@ -1057,7 +1057,7 @@ producing incorrect output.
Options
-------
-The llvm lit configuration allows to customize some things with user options:
+The llvm lit configuration allows some things to be customized with user options:
``llc``, ``opt``, ...
Substitute the respective llvm tool name with a custom command line. This
@@ -1076,8 +1076,8 @@ The llvm lit configuration allows to customize some things with user options:
Other Features
--------------
-To make RUN line writing easier, there are several helper programs. These
-helpers are in the PATH when running tests, so you can just call them using
+To make ``RUN`` line writing easier, several helper programs are available. These
+helpers are in the ``PATH`` when running tests, so you can just call them using
their name. For example:
``not``
|
arsenm
approved these changes
Aug 5, 2025
jh7370
approved these changes
Aug 5, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One suggestion, otherwise LGTM.
Co-authored-by: James Henderson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.