Skip to content

Commit 3a6b60f

Browse files
JDevlieghereJDevlieghere
committed
[lldb/Docs] Add some more info about the test suite structure
Expand on the structure of the LLDB test suite. So far this information has been mostly "tribal knowledge". By writing it down I hope to make it easier to understand our test suite for anyone that's new to the project.
1 parent d736571 commit 3a6b60f

File tree

1 file changed

+190
-17
lines changed

1 file changed

+190
-17
lines changed

‎lldb/docs/resources/test.rst

Lines changed: 190 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,200 @@
11
Testing
22
=======
33

4+
.. contents::
5+
:local:
6+
7+
Test Suite Structure
8+
--------------------
9+
410
The LLDB test suite consists of three different kinds of test:
511

6-
* Unit test. These are located under ``lldb/unittests`` and are written in C++
7-
using googletest.
8-
* Integration tests that test the debugger through the SB API. These are
9-
located under ``lldb/packages/Python/lldbsuite`` and are written in Python
10-
using ``dotest`` (LLDB's custom testing framework on top of unittest2).
11-
* Integration tests that test the debugger through the command line. These are
12-
located under `lldb/test/Shell` and are written in a shell-style format
13-
using FileCheck to verify its output.
12+
* **Unit tests**: written in C++ using the googletest unit testing library.
13+
* **Shell tests**: Integration tests that test the debugger through the command
14+
line. These tests interact with the debugger either through the command line
15+
driver or through ``lldb-test`` which is a tool that exposes the internal
16+
data structures in an easy-to-parse way for testing. Most people will know
17+
these as *lit tests* in LLVM, although lit is the test driver and ShellTest
18+
is the test format that uses ``RUN:`` lines. `FileCheck
19+
<https://llvm.org/docs/CommandGuide/FileCheck.html>`_ is used to verify
20+
the output.
21+
* **API tests**: Integration tests that interact with the debugger through the
22+
SB API. These are written in Python and use LLDB's ``dotest.py`` testing
23+
framework on top of Python's `unittest2
24+
<https://docs.python.org/2/library/unittest.html>`_.
25+
26+
All three test suites use ``lit`` (`LLVM Integrated Tester
27+
<https://llvm.org/docs/CommandGuide/lit.html>`_ ) as the test driver. The test
28+
suites can be run as a whole or separately.
29+
30+
31+
Unit Tests
32+
``````````
33+
34+
Unit tests are located under ``lldb/unittests``. If it's possible to test
35+
something in isolation or as a single unit, you should make it a unit test.
36+
37+
Often you need instances of the core objects such as a debugger, target or
38+
process, in order to test something meaningful. We already have a handful of
39+
tests that have the necessary boiler plate, but this is something we could
40+
abstract away and make it more user friendly.
41+
42+
Shell Tests
43+
```````````
44+
45+
Shell tests are located under ``lldb/test/Shell``. These tests are generally
46+
built around checking the output of ``lldb`` (the command line driver) or
47+
``lldb-test`` using ``FileCheck``. Shell tests are generally small and fast to
48+
write because they require little boilerplate.
49+
50+
``lldb-test`` is a relatively new addition to the test suite. It was the first
51+
tool that was added that is designed for testing. Since then it has been
52+
continuously extended with new subcommands, improving our test coverage. Among
53+
other things you can use it to query lldb for symbol files, for object files
54+
and breakpoints.
55+
56+
Obviously shell tests are great for testing the command line driver itself or
57+
the subcomponents already exposed by lldb-test. But when it comes to LLDB's
58+
vast functionality, most things can be tested both through the driver as well
59+
as the Python API. For example, to test setting a breakpoint, you could do it
60+
from the command line driver with ``b main`` or you could use the SB API and do
61+
something like ``target.BreakpointCreateByName`` [#]_.
62+
63+
A good rule of thumb is to prefer shell tests when what is being tested is
64+
relatively simple. Expressivity is limited compared to the API tests, which
65+
means that you have to have a well-defined test scenario that you can easily
66+
match with ``FileCheck``.
67+
68+
Another thing to consider are the binaries being debugged, which we call
69+
inferiors. For shell tests, they have to be relatively simple. The
70+
``dotest.py`` test framework has extensive support for complex build scenarios
71+
and different variants, which is described in more detail below, while shell
72+
tests are limited to single lines of shell commands with compiler and linker
73+
invocations.
74+
75+
On the same topic, another interesting aspect of the shell tests is that there
76+
you can often get away with a broken or incomplete binary, whereas the API
77+
tests almost always require a fully functional executable. This enables testing
78+
of (some) aspects of handling of binaries with non-native architectures or
79+
operating systems.
80+
81+
Finally, the shell tests always run in batch mode. You start with some input
82+
and the test verifies the output. The debugger can be sensitive to its
83+
environment, such as the the platform it runs on. It can be hard to express
84+
that the same test might behave slightly differently on macOS and Linux.
85+
Additionally, the debugger is an interactive tool, and the shell test provide
86+
no good way of testing those interactive aspects, such as tab completion for
87+
example.
88+
89+
API Tests
90+
`````````
91+
92+
API tests are located under ``lldb/test/API``. They are run with the
93+
``dotest.py``. Tests are written in Python and test binaries (inferiors) are
94+
compiled with Make. The majority of API tests are end-to-end tests that compile
95+
programs from source, run them, and debug the processes.
96+
97+
As mentioned before, ``dotest.py`` is LLDB's testing framework. The
98+
implementation is located under ``lldb/packages/Python/lldbsuite``. We have
99+
several extensions and custom test primitives on top of what's offered by
100+
`unittest2 <https://docs.python.org/2/library/unittest.html>`_. Those can be
101+
found in
102+
`lldbtest.py <https://github.com/llvm/llvm-project/blob/master/lldb/packages/Python/lldbsuite/test/lldbtest.py>`_.
103+
104+
Below is the directory layout of the `example API test
105+
<https://github.com/llvm/llvm-project/tree/master/lldb/test/API/sample_test>`_.
106+
The test directory will always contain a python file, starting with ``Test``.
107+
Most of the tests are structured as a binary being debugged, so there will be
108+
one or more source files and a ``Makefile``.
14109

15-
All three test suites use the `LLVM Integrated Tester
16-
<https://llvm.org/docs/CommandGuide/lit.html>`_ (lit) as their test driver. The
17-
test suites can be run as a whole or separately.
110+
::
18111

19-
Many of the tests are accompanied by a C (C++, ObjC, etc.) source file. Each
20-
test first compiles the source file and then uses LLDB to debug the resulting
21-
executable.
112+
sample_test
113+
├── Makefile
114+
├── TestSampleTest.py
115+
└── main.c
116+
117+
Let's start with the Python test file. Every test is its own class and can have
118+
one or more test methods, that start with ``test_``. Many tests define
119+
multiple test methods and share a bunch of common code. For example, for a
120+
fictive test that makes sure we can set breakpoints we might have one test
121+
method that ensures we can set a breakpoint by address, on that sets a
122+
breakpoint by name and another that sets the same breakpoint by file and line
123+
number. The setup, teardown and everything else other than setting the
124+
breakpoint could be shared.
125+
126+
Our testing framework also has a bunch of utilities that abstract common
127+
operations, such as creating targets, setting breakpoints etc. When code is
128+
shared across tests, we extract it into a utility in ``lldbutil``. It's always
129+
worth taking a look at `lldbutil
130+
<https://github.com/llvm/llvm-project/blob/master/lldb/packages/Python/lldbsuite/test/lldbutil.py>`_
131+
to see if there's a utility to simplify some of the testing boiler plate.
132+
Because we can't always audit every existing test, this is doubly true when
133+
looking at an existing test for inspiration.
134+
135+
It's possible to skip or `XFAIL
136+
<https://ftp.gnu.org/old-gnu/Manuals/dejagnu-1.3/html_node/dejagnu_6.html>`_
137+
tests using decorators. You'll see them a lot. The debugger can be sensitive to
138+
things like the architecture, the host and target platform, the compiler
139+
version etc. LLDB comes with a range of predefined decorators for these
140+
configurations.
22141

23-
.. contents::
24-
:local:
142+
::
143+
144+
@expectedFailureAll(archs=["aarch64"], oslist=["linux"]
145+
146+
Another great thing about these decorators is that they're very easy to extend,
147+
it's even possible to define a function in a test case that determines whether
148+
the test should be run or not.
149+
150+
::
151+
152+
@expectedFailure(checking_function_name)
153+
154+
In addition to providing a lot more flexibility when it comes to writing the
155+
test, the API test also allow for much more complex scenarios when it comes to
156+
building inferiors. Every test has its own ``Makefile``, most of them only a
157+
few lines long. A shared ``Makefile`` (``Makefile.rules``) with about a
158+
thousand lines of rules takes care of most if not all of the boiler plate,
159+
while individual make files can be used to build more advanced tests. 

160+
161+
Here's an example of a simple ``Makefile`` used by the example test.
162+
163+
::
164+
165+
C_SOURCES := main.c
166+
CFLAGS_EXTRAS := -std=c99
167+
168+
include Makefile.rules
169+
170+
Finding the right variables to set can be tricky. You can always take a look at
171+
`Makefile.rules <https://github.com/llvm/llvm-project/blob/master/lldb/packages/Python/lldbsuite/test/make/Makefile.rules>`_
172+
but often it's easier to find an existing ``Makefile`` that does something
173+
similar to what you want to do.
174+
175+
Another thing this enables is having different variants for the same test
176+
case. By default, we run every test for all 3 debug info formats, so once with
177+
DWARF from the object files, once with gmodules and finally with a dSYM on
178+
macOS or split DWARF (DWO) on Linux. But there are many more things we can test
179+
that are orthogonal to the test itself. On GreenDragon we have a matrix bot
180+
that runs the test suite under different configurations, with older host
181+
compilers and different DWARF versions.
182+
183+
As you can imagine, this quickly lead to combinatorial explosion in the number
184+
of variants. It's very tempting to add more variants because it's an easy way
185+
to increase test coverage. It doesn't scale. It's easy to set up, but increases
186+
the runtime of the tests and has a large ongoing cost.
187+
188+
The key take away is that the different variants don't obviate the need for
189+
focused tests. So relying on it to test say DWARF5 is a really bad idea.
190+
Instead you should write tests that check the specific DWARF5 feature, and have
191+
the variant as a nice-to-have.
192+
193+
In conclusion, you'll want to opt for an API test to test the API itself or
194+
when you need the expressivity, either for the test case itself or for the
195+
program being debugged. The fact that the API tests work with different
196+
variants mean that more general tests should be API tests, so that they can be
197+
run against the different variants.
25198

26199
Running The Tests
27200
-----------------
@@ -244,4 +417,4 @@ A quick guide to getting started with PTVS is as follows:
244417

245418
--arch=i686 --executable D:/src/llvmbuild/ninja/bin/lldb.exe -s D:/src/llvmbuild/ninja/lldb-test-traces -u CXXFLAGS -u CFLAGS --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe -p TestPaths.py D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test --no-multiprocess
246419

247-
420+
.. [#] `https://lldb.llvm.org/python_reference/lldb.SBTarget-class.html#BreakpointCreateByName <https://lldb.llvm.org/python_reference/lldb.SBTarget-class.html#BreakpointCreateByName>`_

0 commit comments

Comments
 (0)