1This README file describes the files and directories related to the Python test
2suite under the current 'test' directory.
3
4o dotest.py
5
6  Provides the test driver for the test suite.  To invoke it, cd to the 'test'
7  directory and issue the './dotest.py' command or './dotest.py -v' for more
8  verbose output.  '.dotest.py -h' prints out the help messge.
9
10  A specific naming pattern is followed by the .py script under the 'test'
11  directory in order to be recognized by 'dotest.py' test driver as a module
12  which implements a test case, namely, Test*.py.
13
14  Some example usages:
15
16  1. ./dotest.py -v . 2> ~/Developer/Log/lldbtest.log0
17     This runs the test suite and directs the run log to a file.
18
19  2. LLDB_LOG=/tmp/lldb.log GDB_REMOTE_LOG=/tmp/gdb-remote.log ./dotest.py -v . 2> ~/Developer/Log/lldbtest.log
20     This runs the test suite, with logging turned on for the lldb as well as
21     the process.gdb-remote channels and directs the run log to a file.
22
23o lldbtest.py
24
25  Provides an abstract base class of lldb test case named 'TestBase', which in
26  turn inherits from Python's unittest.TestCase.  The concrete subclass can
27  override lldbtest.TestBase in order to inherit the common behavior for
28  unittest.TestCase.setUp/tearDown implemented in this file.
29
30  To provide a test case, the concrete subclass provides methods whose names
31  start with the letters test.  For more details about the Python's unittest
32  framework, go to http://docs.python.org/library/unittest.html.
33
34  ./command_source/TestCommandSource.py provides a simple example of test case
35  which overrides lldbtest.TestBase to exercise the lldb's 'command source'
36  command.  The subclass should override the attribute 'mydir' in order for the
37  runtime to locate the individual test cases when running as part of a large
38  test suite or when running each test case as a separate Python invocation.
39
40  The doc string provides more details about the setup required for running a
41  test case on its own.  To run the whole test suite, 'dotest.py' is all you
42  need to do.
43
44o subdirectories of 'test'
45
46  Most of them predate the introduction of the python test suite and contain
47  example C/C++/ObjC source files which get compiled into executables which are
48  to be exercised by the debugger.
49
50  For such subdirectory which has an associated Test*.py file, it was added as
51  part of the Python-based test suite to test lldb functionality.
52
53  Some of the subdirectories, for example, the 'help' subdirectory, do not have
54  C/C++/ObjC source files; they were created to house the Python test case which
55  does not involve lldb reading in an executable file at all.
56
57o make directory
58
59  Contains Makefile.rules, which can be utilized by test cases to write Makefile
60  based rules to build binaries for the inferiors.
61
62  By default, the built executable name is a.out, which can be overwritten by
63  specifying your EXE make variable, via the Makefile under the specific test
64  directory or via supplying a Python dictionary to the build method in your
65  Python test script.  An example of the latter can be found in
66  test/lang/objc/radar-9691614/TestObjCMethodReturningBOOL.py, where:
67
68    def test_method_ret_BOOL_with_dsym(self):
69        """Test that objective-c method returning BOOL works correctly."""
70        d = {'EXE': self.exe_name}
71        self.buildDsym(dictionary=d)
72        self.setTearDownCleanup(dictionary=d)
73        self.objc_method_ret_BOOL(self.exe_name)
74
75    def test_method_ret_BOOL_with_dwarf(self):
76        """Test that objective-c method returning BOOL works correctly."""
77        d = {'EXE': self.exe_name}
78        self.buildDwarf(dictionary=d)
79        self.setTearDownCleanup(dictionary=d)
80        self.objc_method_ret_BOOL(self.exe_name)
81
82    def setUp(self):
83        # Call super's setUp().
84        TestBase.setUp(self)
85        # We'll use the test method name as the exe_name.
86        self.exe_name = self.testMethodName
87        # Find the line number to break inside main().
88        self.main_source = "main.m"
89        self.line = line_number(self.main_source, '// Set breakpoint here.')
90
91  The exe names for the two test methods are equal to the test method names and
92  are therefore guaranteed different.
93
94o plugins directory
95
96  Contains platform specific plugin to build binaries with dsym/dwarf debugging
97  info.  Other platform specific functionalities may be added in the future.
98
99o unittest2 directory
100
101  Many new features were added to unittest in Python 2.7, including test
102  discovery. unittest2 allows you to use these features with earlier versions of
103  Python.
104
105  It currently has unittest2 0.5.1 from http://pypi.python.org/pypi/unittest2.
106  Version 0.5.1 of unittest2 has feature parity with unittest in Python 2.7
107  final. If you want to ensure that your tests run identically under unittest2
108  and unittest in Python 2.7 you should use unittest2 0.5.1.
109
110  Later versions of unittest2 include changes in unittest made in Python 3.2 and
111  onwards after the release of Python 2.7.
112
113o dotest.pl
114
115  In case you wonder, there is also a 'dotest.pl' perl script file.  It was
116  created to visit each Python test case under the specified directory and
117  invoke Python's builtin unittest.main() on each test case.
118
119  It does not take advantage of the test runner and test suite functionality
120  provided by Python's unitest framework.  Its existence is because we want a
121  different way of running the whole test suite.  As lldb and the Python test
122  suite become more reliable, we don't expect to be using 'dotest.pl' anymore.
123
124  Note: dotest.pl has been moved to the attic directory.
125
126o Profiling dotest.py runs
127
128  I used the following command line thingy to do the profiling on a SnowLeopard
129  machine:
130
131$ DOTEST_PROFILE=YES DOTEST_SCRIPT_DIR=/Volumes/data/lldb/svn/trunk/test /System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/cProfile.py -o my.profile ./dotest.py -v -w 2> ~/Developer/Log/lldbtest.log
132
133  After that, I used the pstats.py module to browse the statistics:
134
135$ python /System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/pstats.py my.profile
136
137o Writing test cases:
138
139 We strongly prefer writing test cases using the SB API's rather than the runCmd & expect.
140 Unless you are actually testing some feature of the command line, please don't write
141 command based tests.  For historical reasons there are plenty of examples of tests in the
142 test suite that use runCmd where they shouldn't, but don't copy them, copy the plenty that
143 do use the SB API's instead.
144
145 The reason for this is that our policy is that we will maintain compatibility with the
146 SB API's.  But we don't make any similar guarantee about the details of command result format.
147 If your test is using the command line, it is going to have to check against the command result
148 text, and you either end up writing your check  pattern by checking as little as possible so
149 you won't be exposed to random changes in the text; in which case you can end up missing some
150 failure, or you test too much and it means irrelevant changes break your tests.
151
152 However, if you use the Python API's it is possible to check all the results you want
153 to check in a very explicit way, which makes the tests much more robust.
154
155 Even if you are testing that a command-line command does some specific thing, it is still
156 better in general to use the SB API's to drive to the point where you want to run the test,
157 then use SBInterpreter::HandleCommand to run the command.  You get the full result text
158 from the command in the command return object, and all the part where you are driving the
159 debugger to the point you want to test will be more robust.
160
161o Attaching in test cases:
162
163 If you need to attach to inferiors in your tests, you must make sure the inferior calls
164 lldb_enable_attach(), before the debugger attempts to attach. This function performs any
165 platform-specific processing needed to enable attaching to this process (e.g., on Linux, we
166 execute prctl(PR_SET_TRACER) syscall to disable protections present in some Linux systems).
167