1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
|
.. _developer-debugging:
==================
Notes on Debugging
==================
This section describes techniques that can be useful in debugging the
compilation and execution of generated code.
.. seealso::
:ref:`debugging-jit-compiled-code`
Memcheck
--------
Memcheck_ is a memory error detector implemented using Valgrind_. It is useful
for detecting memory errors in compiled code, particularly out-of-bounds
accesses and use-after-free errors. Buggy or miscompiled native code can
generate these kinds of errors. The `Memcheck documentation
<https://valgrind.org/docs/manual/mc-manual.html>`_ explains its usage; here, we
discuss only the specifics of using it with Numba.
.. _Memcheck: https://valgrind.org/docs/manual/mc-manual.html
.. _Valgrind: https://valgrind.org/
The Python interpreter and some of the libraries used by Numba can generate
false positives with Memcheck - see `this section of the manual
<https://valgrind.org/docs/manual/mc-manual.html#mc-manual.machine>`_ for more
information on why false positives occur. The false positives can make it
difficult to determine when an actual error has occurred, so it is helpful to
suppress known false positives. This can be done by supplying a suppressions
file, which instructs Memcheck to ignore errors that match the suppressions
defined in it.
The CPython source distribution includes a suppressions file, in the file
``Misc/valgrind-python.supp``. Using this file prevents a lot of spurious errors
generated by Python's memory allocation implementation. Additionally, the Numba
repository includes a suppressions file in ``contrib/valgrind-numba.supp``.
.. note:: It is important to use the suppressions files from the versions of the
Python interpreter and Numba that you are using - these files evolve over
time, so non-current versions can fail to suppress some errors, or
erroneously suppress actual errors.
To run the Python interpreter under Memcheck with both suppressions
files, it is invoked with the following command::
valgrind --tool=memcheck \
--suppressions=${CPYTHON_SRC_DIR}/Misc/valgrind-python.supp \
--suppressions=${NUMBA_SRC_DIR}/contrib/valgrind-numba.supp \
python ${PYTHON_ARGS}
where ``${CPYTHON_SRC_DIR}`` is set to the location of the CPython source
distribution, ``${NUMBA_SRC_DIR}`` is the location of the Numba source dir, and
``${PYTHON_ARGS}`` are the arguments to the Python interpreter.
If there are errors, then messages describing them will be printed to standard
error. An example of an error is::
==77113== at 0x24169A: PyLong_FromLong (longobject.c:251)
==77113== by 0x241881: striter_next (bytesobject.c:3084)
==77113== by 0x2D3C95: _PyEval_EvalFrameDefault (ceval.c:2809)
==77113== by 0x21B499: _PyEval_EvalCodeWithName (ceval.c:3930)
==77113== by 0x26B436: _PyFunction_FastCallKeywords (call.c:433)
==77113== by 0x2D3605: call_function (ceval.c:4616)
==77113== by 0x2D3605: _PyEval_EvalFrameDefault (ceval.c:3124)
==77113== by 0x21B977: _PyEval_EvalCodeWithName (ceval.c:3930)
==77113== by 0x21C2A4: _PyFunction_FastCallDict (call.c:376)
==77113== by 0x2D5129: do_call_core (ceval.c:4645)
==77113== by 0x2D5129: _PyEval_EvalFrameDefault (ceval.c:3191)
==77113== by 0x21B499: _PyEval_EvalCodeWithName (ceval.c:3930)
==77113== by 0x26B436: _PyFunction_FastCallKeywords (call.c:433)
==77113== by 0x2D46DA: call_function (ceval.c:4616)
==77113== by 0x2D46DA: _PyEval_EvalFrameDefault (ceval.c:3139)
==77113==
==77113== Use of uninitialised value of size 8
The traceback provided only outlines the C call stack, which can make it
difficult to determine what the Python interpreter was doing at the time of the
error. One can learn more about the state of the stack by looking at the
backtrace in the `GNU Debugger (GDB) <https://www.gnu.org/software/gdb/>`_.
Launch ``valgrind`` with an additional argument, ``--vgdb-error=0`` and attach
to the process using GDB as instructed by the output. Once an error is
encountered, GDB will stop at the error and the stack can be inspected.
GDB does provide support for backtracing through the Python stack, but this
requires symbols which may not be easily available in your Python distribution.
In this case, it is still possible to determine some information about what was
happening in Python, but this depends on examining the backtrace closely. For
example, in a backtrace corresponding to the above error, we see items such as:
.. code-block::
#18 0x00000000002722da in slot_tp_call (
self=<_wrap_impl(_callable=<_wrap_missing_loc(func=<function at remote
0x1cf66c20>) at remote 0x1d200bd0>, _imp=<function at remote 0x1d0e7440>,
_context=<CUDATargetContext(address_size=64,
typing_context=<CUDATypingContext(_registries={<Registry(functions=[<type
at remote 0x65be5e0>, <type at remote 0x65be9d0>, <type at remote
0x65bedc0>, <type at remote 0x65bf1b0>, <type at remote 0x8b78000>, <type
at remote 0x8b783f0>, <type at remote 0x8b787e0>, <type at remote
0x8b78bd0>, <type at remote 0x8b78fc0>, <type at remote 0x8b793b0>, <type
at remote 0x8b797a0>, <type at remote 0x8b79b90>, <type at remote
0x8b79f80>, <type at remote 0x8b7a370>, <type at remote 0x8b7a760>, <type
at remote 0x8b7ab50>, <type at remote 0x8b7af40>, <type at remote
0x8b7b330>, <type at remote 0x8b7b720>, <type at remote 0x8b7bf00>, <type
at remote 0x8b7c2f0>, <type at remote 0x8b7c6e0>], attributes=[<type at
remote 0x8b7cad0>, <type at remote 0x8b7cec0>, <type at remote
0x8b7d2b0>, <type at remote 0x8b7d6a0>, <type at remote 0x8b7da90>,
<t...(truncated),
args=(<Builder(_block=<Block(parent=<Function(parent=<Module(context=<Context(scope=<NameScope(_useset={''},
_basenamemap={}) at remote 0xbb5ae10>, identified_types={}) at remote
0xbb5add0>, name='cuconstRecAlign$7',
data_layout='e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v32:32:32-v64:64:64-v128:128:128-n16:32:64',
scope=<NameScope(_useset={'',
'_ZN08NumbaEnv5numba4cuda5tests6cudapy13test_constmem19cuconstRecAlign$247E5ArrayIdLi1E1C7mutable7alignedE5ArrayIdLi1E1C7mutable7alignedE5ArrayIdLi1E1C7mutable7alignedE5ArrayIdLi1E1C7mutable7alignedE5ArrayIdLi1E1C7mutable7alignedE',
'_ZN5numba4cuda5tests6cudapy13test_constmem19cuconstRecAlign$247E5ArrayIdLi1E1C7mutable7alignedE5ArrayIdLi1E1C7mutable7alignedE5ArrayIdLi1E1C7mutable7alignedE5ArrayIdLi1E1C7mutable7alignedE5ArrayIdLi1E1C7mutable7alignedE'},
_basenamemap={}) at remote 0x1d27bf10>, triple='nvptx64-nvidia-cuda',
globals={'_ZN08NumbaEnv5numba4cuda5tests6cudapy13test_constmem19cuconstRecAlign$247E5ArrayIdLi1E1C7mutable7ali...(truncated),
kwds=0x0)
We can see some of the arguments, in particular the names of the compiled functions, e.g::
_ZN5numba4cuda5tests6cudapy13test_constmem19cuconstRecAlign$247E5ArrayIdLi1E1C7mutable7alignedE5ArrayIdLi1E1C7mutable7alignedE5ArrayIdLi1E1C7mutable7alignedE5ArrayIdLi1E1C7mutable7alignedE5ArrayIdLi1E1C7mutable7alignedE
We can run this through ``c++filt`` to see a more human-readable representation::
numba::cuda::tests::cudapy::test_constmem::cuconstRecAlign$247(
Array<double, 1, C, mutable, aligned>,
Array<double, 1, C, mutable, aligned>,
Array<double, 1, C, mutable, aligned>,
Array<double, 1, C, mutable, aligned>,
Array<double, 1, C, mutable, aligned>)
which is the fully qualified name of a jitted function and the types with which
it was called.
|
