1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
|
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<HTML>
<HEAD>
<link rel="stylesheet" href="designstyle.css">
<title>Gperftools Heap Profiler</title>
</HEAD>
<BODY>
<p align=right>
<i>Last modified
<script type=text/javascript>
var lm = new Date(document.lastModified);
document.write(lm.toDateString());
</script></i>
</p>
<p>This is the heap profiler we use at Google, to explore how C++
programs manage memory. This facility can be useful for</p>
<ul>
<li> Figuring out what is in the program heap at any given time
<li> Locating memory leaks
<li> Finding places that do a lot of allocation
</ul>
<p>The profiling system instruments all allocations and frees. It
keeps track of various pieces of information per allocation site. An
allocation site is defined as the active stack trace at the call to
<code>malloc</code>, <code>calloc</code>, <code>realloc</code>, or,
<code>new</code>.</p>
<p>There are three parts to using it: linking the library into an
application, running the code, and analyzing the output.</p>
<h1>Linking in the Library</h1>
<p>To install the heap profiler into your executable, add
<code>-ltcmalloc</code> to the link-time step for your executable.
Also, while we don't necessarily recommend this form of usage, it's
possible to add in the profiler at run-time using
<code>LD_PRELOAD</code>:
<pre>% env LD_PRELOAD="/usr/lib/libtcmalloc.so" <binary></pre>
<p>This does <i>not</i> turn on heap profiling; it just inserts the
code. For that reason, it's practical to just always link
<code>-ltcmalloc</code> into a binary while developing; that's what we
do at Google. (However, since any user can turn on the profiler by
setting an environment variable, it's not necessarily recommended to
install profiler-linked binaries into a production, running
system.) Note that if you wish to use the heap profiler, you must
also use the tcmalloc memory-allocation library. There is no way
currently to use the heap profiler separate from tcmalloc.</p>
<h1>Running the Code</h1>
<p>There are several alternatives to actually turn on heap profiling
for a given run of an executable:</p>
<ol>
<li> <p>Define the environment variable HEAPPROFILE to the filename
to dump the profile to. For instance, to profile
<code>/usr/local/bin/my_binary_compiled_with_tcmalloc</code>:</p>
<pre>% env HEAPPROFILE=/tmp/mybin.hprof /usr/local/bin/my_binary_compiled_with_tcmalloc</pre>
<li> <p>In your code, bracket the code you want profiled in calls to
<code>HeapProfilerStart()</code> and <code>HeapProfilerStop()</code>.
(These functions are declared in <code><gperftools/heap-profiler.h></code>.)
<code>HeapProfilerStart()</code> will take the
profile-filename-prefix as an argument. Then, as often as
you'd like before calling <code>HeapProfilerStop()</code>, you
can use <code>HeapProfilerDump()</code> or
<code>GetHeapProfile()</code> to examine the profile. In case
it's useful, <code>IsHeapProfilerRunning()</code> will tell you
whether you've already called HeapProfilerStart() or not.</p>
</ol>
<p>For security reasons, heap profiling will not write to a file --
and is thus not usable -- for setuid programs.</p>
<H2>Modifying Runtime Behavior</H2>
<p>You can more finely control the behavior of the heap profiler via
environment variables.</p>
<table frame=box rules=sides cellpadding=5 width=100%>
<tr valign=top>
<td><code>HEAP_PROFILE_ALLOCATION_INTERVAL</code></td>
<td>default: 1073741824 (1 Gb)</td>
<td>
Dump heap profiling information each time the specified number of
bytes has been allocated by the program.
</td>
</tr>
<tr valign=top>
<td><code>HEAP_PROFILE_INUSE_INTERVAL</code></td>
<td>default: 104857600 (100 Mb)</td>
<td>
Dump heap profiling information whenever the high-water memory
usage mark increases by the specified number of bytes.
</td>
</tr>
<tr valign=top>
<td><code>HEAP_PROFILE_TIME_INTERVAL</code></td>
<td>default: 0</td>
<td>
Dump heap profiling information each time the specified
number of seconds has elapsed.
</td>
</tr>
<tr valign=top>
<td><code>HEAPPROFILESIGNAL</code></td>
<td>default: disabled</td>
<td>
Dump heap profiling information whenever the specified signal is sent to the
process.
</td>
</tr>
<tr valign=top>
<td><code>HEAP_PROFILE_MMAP</code></td>
<td>default: false</td>
<td>
Profile <code>mmap</code>, <code>mremap</code> and <code>sbrk</code>
calls in addition
to <code>malloc</code>, <code>calloc</code>, <code>realloc</code>,
and <code>new</code>. <b>NOTE:</b> this causes the profiler to
profile calls internal to tcmalloc, since tcmalloc and friends use
mmap and sbrk internally for allocations. One partial solution is
to filter these allocations out when running <code>pprof</code>,
with something like
<code>pprof --ignore='DoAllocWithArena|SbrkSysAllocator::Alloc|MmapSysAllocator::Alloc</code>.
</td>
</tr>
<tr valign=top>
<td><code>HEAP_PROFILE_ONLY_MMAP</code></td>
<td>default: false</td>
<td>
Only profile <code>mmap</code>, <code>mremap</code>, and <code>sbrk</code>
calls; do not profile
<code>malloc</code>, <code>calloc</code>, <code>realloc</code>,
or <code>new</code>.
</td>
</tr>
<tr valign=top>
<td><code>HEAP_PROFILE_MMAP_LOG</code></td>
<td>default: false</td>
<td>
Log <code>mmap</code>/<code>munmap</code> calls.
</td>
</tr>
</table>
<H2>Checking for Leaks</H2>
<p>You can use the heap profiler to manually check for leaks, for
instance by reading the profiler output and looking for large
allocations. However, for that task, it's easier to use the <A
HREF="heap_checker.html">automatic heap-checking facility</A> built
into tcmalloc.</p>
<h1><a name="pprof">Analyzing the Output</a></h1>
<p>If heap-profiling is turned on in a program, the program will
periodically write profiles to the filesystem. The sequence of
profiles will be named:</p>
<pre>
<prefix>.0000.heap
<prefix>.0001.heap
<prefix>.0002.heap
...
</pre>
<p>where <code><prefix></code> is the filename-prefix supplied
when running the code (e.g. via the <code>HEAPPROFILE</code>
environment variable). Note that if the supplied prefix
does not start with a <code>/</code>, the profile files will be
written to the program's working directory.</p>
<p>The profile output can be viewed by passing it to the
<code>pprof</code> tool -- the same tool that's used to analyze <A
HREF="cpuprofile.html">CPU profiles</A>.
<p>Here are some examples. These examples assume the binary is named
<code>gfs_master</code>, and a sequence of heap profile files can be
found in files named:</p>
<pre>
/tmp/profile.0001.heap
/tmp/profile.0002.heap
...
/tmp/profile.0100.heap
</pre>
<h3>Why is a process so big</h3>
<pre>
% pprof --gv gfs_master /tmp/profile.0100.heap
</pre>
<p>This command will pop-up a <code>gv</code> window that displays
the profile information as a directed graph. Here is a portion
of the resulting output:</p>
<p><center>
<img src="heap-example1.png">
</center></p>
A few explanations:
<ul>
<li> <code>GFS_MasterChunk::AddServer</code> accounts for 255.6 MB
of the live memory, which is 25% of the total live memory.
<li> <code>GFS_MasterChunkTable::UpdateState</code> is directly
accountable for 176.2 MB of the live memory (i.e., it directly
allocated 176.2 MB that has not been freed yet). Furthermore,
it and its callees are responsible for 729.9 MB. The
labels on the outgoing edges give a good indication of the
amount allocated by each callee.
</ul>
<h3>Comparing Profiles</h3>
<p>You often want to skip allocations during the initialization phase
of a program so you can find gradual memory leaks. One simple way to
do this is to compare two profiles -- both collected after the program
has been running for a while. Specify the name of the first profile
using the <code>--base</code> option. For example:</p>
<pre>
% pprof --base=/tmp/profile.0004.heap gfs_master /tmp/profile.0100.heap
</pre>
<p>The memory-usage in <code>/tmp/profile.0004.heap</code> will be
subtracted from the memory-usage in
<code>/tmp/profile.0100.heap</code> and the result will be
displayed.</p>
<h3>Text display</h3>
<pre>
% pprof --text gfs_master /tmp/profile.0100.heap
255.6 24.7% 24.7% 255.6 24.7% GFS_MasterChunk::AddServer
184.6 17.8% 42.5% 298.8 28.8% GFS_MasterChunkTable::Create
176.2 17.0% 59.5% 729.9 70.5% GFS_MasterChunkTable::UpdateState
169.8 16.4% 75.9% 169.8 16.4% PendingClone::PendingClone
76.3 7.4% 83.3% 76.3 7.4% __default_alloc_template::_S_chunk_alloc
49.5 4.8% 88.0% 49.5 4.8% hashtable::resize
...
</pre>
<p>
<ul>
<li> The first column contains the direct memory use in MB.
<li> The fourth column contains memory use by the procedure
and all of its callees.
<li> The second and fifth columns are just percentage
representations of the numbers in the first and fourth columns.
<li> The third column is a cumulative sum of the second column
(i.e., the <code>k</code>th entry in the third column is the
sum of the first <code>k</code> entries in the second column.)
</ul>
<h3>Ignoring or focusing on specific regions</h3>
<p>The following command will give a graphical display of a subset of
the call-graph. Only paths in the call-graph that match the regular
expression <code>DataBuffer</code> are included:</p>
<pre>
% pprof --gv --focus=DataBuffer gfs_master /tmp/profile.0100.heap
</pre>
<p>Similarly, the following command will omit all paths subset of the
call-graph. All paths in the call-graph that match the regular
expression <code>DataBuffer</code> are discarded:</p>
<pre>
% pprof --gv --ignore=DataBuffer gfs_master /tmp/profile.0100.heap
</pre>
<h3>Total allocations + object-level information</h3>
<p>All of the previous examples have displayed the amount of in-use
space. I.e., the number of bytes that have been allocated but not
freed. You can also get other types of information by supplying a
flag to <code>pprof</code>:</p>
<center>
<table frame=box rules=sides cellpadding=5 width=100%>
<tr valign=top>
<td><code>--inuse_space</code></td>
<td>
Display the number of in-use megabytes (i.e. space that has
been allocated but not freed). This is the default.
</td>
</tr>
<tr valign=top>
<td><code>--inuse_objects</code></td>
<td>
Display the number of in-use objects (i.e. number of
objects that have been allocated but not freed).
</td>
</tr>
<tr valign=top>
<td><code>--alloc_space</code></td>
<td>
Display the number of allocated megabytes. This includes
the space that has since been de-allocated. Use this
if you want to find the main allocation sites in the
program.
</td>
</tr>
<tr valign=top>
<td><code>--alloc_objects</code></td>
<td>
Display the number of allocated objects. This includes
the objects that have since been de-allocated. Use this
if you want to find the main allocation sites in the
program.
</td>
</table>
</center>
<h3>Interactive mode</a></h3>
<p>By default -- if you don't specify any flags to the contrary --
pprof runs in interactive mode. At the <code>(pprof)</code> prompt,
you can run many of the commands described above. You can type
<code>help</code> for a list of what commands are available in
interactive mode.</p>
<h1>Caveats</h1>
<ul>
<li> Heap profiling requires the use of libtcmalloc. This
requirement may be removed in a future version of the heap
profiler, and the heap profiler separated out into its own
library.
<li> If the program linked in a library that was not compiled
with enough symbolic information, all samples associated
with the library may be charged to the last symbol found
in the program before the library. This will artificially
inflate the count for that symbol.
<li> If you run the program on one machine, and profile it on
another, and the shared libraries are different on the two
machines, the profiling output may be confusing: samples that
fall within the shared libaries may be assigned to arbitrary
procedures.
<li> Several libraries, such as some STL implementations, do their
own memory management. This may cause strange profiling
results. We have code in libtcmalloc to cause STL to use
tcmalloc for memory management (which in our tests is better
than STL's internal management), though it only works for some
STL implementations.
<li> If your program forks, the children will also be profiled
(since they inherit the same HEAPPROFILE setting). Each
process is profiled separately; to distinguish the child
profiles from the parent profile and from each other, all
children will have their process-id attached to the HEAPPROFILE
name.
<li> Due to a hack we make to work around a possible gcc bug, your
profiles may end up named strangely if the first character of
your HEAPPROFILE variable has ascii value greater than 127.
This should be exceedingly rare, but if you need to use such a
name, just set prepend <code>./</code> to your filename:
<code>HEAPPROFILE=./Ägypten</code>.
</ul>
<hr>
<address>Sanjay Ghemawat
<!-- Created: Tue Dec 19 10:43:14 PST 2000 -->
</address>
</body>
</html>
|
