1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
|
.. _classtracker_tutorial:
============================================
Tutorial - Tracking Class Instances in SCons
============================================
This tutorial demonstrates the class tracking facility to profile and optimize a
non-trivial program. SCons_ is a next-generation build system with a quite
elaborate architecture and thus an interesting candidate for profiling attempts.
Before we begin, it should be identified what shall be tracked, i.e. what
classes we want to connect to and whose instances are to be sized and profiled.
In this tutorial, the effect of a patch_ is analyzed that tries to size-optimize the very
heart of SCons - the ``Node`` class. Naturally, we will connect to the ``Node`` base
class and its sub-classes. It makes sense to put the profiling data
in context and track additional classes that are believed to contribute
significantly to the total memory consumption.
Installing hooks into SCons
---------------------------
The first step is to find the proper spots for connecting to the classes that
shall be tracked, taking snapshots, and printing the gathered profile data.
SCons has a simple memory profiling tool that we will override. The SCons
MemStats class provides all we need::
from pympler.classtracker import ClassTracker
class MemStats(Stats):
def __init__(self):
Stats.__init__(self)
classes = [SCons.Node.Node, SCons.Node.FS.Base, SCons.Node.FS.File,
SCons.Node.FS.Dir, SCons.Executor.Executor]
self.tracker = ClassTracker()
for c in classes:
self.tracker.track_class(c)
def do_append(self, label):
self.tracker.create_snapshot(label)
def do_print(self):
stats = self.tracker.stats
stats.print_summary()
stats.dump_stats('pympler.stats')
When SCons starts, ``MemStats`` is instantiated and the `ClassTracker` is
connected to a number of classes. SCons has predefined spots where it invokes
its statistics facilities with ``do_append`` being called. This is where
snapshots will be taken of all objects tracked so far.
Because of the large number of instances, only a summary is printed to the
console via ``stats.print_summary()`` and the profile data is dumped to a file
in case per-instance profile information is needed later.
Test run
--------
Time for a test. In the following examples, SCons builds a non-trivial program
with a fair number of nodes. Running SCons via ``scons --debug=memory`` will
print the gathered data to the console::
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
---- SUMMARY ------------------------------------------------------------------
before reading SConscript files: active 4.17 MB average pct
SCons.Executor.Executor 7 7.53 KB 1.08 KB 0%
SCons.Node.FS.Base 1 9.30 KB 9.30 KB 0%
SCons.Node.FS.Dir 6 17.77 KB 2.96 KB 0%
SCons.Node.FS.File 1 2.91 KB 2.91 KB 0%
SCons.Node.Node 0 0 B 0 B 0%
after reading SConscript files: active 13.06 MB average pct
[...]
before building targets: active 13.41 MB average pct
[...]
after building targets: active 34.77 MB average pct
SCons.Executor.Executor 1311 3.57 MB 2.79 KB 10%
SCons.Node.FS.Base 1102 4.84 MB 4.50 KB 13%
SCons.Node.FS.Dir 108 5.67 MB 53.72 KB 16%
SCons.Node.FS.File 2302 10.45 MB 4.65 KB 30%
SCons.Node.Node 1 84.93 KB 84.93 KB 0%
-------------------------------------------------------------------------------
Making sense of the data
------------------------
The console output may give a brief overview how much memory is allocated by
instances of the individual tracked classes. A more appealing and well arranged
representation of the data can be generated with the ``HtmlStats`` class. The
dump generated previously can be loaded and a set of HTML pages can be emitted::
from pympler.classtracker_stats import HtmlStats
stats = HtmlStats()
stats.load_stats('pympler.stats')
stats.create_html('pympler.html')
If ``matplotlib`` is installed, charts will be embedded in the HTML output:
.. image:: ../images/classtracker_timespace.png
At first sight it might seem suspicious that the tracked classes appear to be
the sole contributors to the total memory footprint of the application. Because
the tracked objects are sized recursively, referenced objects which are not
tracked themselves are added to the referrers account. Thus, a root object's
size will include the size of every leaf unless the leaf is also tracked by the
`ClassTracker`.
Optimization attempt
--------------------
After applying the patch_ by Jean Brouwers, SCons is rerun under the supervision
of the `ClassTracker`. The differences in the last snapshot show that the
changes indeed reduce the memory footprint of ``Node`` instances::
$ scons --debug=memory
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
---- SUMMARY ------------------------------------------------------------------
[...]
after building targets: active 32.41 MB average pct
SCons.Executor.Executor 1311 3.50 MB 2.73 KB 10%
SCons.Node.FS.Base 1102 4.29 MB 3.98 KB 13%
SCons.Node.FS.Dir 108 5.52 MB 52.30 KB 17%
SCons.Node.FS.File 2302 8.82 MB 3.92 KB 27%
SCons.Node.Node 1 84.32 KB 84.32 KB 0%
-------------------------------------------------------------------------------
The total measured memory footprint dropped from 34.8MB to 32.4MB, ``File``
nodes' average size from 4.6KB to 3.9KB.
Summary
-------
This tutorial illustrated how applications can be profiled with the
`ClassTracker` facility. It has been shown how the memory impact of changes
can be quantified.
.. _SCons: http://www.scons.org
.. _patch: http://scons.tigris.org/issues/show_bug.cgi?id=2198
|
