File: README.rst

package info (click to toggle)
python-memory-profiler 0.41%2Bgit20161018-1
  • links: PTS, VCS
  • area: main
  • in suites: stretch
  • size: 280 kB
  • ctags: 163
  • sloc: python: 1,469; makefile: 24
file content (418 lines) | stat: -rw-r--r-- 14,528 bytes parent folder | download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
.. image:: https://travis-ci.org/fabianp/memory_profiler.svg?branch=master
    :target: https://travis-ci.org/fabianp/memory_profiler

=================
 Memory Profiler
=================

This is a python module for monitoring memory consumption of a process
as well as line-by-line analysis of memory consumption for python
programs. It is a pure python module and has the `psutil
<http://pypi.python.org/pypi/psutil>`_ module as optional (but highly
recommended) dependencies.


==============
 Installation
==============
To install through easy_install or pip::

    $ easy_install -U memory_profiler # pip install -U memory_profiler

To install from source, download the package, extract and type::

    $ python setup.py install


=======
 Usage
=======


line-by-line memory usage
=========================

The line-by-line memory usage mode is used much in the same way of the
`line_profiler <https://pypi.python.org/pypi/line_profiler/>`_: first
decorate the function you would like to profile with ``@profile`` and
then run the script with a special script (in this case with specific
arguments to the Python interpreter).

In the following example, we create a simple function ``my_func`` that
allocates lists ``a``, ``b`` and then deletes ``b``::


    @profile
    def my_func():
        a = [1] * (10 ** 6)
        b = [2] * (2 * 10 ** 7)
        del b
        return a

    if __name__ == '__main__':
        my_func()


Execute the code passing the option ``-m memory_profiler`` to the
python interpreter to load the memory_profiler module and print to
stdout the line-by-line analysis. If the file name was example.py,
this would result in::

    $ python -m memory_profiler example.py

Output will follow::

    Line #    Mem usage  Increment   Line Contents
    ==============================================
         3                           @profile
         4      5.97 MB    0.00 MB   def my_func():
         5     13.61 MB    7.64 MB       a = [1] * (10 ** 6)
         6    166.20 MB  152.59 MB       b = [2] * (2 * 10 ** 7)
         7     13.61 MB -152.59 MB       del b
         8     13.61 MB    0.00 MB       return a


The first column represents the line number of the code that has been
profiled, the second column (*Mem usage*) the memory usage of the
Python interpreter after that line has been executed. The third column
(*Increment*) represents the difference in memory of the current line
with respect to the last one. The last column (*Line Contents*) prints
the code that has been profiled.

Decorator
=========
A function decorator is also available.  Use as follows::

    from memory_profiler import profile

    @profile
    def my_func():
        a = [1] * (10 ** 6)
        b = [2] * (2 * 10 ** 7)
        del b
        return a

In this case the script can be run without specifying ``-m
memory_profiler`` in the command line.

In function decorator, you can specify the precision as an argument to the
decorator function.  Use as follows::

    from memory_profiler import profile

    @profile(precision=4)
    def my_func():
        a = [1] * (10 ** 6)
        b = [2] * (2 * 10 ** 7)
        del b
        return a

If a python script with decorator ``@profile`` is called using ``-m 
memory_profiler`` in the command line, the ``precision`` parameter is ignored.

Time-based memory usage 
==========================
Sometimes it is useful to have full memory usage reports as a function of
time (not line-by-line) of external processes (be it Python scripts or not).
In this case the executable ``mprof`` might be useful. Use it like::

    mprof run <executable>
    mprof plot

The first line run the executable and record memory usage along time,
in a file written in the current directory.
Once it's done, a graph plot can be obtained using the second line.
The recorded file contains a timestamps, that allows for several
profiles to be kept at the same time.

Help on each `mprof` subcommand can be obtained with the `-h` flag,
e.g. `mprof run -h`.

In the case of a Python script, using the previous command does not
give you any information on which function is executed at a given
time. Depending on the case, it can be difficult to identify the part
of the code that is causing the highest memory usage. 

Adding the `profile` decorator to a function and running the Python
script with 

    mprof run <script>

will record timestamps when entering/leaving the profiled function. Runnning

    mprof plot

afterward will plot the result, making plots (using matplotlib) similar to these:

.. image:: https://camo.githubusercontent.com/3a584c7cfbae38c9220a755aa21b5ef926c1031d/68747470733a2f2f662e636c6f75642e6769746875622e636f6d2f6173736574732f313930383631382f3836313332302f63623865376337382d663563632d313165322d386531652d3539373237623636663462322e706e67
   :target: https://github.com/scikit-learn/scikit-learn/pull/2248
   :height: 350px

A discussion of these capabilities can be found `here <http://fa.bianp.net/blog/2014/plot-memory-usage-as-a-function-of-time/>`_.

.. warning:: If your Python file imports the memory profiler `from memory_profiler import profile` these timestamps will not be recorded. Comment out the import, leave your functions decorated, and re-run.

The available commands for `mprof` are: 

  - ``mprof run``: running an executable, recording memory usage  
  - ``mprof plot``: plotting one the recorded memory usage (by default,
    the last one)
  - ``mprof list``: listing all recorded memory usage files in a
    user-friendly way.
  - ``mprof clean``: removing all recorded memory usage files.
  - ``mprof rm``: removing specific recorded memory usage files

Setting debugger breakpoints
=============================
It is possible to set breakpoints depending on the amount of memory used.
That is, you can specify a threshold and as soon as the program uses more
memory than what is specified in the threshold it will stop execution
and run into the pdb debugger. To use it, you will have to decorate
the function as done in the previous section with ``@profile`` and then
run your script with the option ``-m memory_profiler --pdb-mmem=X``,
where X is a number representing the memory threshold in MB. For example::

    $ python -m memory_profiler --pdb-mmem=100 my_script.py

will run ``my_script.py`` and step into the pdb debugger as soon as the code
uses more than 100 MB in the decorated function.

.. TODO: alternatives to decoration (for example when you don't want to modify
    the file where your function lives).

=====
 API
=====
memory_profiler exposes a number of functions to be used in third-party
code.



``memory_usage(proc=-1, interval=.1, timeout=None)`` returns the memory usage
over a time interval. The first argument, ``proc`` represents what
should be monitored.  This can either be the PID of a process (not
necessarily a Python program), a string containing some python code to
be evaluated or a tuple ``(f, args, kw)`` containing a function and its
arguments to be evaluated as ``f(*args, **kw)``. For example,


    >>> from memory_profiler import memory_usage
    >>> mem_usage = memory_usage(-1, interval=.2, timeout=1)
    >>> print(mem_usage)
	[7.296875, 7.296875, 7.296875, 7.296875, 7.296875]


Here I've told memory_profiler to get the memory consumption of the
current process over a period of 1 second with a time interval of 0.2
seconds. As PID I've given it -1, which is a special number (PIDs are
usually positive) that means current process, that is, I'm getting the
memory usage of the current Python interpreter. Thus I'm getting
around 7MB of memory usage from a plain python interpreter. If I try
the same thing on IPython (console) I get 29MB, and if I try the same
thing on the IPython notebook it scales up to 44MB.


If you'd like to get the memory consumption of a Python function, then
you should specify the function and its arguments in the tuple ``(f,
args, kw)``. For example::


    >>> # define a simple function
    >>> def f(a, n=100):
        ...     import time
        ...     time.sleep(2)
        ...     b = [a] * n
        ...     time.sleep(1)
        ...     return b
        ...
    >>> from memory_profiler import memory_usage
    >>> memory_usage((f, (1,), {'n' : int(1e6)}))

This will execute the code `f(1, n=int(1e6))` and return the memory
consumption during this execution.

=========
REPORTING
=========

The output can be redirected to a log file by passing IO stream as
parameter to the decorator like @profile(stream=fp)

    >>> fp=open('memory_profiler.log','w+')
    >>> @profile(stream=fp)
    >>> def my_func():
        ...     a = [1] * (10 ** 6)
        ...     b = [2] * (2 * 10 ** 7)
        ...     del b
        ...     return a

    For details refer: examples/reporting_file.py

``Reporting via logger Module:``

Sometime it would be very convenient to use logger module specially
when we need to use RotatingFileHandler.

The output can be redirected to logger module by simply making use of
LogFile of memory profiler module.

    >>> from memory_profiler import LogFile
    >>> import sys
    >>> sys.stdout = LogFile('memory_profile_log')

``Customised reporting:``

Sending everything to the log file while running the memory_profiler
could be cumbersome and one can choose only entries with increments
by passing True to reportIncrementFlag, where reportIncrementFlag is
a parameter to LogFile class of memory profiler module.

    >>> from memory_profiler import LogFile
    >>> import sys
    >>> sys.stdout = LogFile('memory_profile_log', reportIncrementFlag=False)

    For details refer: examples/reporting_logger.py

=====================
 IPython integration
=====================
After installing the module, if you use IPython, you can use the `%mprun`, `%%mprun`,
`%memit` and `%%memit` magics.

For IPython 0.11+, you can use the module directly as an extension, with
``%load_ext memory_profiler``

To activate it whenever you start IPython, edit the configuration file for your
IPython profile, ~/.ipython/profile_default/ipython_config.py, to register the
extension like this (If you already have other extensions, just add this one to
the list)::

    c.InteractiveShellApp.extensions = [
        'memory_profiler',
    ]

(If the config file doesn't already exist, run ``ipython profile create`` in
a terminal.)

It then can be used directly from IPython to obtain a line-by-line
report using the `%mprun` or `%%mprun` magic command. In this case, you can skip
the `@profile` decorator and instead use the `-f` parameter, like
this. Note however that function my_func must be defined in a file
(cannot have been defined interactively in the Python interpreter)::

    In [1]: from example import my_func, my_func_2

    In [2]: %mprun -f my_func my_func()

or in cell mode::

    In [3]: %%mprun -f my_func -f my_func_2
       ...: my_func()
       ...: my_func_2()

Another useful magic that we define is `%memit`, which is analogous to
`%timeit`. It can be used as follows::

    In [1]: %memit range(10000)
    peak memory: 21.42 MiB, increment: 0.41 MiB

    In [2]: %memit range(1000000)
    peak memory: 52.10 MiB, increment: 31.08 MiB

or in cell mode (with setup code)::

    In [3]: %%memit l=range(1000000)
       ...: len(l)
       ...:
    peak memory: 52.14 MiB, increment: 0.08 MiB

For more details, see the docstrings of the magics.

For IPython 0.10, you can install it by editing the IPython configuration
file ~/.ipython/ipy_user_conf.py to add the following lines::

    # These two lines are standard and probably already there.
    import IPython.ipapi
    ip = IPython.ipapi.get()

    # These two are the important ones.
    import memory_profiler
    memory_profiler.load_ipython_extension(ip)

============================
 Frequently Asked Questions
============================
    * Q: How accurate are the results ?
    * A: This module gets the memory consumption by querying the
      operating system kernel about the amount of memory the current
      process has allocated, which might be slightly different from
      the amount of memory that is actually used by the Python
      interpreter. Also, because of how the garbage collector works in
      Python the result might be different between platforms and even
      between runs.

    * Q: Does it work under windows ?
    * A: Yes, but you will need the
      `psutil <http://pypi.python.org/pypi/psutil>`_ module.



===========================
 Support, bugs & wish list
===========================
For support, please ask your question on `stack overflow
<http://stackoverflow.com/>`_ and add the `*memory-profiling* tag <http://stackoverflow.com/questions/tagged/memory-profiling>`_.
Send issues, proposals, etc. to `github's issue tracker
<https://github.com/fabianp/memory_profiler/issues>`_ .

If you've got questions regarding development, you can email me
directly at fabian@fseoane.net

.. image:: http://fseoane.net/static/tux_memory_small.png


=============
 Development
=============
Latest sources are available from github:

    https://github.com/fabianp/memory_profiler

===============================
Projects using memory_profiler
===============================

`Benchy <https://github.com/python-recsys/benchy>`_

`IPython memory usage <https://github.com/ianozsvald/ipython_memory_usage_>`_

`PySpeedIT <https://github.com/peter1000/PySpeedIT>`_ (uses a reduced version of memory_profiler)

`pydio-sync <https://github.com/pydio/pydio-sync>`_ (uses custom wrapper on top of memory_profiler)

=========
 Authors
=========
This module was written by `Fabian Pedregosa <http://fseoane.net>`_
and `Philippe Gervais <https://github.com/pgervais>`_
inspired by Robert Kern's `line profiler
<http://packages.python.org/line_profiler/>`_.

`Tom <http://tomforb.es/>`_ added windows support and speed improvements via the
`psutil <http://pypi.python.org/pypi/psutil>`_ module.

`Victor <https://github.com/octavo>`_ added python3 support, bugfixes and general
cleanup.

`Vlad Niculae <http://vene.ro/>`_ added the `%mprun` and `%memit` IPython magics.

`Thomas Kluyver <https://github.com/takluyver>`_ added the IPython extension.

`Sagar UDAY KUMAR <https://github.com/sagaru>`_ added Report generation feature and examples.

`Dmitriy Novozhilov <https://github.com/demiurg906>`_ and `Sergei Lebedev <https://github.com/superbobry>`_ added support for `tracemalloc <https://docs.python.org/3/library/tracemalloc.html>`_.

=========
 License
=========
BSD License, see file COPYING for full text.