1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
|
``ioutils`` - Input/output enhancements
=======================================
.. automodule:: boltons.ioutils
Spooled Temporary Files
-----------------------
Spooled Temporary Files are file-like objects that start out mapped to
in-memory objects, but automatically roll over to a temporary file once they
reach a certain (configurable) threshold. Unfortunately the built-in
SpooledTemporaryFile class in Python does not implement the exact API that some
common classes like StringIO do. SpooledTemporaryFile also spools all of it's
in-memory files as cStringIO instances. cStringIO instances cannot be
deep-copied, and they don't work with the zip library either. This along with
the incompatible api makes it useless for several use-cases.
To combat this but still gain the memory savings and usefulness of a true
spooled file-like-object, two custom classes have been implemented which have
a compatible API.
.. _spooledbytesio:
SpooledBytesIO
^^^^^^^^^^^^^^
.. autoclass:: boltons.ioutils.SpooledBytesIO
.. _spooledstringio:
SpooledStringIO
^^^^^^^^^^^^^^^
.. autoclass:: boltons.ioutils.SpooledStringIO
Examples
--------
It's not uncommon to find excessive usage of StringIO in older Python code. A
SpooledTemporaryFile would be a nice replacement if one wanted to reduce memory
overhead, but unfortunately its api differs too much. This is a good candidate
for :ref:`spooledbytesio` as it is api compatible and thus may be used as a
drop-in replacement.
Old Code::
flo = StringIO()
flo.write(gigantic_string)
Updated::
from boltons.ioutils import SpooledBytesIO
flo = SpooledBytesIO()
flo.write(gigantic_string)
Another good use case is downloading a file from some remote location. It's
nice to keep it in memory if it's small, but writing a large file into memory
can make servers quite grumpy. If the file being downloaded happens to be a zip
file then things are worse. You can't use a normal SpooledTemporaryFile because
it isn't compatible. A :ref:`spooledbytesio` instance is a good alternative.
Here is a simple example using the requests library to download a zip file::
from zipfile import ZipFile
import requests
from boltons import ioutils
# Using a context manager with stream=True ensures the connection is closed. See:
# http://docs.python-requests.org/en/master/user/advanced/#body-content-workflow
with requests.get("http://127.0.0.1/test_file.zip", stream=True) as r:
if r.status_code == 200:
with ioutils.SpooledBytesIO() as flo:
for chunk in r.iter_content(chunk_size=64000):
flo.write(chunk)
flo.seek(0)
zip_doc = ZipFile(flo)
# Print all the files in the zip
print(zip_doc.namelist())
Multiple Files
--------------
.. _multifilereader:
MultiFileReader
^^^^^^^^^^^^^^^
.. autoclass:: boltons.ioutils.MultiFileReader
|
