1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
|
Metadata-Version: 1.1
Name: zodbpickle
Version: 1.0
Summary: Fork of Python 3 pickle module.
Home-page: https://github.com/zopefoundation/zodbpickle
Author: Python and Zope Foundation
Author-email: zodb-dev@zope.org
License: PSFL 2 and ZPL 2.1
Description-Content-Type: UNKNOWN
Description: ``zodbpickle`` README
=====================
.. image:: https://travis-ci.org/zopefoundation/zodbpickle.svg?branch=master
:target: https://travis-ci.org/zopefoundation/zodbpickle
.. image:: https://img.shields.io/pypi/v/zodbpickle.svg
:target: https://pypi.python.org/pypi/zodbpickle
:alt: PyPI
.. image:: https://img.shields.io/pypi/pyversions/zodbpickle.svg
:target: https://pypi.python.org/pypi/zodbpickle
:alt: Python versions
This package presents a uniform pickling interface for ZODB:
- Under Python2, this package forks both Python 2.7's ``pickle`` and
``cPickle`` modules, adding support for the ``protocol 3`` opcodes.
It also provides a new subclass of ``bytes``, ``zodbpickle.binary``,
which Python2 applications can use to pickle binary values such that
they will be unpickled as ``bytes`` under Py3k.
- Under Py3k, this package forks the ``pickle`` module (and the supporting
C extension) from both Python 3.2 and Python 3.3. The fork add support
for the ``noload`` operations used by ZODB.
Caution
-------
``zodbpickle`` relies on Python's ``pickle`` module.
The ``pickle`` module is not intended to be secure against erroneous or
maliciously constructed data. Never unpickle data received from an
untrusted or unauthenticated source as arbitrary code might be executed.
Also see https://docs.python.org/3.6/library/pickle.html
General Usage
-------------
To get compatibility between Python 2 and 3 pickling, replace::
import pickle
by::
from zodbpickle import pickle
This provides compatibility, but has the effect that you get the fast implementation
in Python 3, while Python 2 uses the slow version.
To get a more deterministic choice of the implementation, use one of::
from zodbpickle import fastpickle # always C
from zodbpickle import slowpickle # always Python
Both modules can co-exist which is helpful for comparison.
But there is a bit more to consider, so please read on!
Loading/Storing Python 2 Strings
--------------------------------
In all their wisdom, the Python developers have decided that Python 2 ``str``
instances should be loaded as Python 3 ``str`` objects (i.e. unicode
strings). Patches were proposed in Python `issue 6784`__ but were never
applied. This code base contains those patches.
.. __: http://bugs.python.org/issue6784
Example 1: Loading Python 2 pickles on Python 3 ::
$ python2
>>> import pickle
>>> pickle.dumps('\xff', protocol=0)
"S'\\xff'\np0\n."
>>> pickle.dumps('\xff', protocol=1)
'U\x01\xffq\x00.'
>>> pickle.dumps('\xff', protocol=2)
'\x80\x02U\x01\xffq\x00.'
$ python3
>>> from zodbpickle import pickle
>>> pickle.loads(b"S'\\xff'\np0\n.", encoding='bytes')
b'\xff'
>>> pickle.loads(b'U\x01\xffq\x00.', encoding='bytes')
b'\xff'
>>> pickle.loads(b'\x80\x02U\x01\xffq\x00.', encoding='bytes')
b'\xff'
Example 2: Loading Python 3 pickles on Python 2 ::
$ python3
>>> from zodbpickle import pickle
>>> pickle.dumps(b"\xff", protocol=0)
b'c_codecs\nencode\np0\n(V\xff\np1\nVlatin1\np2\ntp3\nRp4\n.'
>>> pickle.dumps(b"\xff", protocol=1)
b'c_codecs\nencode\nq\x00(X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02tq\x03Rq\x04.'
>>> pickle.dumps(b"\xff", protocol=2)
b'\x80\x02c_codecs\nencode\nq\x00X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02\x86q\x03Rq\x04.'
$ python2
>>> import pickle
>>> pickle.loads('c_codecs\nencode\np0\n(V\xff\np1\nVlatin1\np2\ntp3\nRp4\n.')
'\xff'
>>> pickle.loads('c_codecs\nencode\nq\x00(X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02tq\x03Rq\x04.')
'\xff'
>>> pickle.loads('\x80\x02c_codecs\nencode\nq\x00X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02\x86q\x03Rq\x04.')
'\xff'
Example 3: everything breaks down ::
$ python2
>>> class Foo(object):
... def __init__(self):
... self.x = 'hello'
...
>>> import pickle
>>> pickle.dumps(Foo(), protocol=0)
"ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nS'hello'\np7\nsb."
>>> pickle.dumps(Foo(), protocol=1)
'ccopy_reg\n_reconstructor\nq\x00(c__main__\nFoo\nq\x01c__builtin__\nobject\nq\x02Ntq\x03Rq\x04}q\x05U\x01xq\x06U\x05helloq\x07sb.'
>>> pickle.dumps(Foo(), protocol=2)
'\x80\x02c__main__\nFoo\nq\x00)\x81q\x01}q\x02U\x01xq\x03U\x05helloq\x04sb.'
$ python3
>>> from zodbpickle import pickle
>>> class Foo(object): pass
...
>>> foo = pickle.loads("ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nS'hello'\np7\nsb.", encoding='bytes')
>>> foo.x
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Foo' object has no attribute 'x'
wait what? ::
>>> foo.__dict__
{b'x': b'hello'}
oooh. So we use ``encoding='ASCII'`` (the default) and ``errors='bytes'`` and
hope it works::
>>> foo = pickle.loads("ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nS'hello'\np7\nsb.", errors='bytes')
>>> foo.x
'hello'
falling back to bytes if necessary ::
>>> pickle.loads(b'\x80\x02U\x01\xffq\x00.', errors='bytes')
b'\xff'
Support for ``noload()``
------------------------
The ZODB uses `cPickle`'s ``noload()`` method to retrieve all persistent
references from a pickle without loading any objects. This feature was removed
from Python 3's pickle. Unfortuantely, this unnecessarily fills the pickle
cache.
This module provides a ``noload()`` method again.
``zodbpickle`` Changelog
========================
1.0 (2018-02-09)
----------------
- Add a warning to the readme not to use untrusted pickles.
- Drop support for Python 3.3.
0.7.0 (2017-09-22)
------------------
- Drop support for Python 2.6 and 3.2.
- Add support for Jython 2.7.
- Add support for Python 3.5 and 3.6.
0.6.0 (2015-04-02)
------------------
- Restore the ``noload`` behaviour from Python 2.6 and provide the
``noload`` method on the non-C-accelerated unpicklers under PyPy and
Python 2.
- Add support for PyPy, PyPy3, and Python 3.4.
0.5.2 (2013-08-17)
------------------
- Import accelerator from *our* extension module under Py3k.
See https://github.com/zopefoundation/zodbpickle/issues/6,
https://github.com/zopefoundation/zodbpickle/issues/7.
- Fix unpickler's ``load_short_binstring`` across supported platforms.
0.5.1 (2013-07-06)
------------------
- Update all code and tests to Python 2.6.8, 2.7.5, 3.2.5, 3.3.2 .
- Add the modules ``zodbpickle.fastpickle`` and ``zodbpickle.slowpickle``.
This provides a version-independent choice of the C or Python
implementation.
- Fix a minor bug on OS X
0.5.0 (2013-06-14)
------------------
- Removed support for the ``bytes_as_strings`` arguments to pickling APIs:
the pickles created when that argument was true might not be unpickled
without passing ``encoding='bytes'``, which ZODB couldn't reliably enforce.
On Py3k, ZODB will be using ``protocol=3`` pickles anyway.
0.4.4 (2013-06-07)
------------------
- Add protocol 3 opcodes to the C version of the ``noload()`` dispatcher.
0.4.3 (2013-06-07)
------------------
- Packaging error: remove spurious ``-ASIDE`` file from sdist.
0.4.2 (2013-06-07)
------------------
- Fix NameError in pure-Python version of ``Unpickler.noload_appends``.
- Fix NameError in pure-Python version of ``Unpickler.noload_setitems``.
0.4.1 (2013-04-29)
------------------
- Fix typo in Python2 version of ``zodbpickle.pickle`` module.
0.4 (2013-04-28)
----------------
- Support the common pickle module interface for Python 2.6, 2.7, 3.2, and 3.3.
- Split the Python implementations / tests into Python2- and Py3k-specific
variants.
- Added a fork of the Python 2.7 ``_pickle.c``, for use under Python2.
The fork adds support for the Py3k ``protocol 3`` opcodes.
- Added a custom ``binary`` type for use in Python2 apps.
Derived from ``bytes``, the ``binary`` type allows Python2 apps to pickle
binary data using opcodes which will cause it to be unpickled as ``bytes``
on Py3k. Under Py3k, the ``binary`` type is just an alias for ``bytes``.
0.3 (2013-03-18)
----------------
- Added ``noload`` code to Python 3.2 version of ``Unpickler``. As with
the Python 3.3 version, this code remains untested.
- Added ``bytes_as_strings`` option to the Python 3.2 version of
``Pickler``, ``dump``, and ``dumps``.
0.2 (2013-03-05)
----------------
- Added ``bytes_as_strings`` option to ``Pickler``, ``dump``, and ``dumps``.
- Incomplete support for Python 3.2:
- Move ``_pickle.c`` -> ``_pickle_33.c``.
- Clone Python 3.2.3's ``_pickle.c`` -> ``_pickle_32.c`` and apply the
same patch.
- Choose between them at build time based on ``sys.version_info``.
- Disable some tests of 3.3-only features.
- Missing: implementation of ``noload()`` in ``_pickle_32.c``.
- Missing: implementation of ``bytes_as_strings=True`` in ``_pickle_32.c``.
0.1.0 (2013-02-27)
------------------
- Initial release of Python 3.3's pickle with the patches of Python
`issue 6784`__ applied.
.. __: http://bugs.python.org/issue6784#msg156166
- Added support for ``errors="bytes"``.
Keywords: zodb pickle
Platform: any
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: Zope Public License
Classifier: License :: OSI Approved :: Python Software Foundation License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Programming Language :: Python :: Implementation :: Jython
Classifier: Framework :: ZODB
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: Unix
Classifier: Operating System :: MacOS :: MacOS X
|
