1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
|
.. _uploading-data:
Uploading Data
==============
Streaming Multipart Data Encoder
--------------------------------
Requests has `support for multipart uploads`_, but the API means that using
that functionality to build exactly the Multipart upload you want can be
difficult or impossible. Additionally, when using Requests' Multipart upload
functionality all the data must be read into memory before being sent to the
server. In extreme cases, this can make it impossible to send a file as part of
a ``multipart/form-data`` upload.
The toolbelt contains a class that allows you to build multipart request bodies
in exactly the format you need, and to avoid reading files into memory. An
example of how to use it is like this:
.. code-block:: python
import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder
m = MultipartEncoder(
fields={'field0': 'value', 'field1': 'value',
'field2': ('filename', open('file.py', 'rb'), 'text/plain')}
)
r = requests.post('http://httpbin.org/post', data=m,
headers={'Content-Type': m.content_type})
The :class:`~requests_toolbelt.multipart.encoder.MultipartEncoder` has the
``.to_string()`` convenience method, as well. This method renders the
multipart body into a string. This is useful when developing your code,
allowing you to confirm that the multipart body has the form you expect before
you send it on.
The toolbelt also provides a way to monitor your streaming uploads with
the :class:`~requests_toolbelt.multipart.encoder.MultipartEncoderMonitor`.
.. autoclass:: requests_toolbelt.multipart.encoder.MultipartEncoder
.. _support for multipart uploads: http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file
Monitoring Your Streaming Multipart Upload
------------------------------------------
If you need to stream your ``multipart/form-data`` upload then you're probably
in the situation where it might take a while to upload the content. In these
cases, it might make sense to be able to monitor the progress of the upload.
For this reason, the toolbelt provides the
:class:`~requests_toolbelt.multipart.encoder.MultipartEncoderMonitor`. The
monitor wraps an instance of a
:class:`~requests_toolbelt.multipart.encoder.MultipartEncoder` and is used
exactly like the encoder. It provides a similar API with some additions:
- The monitor accepts a function as a callback. The function is called every
time ``requests`` calls ``read`` on the monitor and passes in the monitor as
an argument.
- The monitor tracks how many bytes have been read in the course of the
upload.
You might use the monitor to create a progress bar for the upload. Here is `an
example using clint`_ which displays the progress bar.
To use the monitor you would follow a pattern like this:
.. code-block:: python
import requests
from requests_toolbelt.multipart import encoder
def my_callback(monitor):
# Your callback function
pass
e = encoder.MultipartEncoder(
fields={'field0': 'value', 'field1': 'value',
'field2': ('filename', open('file.py', 'rb'), 'text/plain')}
)
m = encoder.MultipartEncoderMonitor(e, my_callback)
r = requests.post('http://httpbin.org/post', data=m,
headers={'Content-Type': m.content_type})
If you have a very simple use case you can also do:
.. code-block:: python
import requests
from requests_toolbelt.multipart.encoder import MultipartEncoderMonitor
def my_callback(monitor):
# Your callback function
pass
m = MultipartEncoderMonitor.from_fields(
fields={'field0': 'value', 'field1': 'value',
'field2': ('filename', open('file.py', 'rb'), 'text/plain')},
callback=my_callback
)
r = requests.post('http://httpbin.org/post', data=m,
headers={'Content-Type': m.content_type})
.. autoclass:: requests_toolbelt.multipart.encoder.MultipartEncoderMonitor
.. _an example using clint:
https://github.com/requests/toolbelt/blob/master/examples/monitor/progress_bar.py
Streaming Data from a Generator
-------------------------------
There are cases where you, the user, have a generator of some large quantity
of data and you already know the size of that data. If you pass the generator
to ``requests`` via the ``data`` parameter, ``requests`` will assume that you
want to upload the data in chunks and set a ``Transfer-Encoding`` header value
of ``chunked``. Often times, this causes the server to behave poorly. If you
want to avoid this, you can use the
:class:`~requests.toolbelt.streaming_iterator.StreamingIterator`. You pass it
the size of the data and the generator.
.. code-block:: python
import requests
from requests_toolbelt.streaming_iterator import StreamingIterator
generator = some_function() # Create your generator
size = some_function_size() # Get your generator's size
content_type = content_type() # Get the content-type of the data
streamer = StreamingIterator(size, generator)
r = requests.post('https://httpbin.org/post', data=streamer,
headers={'Content-Type': content_type})
The streamer will handle your generator for you and buffer the data before
passing it to ``requests``.
.. versionchanged:: 0.4.0
File-like objects can be passed instead of a generator.
If, for example, you need to upload data being piped into standard in, you
might otherwise do:
.. code-block:: python
import requests
import sys
r = requests.post(url, data=sys.stdin)
This would stream the data but would use a chunked transfer-encoding. If
instead, you know the length of the data that is being sent to ``stdin`` and
you want to prevent the data from being uploaded in chunks, you can use the
:class:`~requests_toolbelt.streaming_iterator.StreamingIterator` to stream the
contents of the file without relying on chunking.
.. code-block:: python
import requests
from requests_toolbelt.streaming_iterator import StreamingIterator
import sys
stream = StreamingIterator(size, sys.stdin)
r = requests.post(url, data=stream,
headers={'Content-Type': content_type})
.. autoclass:: requests_toolbelt.streaming_iterator.StreamingIterator
|
