1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
|
Asynchronous Python HTTP Requests for Humans
============================================
.. image:: https://travis-ci.org/tardyp/txrequests.png?branch=master
:target: https://travis-ci.org/tardyp/txrequests
Small add-on for the python requests_ http library. Makes use twisted's ThreadPool,
so that the requests'API returns deferred
The additional API and changes are minimal and strives to avoid surprises.
The following synchronous code:
.. code-block:: python
from requests import Session
session = Session()
# first requests starts and blocks until finished
response_one = session.get('http://httpbin.org/get')
# second request starts once first is finished
response_two = session.get('http://httpbin.org/get?foo=bar')
# both requests are complete
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)
Can be translated to make use of futures, and thus be asynchronous by creating
a FuturesSession and catching the returned Future in place of Response. The
Response can be retrieved by calling the result method on the Future:
.. code-block:: python
from txrequests import Session
from twisted.internet import defer
@defer.inlineCallbacks
def main():
# use with statement to cleanup session's threadpool, and connectionpool after use
# you can also use session.close() if want to use session for long term use
with Session() as session:
# first request is started in background
d1 = session.get('http://httpbin.org/get')
# second requests is started immediately
d2 = session.get('http://httpbin.org/get?foo=bar')
# wait for the first request to complete, if it hasn't already
response_one = yield d1
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
# wait for the second request to complete, if it hasn't already
response_two = yield d2
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)
By default a ThreadPool is created with 4 max workers. If you would like to
adjust that value or share a threadpool across multiple sessions you can provide
one to the Session constructor.
.. code-block:: python
from twisted.python.threadpool import ThreadPool
from txrequests import Session
session = FuturesSession(pool=ThreadPool(maxthreads=10))
# ...
As a shortcut in case of just increasing workers number you can pass
`minthreads` and/or `maxthreads` straight to the `Session` constructor:
.. code-block:: python
from txrequests import Session
session = Session(maxthreads=10)
That's it. The api of requests.Session is preserved without any modifications
beyond returning a Deferred rather than Response. As with all futures exceptions
are shifted to the deferred errback.
Working in the Background
=========================
There is one additional parameter to the various request functions,
background_callback, which allows you to work with the Response objects in the
background thread. This can be useful for shifting work out of the foreground,
for a simple example take json parsing.
.. code-block:: python
from pprint import pprint
from txrequests import Session
from twisted.internet import defer
@defer.inlineCallbacks
def main():
with Session() as session:
def bg_cb(sess, resp):
# parse the json storing the result on the response object
resp.data = resp.json()
return resp
d = session.get('http://httpbin.org/get', background_callback=bg_cb)
# do some other stuff, send some more requests while this one works
response = yield d
print('response status {0}'.format(response.status_code))
# data will have been attached to the response object in the background
pprint(response.data)
Installation
============
pip install txrequests
Credits
========
txrequests is based on requests_future_, from Ross McFarland
.. _`requests`: https://github.com/kennethreitz/requests
.. _`requests_future`: https://github.com/ross/requests-futures
|
