1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
|
<!DOCTYPE html
PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>Chapter 11. HTTP Web Services</title>
<link rel="stylesheet" href="../diveintopython.css" type="text/css">
<link rev="made" href="mailto:f8dy@diveintopython.org">
<meta name="generator" content="DocBook XSL Stylesheets V1.52.2">
<meta name="keywords" content="Python, Dive Into Python, tutorial, object-oriented, programming, documentation, book, free">
<meta name="description" content="Python from novice to pro">
<link rel="home" href="../toc/index.html" title="Dive Into Python">
<link rel="up" href="../toc/index.html" title="Dive Into Python">
<link rel="previous" href="../scripts_and_streams/summary.html" title="10.8. Summary">
<link rel="next" href="review.html" title="11.2. How not to fetch data over HTTP">
</head>
<body>
<table id="Header" width="100%" border="0" cellpadding="0" cellspacing="0" summary="">
<tr>
<td id="breadcrumb" colspan="5" align="left" valign="top">You are here: <a href="../index.html">Home</a> > <a href="../toc/index.html">Dive Into Python</a> > <span class="thispage">HTTP Web Services</span></td>
<td id="navigation" align="right" valign="top"> <a href="../scripts_and_streams/summary.html" title="Prev: “Summary”"><<</a> <a href="review.html" title="Next: “How not to fetch data over HTTP”">>></a></td>
</tr>
<tr>
<td colspan="3" id="logocontainer">
<h1 id="logo"><a href="../index.html" accesskey="1">Dive Into Python</a></h1>
<p id="tagline">Python from novice to pro</p>
</td>
<td colspan="3" align="right">
<form id="search" method="GET" action="http://www.google.com/custom">
<p><label for="q" accesskey="4">Find: </label><input type="text" id="q" name="q" size="20" maxlength="255" value=" "> <input type="submit" value="Search"><input type="hidden" name="cof" value="LW:752;L:http://diveintopython.org/images/diveintopython.png;LH:42;AH:left;GL:0;AWFID:3ced2bb1f7f1b212;"><input type="hidden" name="domains" value="diveintopython.org"><input type="hidden" name="sitesearch" value="diveintopython.org"></p>
</form>
</td>
</tr>
</table>
<!--#include virtual="/inc/ads" -->
<div class="chapter" lang="en">
<div class="titlepage">
<div>
<div>
<h2 class="title"><a name="oa"></a>Chapter 11. HTTP Web Services
</h2>
</div>
</div>
<div></div>
</div>
<div class="toc">
<ul>
<li><span class="section"><a href="index.html#oa.divein">11.1. Diving in</a></span></li>
<li><span class="section"><a href="review.html">11.2. How not to fetch data over HTTP</a></span></li>
<li><span class="section"><a href="http_features.html">11.3. Features of HTTP</a></span><ul>
<li><span class="section"><a href="http_features.html#d0e27596">11.3.1. User-Agent</a></span></li>
<li><span class="section"><a href="http_features.html#d0e27616">11.3.2. Redirects</a></span></li>
<li><span class="section"><a href="http_features.html#d0e27689">11.3.3. Last-Modified/If-Modified-Since</a></span></li>
<li><span class="section"><a href="http_features.html#d0e27724">11.3.4. ETag/If-None-Match</a></span></li>
<li><span class="section"><a href="http_features.html#d0e27752">11.3.5. Compression</a></span></li>
</ul>
</li>
<li><span class="section"><a href="debugging.html">11.4. Debugging HTTP web services</a></span></li>
<li><span class="section"><a href="user_agent.html">11.5. Setting the User-Agent</a></span></li>
<li><span class="section"><a href="etags.html">11.6. Handling Last-Modified and ETag</a></span></li>
<li><span class="section"><a href="redirects.html">11.7. Handling redirects</a></span></li>
<li><span class="section"><a href="gzip_compression.html">11.8. Handling compressed data</a></span></li>
<li><span class="section"><a href="alltogether.html">11.9. Putting it all together</a></span></li>
<li><span class="section"><a href="summary.html">11.10. Summary</a></span></li>
</ul>
</div>
<div class="section" lang="en">
<div class="titlepage">
<div>
<div>
<h2 class="title"><a name="oa.divein"></a>11.1. Diving in
</h2>
</div>
</div>
<div></div>
</div>
<div class="abstract">
<p>You've learned about <a href="../html_processing/index.html" title="Chapter 8. HTML Processing">HTML processing</a> and <a href="../xml_processing/index.html" title="Chapter 9. XML Processing">XML processing</a>, and along the way you saw <a href="../html_processing/extracting_data.html#dialect.extract.urllib" title="Example 8.5. Introducing urllib">how to download a web page</a> and <a href="../scripts_and_streams/index.html#kgp.openanything.urllib" title="Example 10.2. Parsing XML from a URL">how to parse XML from a URL</a>, but let's dive into the more general topic of HTTP web services.
</p>
</div>
<p>Simply stated, HTTP web services are programmatic ways of sending and receiving data from remote servers using the operations
of HTTP directly. If you want to get data from the server, use a straight HTTP GET; if you want to send new data to the server,
use HTTP POST. (Some more advanced HTTP web service APIs also define ways of modifying existing data and deleting data, using
HTTP PUT and HTTP DELETE.) In other words, the “<span class="quote">verbs</span>” built into the HTTP protocol (GET, POST, PUT, and DELETE) map directly to application-level operations for receiving, sending,
modifying, and deleting data.
</p>
<p>The main advantage of this approach is simplicity, and its simplicity has proven popular with a lot of different sites. Data
-- usually XML data -- can be built and stored statically, or generated dynamically by a server-side script, and all major
languages include an HTTP library for downloading it. Debugging is also easier, because you can load up the web service in
any web browser and see the raw data. Modern browsers will even nicely format and pretty-print XML data for you, to allow
you to quickly navigate through it.
</p>
<p>Examples of pure XML-over-HTTP web services:</p>
<div class="itemizedlist">
<ul>
<li><a href="http://www.amazon.com/webservices">Amazon API</a> allows you to retrieve product information from the Amazon.com online store.
</li>
<li><a href="http://www.nws.noaa.gov/alerts/">National Weather Service</a> (United States) and <a href="http://demo.xml.weather.gov.hk/">Hong Kong Observatory</a> (Hong Kong) offer weather alerts as a web service.
</li>
<li><a href="http://atomenabled.org/">Atom API</a> for managing web-based content.
</li>
<li><a href="http://syndic8.com/">Syndicated feeds</a> from weblogs and news sites bring you up-to-the-minute news from a variety of sites.
</li>
</ul>
</div>
<p>In later chapters, you'll explore APIs which use HTTP as a transport for sending and receiving data, but don't map application
semantics to the underlying HTTP semantics. (They tunnel everything over HTTP POST.) But this chapter will concentrate on
using HTTP GET to get data from a remote server, and you'll explore several HTTP features you can use to get the maximum benefit
out of pure HTTP web services.
</p>
<p>Here is a more advanced version of the <tt class="filename">openanything</tt> module that you saw in <a href="../scripts_and_streams/index.html" title="Chapter 10. Scripts and Streams">the previous chapter</a>:
</p>
<div class="example"><a name="d0e27515"></a><h3 class="title">Example 11.1. <tt class="filename">openanything.py</tt></h3>
<p>If you have not already done so, you can <a href="http://diveintopython.org/download/diveintopython-examples-5.4.zip" title="Download example scripts">download this and other examples</a> used in this book.
</p><pre class="programlisting"><span class='pykeyword'>
import</span> urllib2, urlparse, gzip
<span class='pykeyword'>from</span> StringIO <span class='pykeyword'>import</span> StringIO
USER_AGENT = <span class='pystring'>'OpenAnything/1.0 +http://diveintopython.org/http_web_services/'</span>
<span class='pykeyword'>class</span><span class='pyclass'> SmartRedirectHandler</span>(urllib2.HTTPRedirectHandler):
<span class='pykeyword'>def</span><span class='pyclass'> http_error_301</span>(self, req, fp, code, msg, headers):
result = urllib2.HTTPRedirectHandler.http_error_301(
self, req, fp, code, msg, headers)
result.status = code
<span class='pykeyword'>return</span> result
<span class='pykeyword'>def</span><span class='pyclass'> http_error_302</span>(self, req, fp, code, msg, headers):
result = urllib2.HTTPRedirectHandler.http_error_302(
self, req, fp, code, msg, headers)
result.status = code
<span class='pykeyword'>return</span> result
<span class='pykeyword'>class</span><span class='pyclass'> DefaultErrorHandler</span>(urllib2.HTTPDefaultErrorHandler):
<span class='pykeyword'>def</span><span class='pyclass'> http_error_default</span>(self, req, fp, code, msg, headers):
result = urllib2.HTTPError(
req.get_full_url(), code, msg, headers, fp)
result.status = code
<span class='pykeyword'>return</span> result
<span class='pykeyword'>def</span><span class='pyclass'> openAnything</span>(source, etag=None, lastmodified=None, agent=USER_AGENT):
<span class='pystring'>'''URL, filename, or string --> stream
This function lets you define parsers that take any input source
(URL, pathname to local or network file, or actual data as a string)
and deal with it in a uniform manner. Returned object is guaranteed
to have all the basic stdio read methods (read, readline, readlines).
Just .close() the object when you're done with it.
If the etag argument is supplied, it will be used as the value of an
If-None-Match request header.
If the lastmodified argument is supplied, it must be a formatted
date/time string in GMT (as returned in the Last-Modified header of
a previous request). The formatted date/time will be used
as the value of an If-Modified-Since request header.
If the agent argument is supplied, it will be used as the value of a
User-Agent request header.
'''</span>
<span class='pykeyword'>if</span> hasattr(source, <span class='pystring'>'read'</span>):
<span class='pykeyword'>return</span> source
<span class='pykeyword'>if</span> source == <span class='pystring'>'-'</span>:
<span class='pykeyword'>return</span> sys.stdin
<span class='pykeyword'>if</span> urlparse.urlparse(source)[0] == <span class='pystring'>'http'</span>:
<span class='pycomment'># open URL with urllib2 </span>
request = urllib2.Request(source)
request.add_header(<span class='pystring'>'User-Agent'</span>, agent)
<span class='pykeyword'>if</span> etag:
request.add_header(<span class='pystring'>'If-None-Match'</span>, etag)
<span class='pykeyword'>if</span> lastmodified:
request.add_header(<span class='pystring'>'If-Modified-Since'</span>, lastmodified)
request.add_header(<span class='pystring'>'Accept-encoding'</span>, <span class='pystring'>'gzip'</span>)
opener = urllib2.build_opener(SmartRedirectHandler(), DefaultErrorHandler())
<span class='pykeyword'>return</span> opener.open(request)
<span class='pycomment'># try to open with native open function (if source is a filename)</span>
<span class='pykeyword'>try</span>:
<span class='pykeyword'>return</span> open(source)
<span class='pykeyword'>except</span> (IOError, OSError):
<span class='pykeyword'>pass</span>
<span class='pycomment'># treat source as string</span>
<span class='pykeyword'>return</span> StringIO(str(source))
<span class='pykeyword'>def</span><span class='pyclass'> fetch</span>(source, etag=None, last_modified=None, agent=USER_AGENT):
<span class='pystring'>'''Fetch data and metadata from a URL, file, stream, or string'''</span>
result = {}
f = openAnything(source, etag, last_modified, agent)
result[<span class='pystring'>'data'</span>] = f.read()
<span class='pykeyword'>if</span> hasattr(f, <span class='pystring'>'headers'</span>):
<span class='pycomment'># save ETag, if the server sent one </span>
result[<span class='pystring'>'etag'</span>] = f.headers.get(<span class='pystring'>'ETag'</span>)
<span class='pycomment'># save Last-Modified header, if the server sent one </span>
result[<span class='pystring'>'lastmodified'</span>] = f.headers.get(<span class='pystring'>'Last-Modified'</span>)
<span class='pykeyword'>if</span> f.headers.get(<span class='pystring'>'content-encoding'</span>, <span class='pystring'>''</span>) == <span class='pystring'>'gzip'</span>:
<span class='pycomment'># data came back gzip-compressed, decompress it </span>
result[<span class='pystring'>'data'</span>] = gzip.GzipFile(fileobj=StringIO(result[<span class='pystring'>'data'</span>]])).read()
<span class='pykeyword'>if</span> hasattr(f, <span class='pystring'>'url'</span>):
result[<span class='pystring'>'url'</span>] = f.url
result[<span class='pystring'>'status'</span>] = 200
<span class='pykeyword'>if</span> hasattr(f, <span class='pystring'>'status'</span>):
result[<span class='pystring'>'status'</span>] = f.status
f.close()
<span class='pykeyword'>return</span> result
</pre></div>
<div class="furtherreading">
<h3>Further reading</h3>
<ul>
<li>Paul Prescod believes that <a href="http://webservices.xml.com/pub/a/ws/2002/02/06/rest.html">pure HTTP web services are the future of the Internet</a>.
</li>
</ul>
</div>
</div>
</div>
<table class="Footer" width="100%" border="0" cellpadding="0" cellspacing="0" summary="">
<tr>
<td width="35%" align="left"><br><a class="NavigationArrow" href="../scripts_and_streams/summary.html"><< Summary</a></td>
<td width="30%" align="center"><br> <span class="divider">|</span> <span class="thispage">1</span> <span class="divider">|</span> <a href="review.html" title="11.2. How not to fetch data over HTTP">2</a> <span class="divider">|</span> <a href="http_features.html" title="11.3. Features of HTTP">3</a> <span class="divider">|</span> <a href="debugging.html" title="11.4. Debugging HTTP web services">4</a> <span class="divider">|</span> <a href="user_agent.html" title="11.5. Setting the User-Agent">5</a> <span class="divider">|</span> <a href="etags.html" title="11.6. Handling Last-Modified and ETag">6</a> <span class="divider">|</span> <a href="redirects.html" title="11.7. Handling redirects">7</a> <span class="divider">|</span> <a href="gzip_compression.html" title="11.8. Handling compressed data">8</a> <span class="divider">|</span> <a href="alltogether.html" title="11.9. Putting it all together">9</a> <span class="divider">|</span> <a href="summary.html" title="11.10. Summary">10</a> <span class="divider">|</span>
</td>
<td width="35%" align="right"><br><a class="NavigationArrow" href="review.html">How not to fetch data over HTTP >></a></td>
</tr>
<tr>
<td colspan="3"><br></td>
</tr>
</table>
<div class="Footer">
<p class="copyright">Copyright © 2000, 2001, 2002, 2003, 2004 <a href="mailto:mark@diveintopython.org">Mark Pilgrim</a></p>
</div>
</body>
</html>
|
