1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
|
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
@# This file is processed by EmPy
<!--This file was generated by EmPy from README.html.in : do not edit-->
@# http://wwwsearch.sf.net/bits/colorize.py
@{
from colorize import colorize
import time
import release
last_modified = release.svn_id_to_time("$Id: README.html.in 56674 2008-07-19 13:44:41Z jjlee $")
try:
base
except NameError:
base = False
}
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<meta name="author" content="John J. Lee <jjl@@pobox.com>">
<meta name="date" content="@(time.strftime("%Y-%m-%d", last_modified))">
<meta name="keywords" content="form,HTML,Python,web,client,client-side">
<title>ClientForm</title>
<style type="text/css" media="screen">@@import "../styles/style.css";</style>
@[if base]<base href="http://wwwsearch.sourceforge.net/ClientForm/">@[end if]
</head>
<body>
<div id="sf"><a href="http://sourceforge.net">
<img src="http://sourceforge.net/sflogo.php?group_id=48205&type=2"
width="125" height="37" alt="SourceForge.net Logo"></a></div>
<h1>ClientForm</h1>
<div id="Content">
<p>ClientForm is a Python module for handling HTML forms on the client
side, useful for parsing HTML forms, filling them in and returning the
completed forms to the server. It developed from a port of Gisle Aas'
Perl module <code>HTML::Form</code>, from the <a
href="http://www.linpro.no/lwp/">libwww-perl</a> library, but the
interface is not the same.
<p>Simple working example:
@{colorize("".join(open("examples/simple.py").readlines()[2:]))}
<p>A more complicated working example (<em><strong>Note</strong>: this
example makes use of the ClientForm 0.2 API; refer to the README.html
file in the latest 0.1 release for the corresponding code for that
version.</em>):
<a name="example"></a>
@{colorize("".join(open("examples/example.py").readlines()[2:]))}
<a name="notes"></a>
<p>All of the standard control types are supported: <code>TEXT</code>,
<code>PASSWORD</code>, <code>HIDDEN</code>, <code>TEXTAREA</code>,
<code>ISINDEX</code>, <code>RESET</code>, <code>BUTTON</code> (<code>INPUT
TYPE=BUTTON</code> and the various <code>BUTTON</code> types),
<code>SUBMIT</code>, <code>IMAGE</code>, <code>RADIO</code>,
<code>CHECKBOX</code>, <code>SELECT</code>/<code>OPTION</code> and
<code>FILE</code> (for file upload). Both standard form encodings
(<code>application/x-www-form-urlencoded</code> and
<code>multipart/form-data</code>) are supported.
<p>The module is designed for testing and automation of web
interfaces, not for implementing interactive user agents.
<p><strong><em>Security note</em>: Remember that any passwords you store in
<code>HTMLForm</code> instances will be saved to disk in the clear if you
pickle them (directly or indirectly). The simplest solution to this is to
avoid pickling <code>HTMLForm</code> objects. You could also pickle before
filling in any password, or just set the password to <code>""</code> before
pickling.</strong>
<p>Python 2.0 or above is required. To run the tests, you need the
<code>unittest</code> module (from <a
href="http://pyunit.sourceforge.net/">PyUnit</a>). <code>unittest</code> is a
standard library module with Python 2.1 and above.
<p>For full documentation, see the docstrings in ClientForm.py.
<p><em><strong>Note: this page describes the 0.2 (stable release)
interface. See <a href="./src/README-0_1_17.html">here</a> for the
old 0.1 interface.</strong> </em>
<a name="parsers"></a>
<h2>Parsers</h2>
<p>ClientForm contains two parsers. See <a href="./#faq">the FAQ entry on
XHTML</a> for details.
<p><a href="http://www.egenix.com/files/python/mxTidy.html">mxTidy</a> or <a
href="http://utidylib.berlios.de/">Tidylib</a> can be useful for dealing with
bad HTML.
<p>I think it would be nice to have an implementation of ClientForm based on <a
href="http://www.crummy.com/software/BeautifulSoup/">BeautifulSoup</a>
(i.e. all methods and attributes implemented using the BeautifulSoup API),
since that module does tolerant HTML parsing with a nice API for doing
non-forms stuff. (I'm not about to do this, though. For anybody interested in
doing this, note that the ClientForm tests would need making
constructor-independent first.)
<a name="compat"></a>
<h2>Backwards-compatibility mode</h2>
<p>ClientForm 0.2 includes three minor backwards-incompatible interface
changes from version 0.1.
<p>To make upgrading from 0.1 easier, and to allow me to stop supporting
version 0.1 sooner, version 0.2 contains support for operating in a
backwards-compatible mode, under which code written for 0.1 should work without
modification. This is done on a per-<code>HTMLForm</code> basis via the
<code>.backwards_compat</code> attribute, but for convenience the
ParseResponse() and ParseFile() factory functions accept
<code>backwards_compat</code> arguments. These backwards-compatibility
features will be removed in version 0.3. The default is to operate in
backwards-compatible mode. To run with backwards compatible mode turned
<em><strong>OFF</strong></em> (<strong>strongly recommended</strong>):
@{colorize(r"""
from urllib2 import urlopen
from ClientForm import ParseResponse
forms = ParseResponse(urlopen("http://example.com/"), backwards_compat=False)
# ...
""")}
<p>The backwards-incompatible changes are:
<ul>
<li><p>Ambiguous specification of controls or items now results in
AmbiguityError. If you want the old behaviour, explicitly pass
<code>nr=0</code> to indicate you want the first matching control or item.
<li><p>Item label matching is now done by substring, not by strict
string-equality (but note leading and trailing space is always stripped).
(Control label matching is always done by substring.)
<li><p>Handling of disabled list items has changed. First, note that handling
of disabled list items in 0.1 (and in 0.2's backwards-compatibility mode!) is
buggy: disabled items are successful (ie. disabled item names are sent back to
the server). As a result, there was no distinction to be made between
successful items and selected items. In 0.2, the bug is fixed, so this is no
longer the case, and it is important to note that list controls'
<code>.value</code> attribute contains only the <em>successful</em> item names;
items that are <em>selected </em> but not successful (because disabled) are not
included in <code>.value</code>. Second, disabled list items may no longer be
deselected: AttributeError is raised in 0.2, whereas deselection was allowed in
0.1. The bug in 0.1 and in 0.2's backwards-compatibility mode will not be
fixed, to preserve compatibility and to encourage people to upgrade to the new
0.2 <code>backwards_compat=False</code> behaviour. </ul>
<a name="credits"></a>
<h2>Credits</h2>
<p>Apart from Gisle Aas for allowing the original port from
libwww-perl, particular credit is due to Gary Poster and Benji York,
and their employer, Zope Corporation, for their contributions which
led to ClientForm 0.2 being released. Thanks also to the many people
who have contributed bug reports.
<a name="download"></a>
<h2>Download</h2>
<p>For installation instructions, see the INSTALL.txt file included in the
distribution.
<p><span class="spanhdr">Stable release</span> There have been three fairly
minor backwards-incompatible interface changes since version 0.1 (see <a
href="./#compat">above</a>), but by default the code operates in a
backwards-compatible mode so that code written for 0.1 should work without
changes.
<p>0.2 includes better support for labels, and a simpler interface (all the old
methods are still there, but some have been deprecated and a few added).
<ul>
@{version = "0.2.10"}
<li><a href="./src/ClientForm-@(version).tar.gz">ClientForm-@(version).tar.gz</a>
<li><a href="./src/ClientForm-@(version).zip">ClientForm-@(version).zip</a>
<li><a href="./src/ChangeLog.txt">Change Log</a> (included in distribution)
<li><a href="./src/">Older releases.</a>
</ul>
<br>
<p><span class="spanhdr">Old release</span> No longer maintained. I recommend
upgrading from 0.1 to 0.2.
<p>There were many interface changes between 0.0 and 0.1, so you should take
care if upgrading old code from 0.0.
<p>0.1 includes <code>FILE</code> control support for file upload, handling
of disabled list items, and a redesigned interface.
<ul>
@{version = "0.1.17"}
@{win_version = release.win_version(version)}
<li><a href="./src/ClientForm-@(version).tar.gz">ClientForm-@(version).tar.gz</a>
<li><a href="./src/ClientForm-@(win_version).zip">ClientForm-@(win_version).zip</a>
<li><a href="./src/ChangeLog.txt">Change Log</a> (included in distribution)
<li><a href="./src/">Older releases.</a>
</ul>
<br>
<p><span class="spanhdr">Ancient release</span> No longer maintained. You
don't want this.
<ul>
@{version = "0.0.16"}
@{win_version = release.win_version(version)}
<li><a href="./src/ClientForm-@(version).tar.gz">ClientForm-@(version).tar.gz</a>
<li><a href="./src/ClientForm-@(win_version).zip">ClientForm-@(win_version).zip</a>
<li><a href="./src/ChangeLog.txt">Change Log</a> (included in distribution)
<li><a href="./src/">Older releases.</a>
</ul>
<a name="svn"></a>
<h2>Subversion</h2>
<p>The <a href="http://subversion.tigris.org/">Subversion (SVN)</a> trunk is <a href="http://codespeak.net/svn/wwwsearch/ClientForm/trunk#egg=ClientForm-dev">http://codespeak.net/svn/wwwsearch/ClientForm/trunk</a>, so to check out the source:
<pre>
svn co http://codespeak.net/svn/wwwsearch/ClientForm/trunk ClientForm
</pre>
<a name="faq"></a>
<h2>FAQs</h2>
<ul>
<li>Doesn't the standard Python library module, <code>cgi</code>, do this?
<p>No: the <code>cgi</code> module does the server end of the job. It
doesn't know how to parse or fill in a form or how to send it back to the
server.
<li>Which version of Python do I need?
<p>2.0 or above (ClientForm 0.2; version 0.1 requires Python 1.5.2 or above).
<li>Is <code>urllib2</code> required?
<p>No.
<li>How do I use it without <code>urllib2</code>?
<p>Use <code>.click_request_data()</code> instead of <code>.click()</code>.
<li>Which <code>urllib2</code> do I need?
<p>You don't. It's convenient, though. If you have Python 2.0, you need to
upgrade to the version from Python 2.1 (available from <a
href="http://www.python.org/">www.python.org</a>). Otherwise, you're OK.
<li>Which license?
<p>ClientForm is dual-licensed: you may pick either the
<a href="http://www.opensource.org/licenses/bsd-license.php">BSD license</a>,
or the <a href="http://www.zope.org/Resources/ZPL">ZPL 2.1</a> (both are
included in the distribution).
<a name="xhtml"></a>
<li>Is XHTML supported?
<p>Yes. You must pass
<code>form_parser_class=ClientForm.XHTMLCompatibleFormParser</code> to
<code>ParseResponse()</code> / <code>ParseFile()</code>. Note this parser
is less tolerant of bad HTML than the default,
<code>ClientForm.FormParser</code>
<li>How do I figure out what control names and values to use?
<p><code>print form</code> is usually all you need.
In your code, things like the <code>HTMLForm.items</code> attribute of
<code>HTMLForm</code> instances can be useful to inspect forms at
runtime. Note that it's possible to use item labels instead of item
names, which can be useful — use the <code>by_label</code>
arguments to the various methods, and the <code>.get_value_by_label()</code> /
<code>.set_value_by_label()</code> methods on <code>ListControl</code>.
<li>What do those <code>'*'</code> characters mean in the string
representations of list controls?
<p>A <code>*</code> next to an item means that item is selected.
<li>What do those parentheses (round brackets) mean in the string
representations of list controls?
<p>Parentheses <code>(foo)</code> around an item mean that item is disabled.
<li>Why doesn't <some control> turn up in the data returned by
<code>.click*()</code> when that control has non-<code>None</code> value?
<p>Either the control is disabled, or it is not successful for some other
reason. 'Successful' (see HTML 4 specification) means that the control
will cause data to get sent to the server.
<li>Why does ClientForm not follow the HTML 4.0 / RFC 1866 standards for
<code>RADIO</code> and multiple-selection <code>SELECT</code> controls?
<p>Because by default, it follows browser behaviour when setting the
initially-selected items in list controls that have no items explicitly
selected in the HTML. Use the <code>select_default</code> argument to
<code>ParseResponse</code> if you want to follow the RFC 1866 rules
instead. Note that browser behaviour violates the HTML 4.01 specification
in the case of <code>RADIO</code> controls.
<li>Why does <code>.click()</code>ing on a button not work for me?
<ul>
<li>Clicking on a <code>RESET</code> button doesn't do anything, by design
- this is a library for web automation, not an interactive browser.
Even in an interactive browser, clicking on <code>RESET</code> sends
nothing to the server, so there is little point in having
<code>.click()</code> do anything special here.
<li>Clicking on a <code>BUTTON TYPE=BUTTON</code> doesn't do anything
either, also by design. This time, the reason is that that
<code>BUTTON</code> is only in the HTML standard so that one can attach
callbacks to its events. The callbacks are functions in
<code>SCRIPT</code> elements (such as Javascript) embedded in the HTML,
and their execution may result in information getting sent back to the
server. ClientForm, however, knows nothing about these callbacks, so
it can't do anything useful with a click on a <code>BUTTON</code> whose
type is <code>BUTTON</code>.
<li>Generally, embedded script may be messing things up in all kinds of
ways. See the answer to the next question.
</ul>
<li>Embedded script is messing up my form filling. What do I do?
<p>See the <a href="../bits/GeneralFAQ.html">General FAQs</a> page and the
next FAQ entry for what to do about this.
<!-- XXX example here -->
<li>How do I change <code>INPUT TYPE=HIDDEN</code> field values (for example,
to emulate the effect of JavaScript code)?
<p>As with any control, set the control's <code>readonly</code> attribute
false.
<p>@{colorize(r"""
form.find_control("foo").readonly = False # allow changing .value of control foo
form.set_all_readonly(False) # allow changing the .value of all controls
""")}
</li>
<li>I'm having trouble debugging my code.
<p>The <a href="../ClientCookie/">ClientCookie</a> package makes it
easy to get <code>.seek()</code>able response objects, which is
convenient for debugging. See also <a
href="../ClientCookie/doc.html#debugging">here</a> for few
relevant tips. Also see <a href="../bits/GeneralFAQ.html"> General
FAQs</a>.
<li>I have a control containing a list of integers. How do I select the one
whose value is nearest to the one I want?
<p>@{colorize(r"""
import bisect
def closest_int_value(form, ctrl_name, value):
values = map(int, [item.name for item in form.find_control(ctrl_name).items])
return str(values[bisect.bisect(values, value) - 1])
form["distance"] = [closest_int_value(form, "distance", 23)]
""")}
</li>
<li>Where can I find out more about the HTML and HTTP standards?
<ul>
<li>W3C <a href="http://www.w3.org/TR/html401/">HTML 4.01
Specification</a>.
<li><a href="http://www.ietf.org/rfc/rfc1866.txt">RFC 1866</a> -
the HTML 2.0 standard.
<li><a href="http://www.ietf.org/rfc/rfc1867.txt">RFC 1867</a> -
Form-based file upload.
<li><a href="http://www.ietf.org/rfc/rfc2616.txt">RFC 2616</a> -
HTTP 1.1 Specification.
</ul>
</ul>
<p>I prefer questions and comments to be sent to the <a
href="http://lists.sourceforge.net/lists/listinfo/wwwsearch-general">
mailing list</a> rather than direct to me.
<p><a href="mailto:jjl@@pobox.com">John J. Lee</a>,
@(time.strftime("%B %Y", last_modified)).
</div>
<div id="Menu">
@(release.navbar('ClientForm'))
<br>
<a href="../#other">Other stuff</a><br>
<br>
<a href="./#example">Example</a><br>
<a href="./#notes">Notes</a><br>
<a href="./#parsers">Parsers</a><br>
<a href="./#compat">Compatibility</a><br>
<a href="./#credits">Credits</a><br>
<a href="./#download">Download</a><br>
<a href="./#faq">FAQs</a><br>
</div>
</body>
</html>
|
