File: orr.html

package info (click to toggle)
lg-issue98 1-1
links: PTS
area: main
in suites: sarge
size: 1,180 kB
ctags: 91
sloc: sh: 78; makefile: 34; perl: 23
file content (550 lines) | stat: -rw-r--r-- 23,110 bytes

<html>
<head>
<link href="../lg.css" rel="stylesheet" type="text/css" media="screen, projection"  />
<title>Python Simplicity LG #98</title>

<style type="text/css" media="screen, projection">
<!--


.articlecontent {
	position:absolute;
	top:143px;
}


-->
</style>


</head>

<body>


<img src="../gx/2003/newlogo-blank-200-gold2.jpg" id="logo" alt="Linux Gazette"/>
<p id="fun">...making Linux just a little more fun!</p>


<div class="content articlecontent">

<div id="previousnexttop">
<A HREF="moen.html" >&lt;-- prev</A> | <A HREF="pramode.html" >next --&gt;</A>
</div>



<h1>Python Simplicity</h1>
<p id="by"><b>By <A HREF="../authors/orr.html">Mike Orr (Sluggo)</A></b></p>

<p>
This article was inspired by
<A HREF="http://seapig.org/BrianDorsey">Brian Dorsey</A>, who hosted a 
<A HREF="http://seapig.org/">SeaPIG</A> meeting last month.  (SeaPIG is the
Seattle Python Interest Group.)  As I was perusing the bookshelves at his
house, I saw that his books on the Simplicity Movement (see below) match his
fanaticism for simplicity in programming.  Brian is a big-shot database
administrator.  (At one of Paul Allen's companies, boo, hiss.)  He knows more
SQL than most people I know.  You'd think that means he loves complexity.
Don't <EM>all</EM> database administrators love complexity?  (It keeps them
employed, after all.)  But Brian stunned me by revealing his enthusiasm for
trying out all the little Python database modules that are <EM>un</EM>complex.
He follows the 80/20 rule: sometimes 20% of the features solve the problem for
80% of the population.  Brian has also been demonstrating several
<EM>other</EM> simple modules over our last several Python meetings, so I'd
like to share with you a few of those.

<P> In most articles, the author walks through code he's worked with
extensively.  (Or <EM>pretends</EM> he has worked with extensively.)  In this
article, I'm discussing modules I have <EM>not</EM> used.  The point is not to
say these modules are the best thing since sliced bread (although some of them
are nifty).  The point is that these modules demonstrate simplicity,
either in their code or in their use.

<P> By <EM>in their code</EM>, I mean the module itself is short.  Less code
means less chance for things to go wrong, so more reliability.  That's the KISS
principle: "Keep it Simple, Stupid!"  <EM>In their use</EM> refers to the
user's perspective: it takes only a few lines to activate their features.  Some
modules are simple in one way or the other, while others are simple in both.

<H2>Doc XML-RPC Server</H2>

<P> Doc XML-RPC Server has <EM>got</EM> to be one of the easiest ways ever to
offer services on the Internet.  It's one of those inventions that makes you
bonk your head and think, "Why didn't somebody think of this sooner?"  Say
you've written your services as methods of a certain class.  You want to offer
these services on the Internet or on an intranet.  It takes just six lines of
code:

<PRE>
from DocXMLRPCServer import DocXMLRPCServer
server = DocXMLRPCServer(('', 8000), logRequests=0)
server.register_introspection_functions()
server.register_instance(SimpleShareServer())
server.serve_forever()
</PRE>

'SimpleShareServer' is a class we created.  We start a server on port 8000,
register an instance of our class, register some optional services that come
with DocXMLRPCServer ("introspection functions"; e.g., 'help'), and away we go.
Here's the services we're offering:

<PRE>
import time
class SimpleShareServer:
     def message(self, msg):
         """message('Print me!') =&gt; True 
         
         Log everything passed to this function"""
         print time.asctime(), msg
         return True
 
     def wait(self, seconds):
         """wait(5) =&gt; 5 
         
         Wait for a certain number of seconds before returning.
         Returns the same number passed in."""
         print time.asctime(), "Waiting %s seconds" % seconds
         time.sleep(seconds)
         print time.asctime(), "Finished waiting %s seconds" % seconds
         return seconds
</PRE>

Of course, a local routine can just instantiate the class and call the methods
directly.  To do the same remotely, you used to have to either write a custom
protocol implementation yourself, or read a long reference books to configure
an off-the-shelf server or library.  But a remote user can access these
services with just a couple lines of code:

<PRE>
import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:8000')
s.message("Hello, simple world!")
result = s.wait(15)
</PRE>

After these lines have executed, 'result' is 15, and "Hello, simple world!"
appears on the server's console (standard output).  Note that the arguments
and return value were passed seamlessly between client and server, just like
invoking a local method.  The server proxy object "stands in" for the remote
instance.  Note that the client is using a generic XML-RPC library; it doesn't
have to use a library specific to DocXMLRPCServer.

<P> RPC (Remote Procedure Call) has been around on Unix systems for decades.
NFS uses it, for example.  But plain RPC (so I'm told) cannot cross programming
languages.  If the server is Python, the client has to be Python too, or
something that knows how to encode/decode Python argument types.  
<EM>XML</EM>-RPC removes the language restriction.  The arguments are converted
to language-neutral XML, and so is the return value.  This has some
limitations:

<UL>
<LI> You can pass only Python standard types (including lists and dictionaries),
but not custom instances.
<LI> You can't pass <CODE>None</CODE> unless you enable a standard but
non-universal option
<LI> To pass a string containing ASCII control characters (0-31 decimal) or
other characters not allowed in XML, you have to wrap it in a 'Binary'
instance.  (But you don't have to do this for '&lt;', '&gt;' and '&amp;', which
DocXMLRPCServer automatically escapes.)
</UL>

In exchange for these limitations, Perl and other clients can access your
server -- just like any web browser can access an HTTP server.

<P> Speaking of HTTP, that's the niftiest part of DocXMKLRPCServer.  If a
client sends an HTTP POST request to the same port, the server recognizes it
and translates it to the corresponding method call.  This could be used to
collect form submissions for a survey, for instance.  If a client sends an
HTTP GET request, the server responds with an HTML page documenting itself.
You've heard of emacs, the Self-Documenting Editor?  Here's the self-documenting
arbitrary server.  Your server class can define three extra methods to
customize the documentation output:

<PRE>
set_server_title(STRING)         # For the &lt;TITLE&gt; tag.
set_server_name(STRING)          # For the &lt;H2&gt; header.
set_server_description(STRING)   # The documentation, in HTML format.
</PRE>
</UL>

<P> DocXMLRPCServer is built on top of SimpleXMLRPCServer, which provides 
everything except the HTTP ability.

<P> Somebody might object, "But it's using XML, and XML is decidedly
<EM>non-</EM>simple."  This is true.  XML is a horrible bastard beast that
should never have seen the light of day.  In theory, it's wonderful.  In
practice, most of the DTD's are are so unnecessarily complex and the namespaces
so nitpickily detailed that it looks like something only a bureaucracy could
have designed -- the union of all attributes lobbied by every single
special-interest group.  You have to trust that the expat parser or whatever
it's using under the hood won't blow up someday.  So DocXMLRPCServer isn't
simple in the code it depends on.  But it's simple <EM>to use</EM>.  Did you 
see any XML above?  I didn't.  I'm all for using XML if you don't have to look
at it.  Like the way Elvis impersonation bands are fun to attend as long as you
keep your back to the stage, so you can enjoy the music without having to look
at the tacky 50s kitsch.  But I digress....

<P> The SimpleShareServer above is based on a server Brian demonstrated at a
SeaPIG meeting, which he describes on a 
<A HREF="http://seapig.org/DocXMLRPCServer">wiki page</A>.

<H2>db_row</H2>

<P> <A HREF="http://opensource.theopalgroup.com/files/db_row.py.html">db_row</A>
is a short module to wrap a SQL result set.  The DB API database modules
(MySQLDb, several PosgreSQL modules, Oracle and
<A HREF="http://www.python.org/topics/database/modules.html">others</A>)
return a query row as a tuple of column values.  Let's fool it with our own
tuple and see what it does.

<PRE>
tup = (1, 2, 3)     # E.g., "SELECT a, b, a+b FROM SomeTable WHERE id=456;"
R = IMetaRow(['a', 'b', 'sum'])
                    # Create a custom class that names the rows in order.
                    # IMetaRow is a "class factory": it creates a class.
r = R(tup)          # Instantiate our custom class.
print r[0], r['b'], r.fields.sum
                    # Prints "1 2 3".  Access values by subscript, key or
                    # attribute.  (The "I" in IMetaRow means case-insensitive.)
print r.keys()      # Look ma, dictionary methods!
print r.dict()      # Just give me a real dictionary, please.
</PRE>

<P>To convert an entire multi-row result set to a list of such jobbies, use a 
list comprehension:

<PRE>
lis = [ R(row) for row in cursor.fetchall() ]
print lis[0]['a'], "+", lis[0]['b']
print lis[0].fields.sum
</PRE>

Wrap the list comprehension in a function, and you only have to see it once.

<P> Why do I like this module?  It's short.  You don't have to wait for it to
be incorporated into your favorite DB API module; it works with all of them
already.  It works with non-SQL and ad-hoc result sets too.  It solves a common
problem in a simple way.  (It's not all <EM>that</EM> simple.  It uses Python
slots, for instance.  But we'll ignore that and hope Python's obscure slots
feature has had most of its early bugs ironed out.)  It claims to use less
memory than a list of dictionaries.

<P> But db_row's simplicity does come at the cost of certain disadvantages.
It has no knowledge of the database field names or data types.  You can lie to
IMetaRow() and rename the fields anything you want.  That may be convenient in
some situations, but in an application with more than a few tables, it can
get out of hand pretty quickly.  Confusing yourself (and future maintainers)
with inconsistent field names is a decidedly <EM>un</EM>simple idea.  Or
doubleplus ungood as Orwell would say.

<P> (The example above was inspired by db_row's docstring.)

<H2>SQLite and pysqlite</H2>

<P> <A HREF="http://www.sqlite.org/">SQLite</A> is an entire SQL server encoded
in a little C library.  <A HREF="http://pysqlite.sourceforge.net/">pysqlite</A>
is a Python wrapper (DB API compatible).  Brian calls this combination, "80% of
what you'll ever need a database for in a single 270K executable (or Python
module)."  

<P> The "80% you need" is ACID-compliant transactions, basic data types
(strings, numbers, BLOBs, DateTimes), auto-increment fields, NULLs, temporary
tables ("CREATE TEMPORARY TABLE"), a command-line utility (&agrave; la 
mysql and psql), dumping a database to SQL statements (PosgreSQL compatible),
and huge databases (2 terabytes).  There's even support for concurrent access
of the database file in multiple processes, which I was pretty amazed at.  You
can't store strings that contain null characters (0 decimal) though.  The
database schema is stored in a table called 'sqlite_master'.  Security is done
by file permissions.

<P> Another quirk of SQLite is typelessness.  You can put letters into a
numeric field, whatever that means.  Actually, it means that a database is
meant to store data, not to impose its will on the data.  The SQLite developers
call the rigid type system in the SQL standard and in most implementations a
misfeature.  (See  the <A
HREF="http://www.sqlite.org/datatypes.html">Datatypes</A> page in the SQLite
documentation for the full justification.)  Fields can be created with all the
usual SQL type specifiers, but those are just hints to the user, not
rules SQLite enforces.  Actually, SQLite does honor the types to some degree:
they influence the sort order and whether two values are identical.  There is
one exception to SQLite's permissivism: auto-increment fields ("INTEGER
PRIMARY KEY") have to be integers.  

<H2>Other modules</H2>

<P> There are a few object-oriented wrappers for SQL access, including
<A HREF="http://sqlobject.org/">SQLObject</A> and 
<A HREF="http://pdo.neurokode.com/index.php">DBO</A>.

<P>
<A HREF="http://starship.python.net/crew/theller/ctypes/index.html">ctypes</A>
is a way to call C libraries directly from Python, which is supposedly easier
than SWIG.  


<H2>Python built-in features</H2>

<P> Python 2.2 introduced three features that took people a bit of time to
get their heads around, but they turned out to be incredibly useful:
iterators, generators and properies.  Iterators let you have a for-loop
without having to pregenerate the entire sequence of values and keep them all
in memory simultaneously.  Generators allow an easy way for a function to
iterate: it dispenses with the "topmost for-loop", leaving you more horizontal
screen space and less clutter.  Properties allow you to define "smart" 
attributes: those that trigger an action when they're get or set.  Properties
are controversial to some purists, but they avoid the clumsiness of
accessor methods (aka parenthesesitis, which is a serious disease among
C/Java-phobics).  

<P> Python 2.3 continues the trend with more features that simplify your
programs.  Sets are like dictionaries without the values ("just the keys,
please").  If you're using a dictionary only to weed out duplicates, why
define "values" you're not going to use?  There's a logging module and a
simple DateTime object.  But the thing I use most is enumerate():

<PRE>
&gt;&gt;&gt; lis = ['vanilla', 'chocolate', 'strawberry']
&gt;&gt;&gt; for i, element in enumerate(lis)::
...    print "Element %d is %s." % (i, element)
...
Element 0 is vanilla.
Element 1 is chocolate.
Element 2 is strawberry.
</PRE>

This is a long-requested feature that avoids the equivalent but clumsier:

<PRE>
&gt;&gt;&gt; for i in range(len(lis)):
...     element = lis[i]
...     print "Element %d is %s." % (i, element)
...
Element 0 is vanilla.
Element 1 is chocolate.
Element 2 is strawberry.
</PRE>

<H2>The simplicity movement</H2>

<P> <EM>"And now for something completely different..."</EM>

<P> The simplicity movement, championed by authors like Amy
Daczyczyn (author of <I>The Tightwad Gazette</I>, a paper zine), 
Joe Dominguez &amp; Vicki Robin, Cecile Andrews, Elaine St James and
others, is about deciding what you really want from life and which material
posessions really matter to you.  Keep the stuff you need or want (e.g., for
a hobby), and get rid of the stuff that's not a priority  so it's not a
distraction.  This may not seem like it has much to do with programming,
but we'll see that it does.  Here's a few gems in the theories:

<P> There are two ways toward a higher standard of living: earn
$100 more per month, or cut your expenses by $100 per month.  Buth achieve
exactly the same thing: $100 more in your pocket.  Most people adopt the
former strategy, but that means depending on somebody else: you have to
convince them to give you the money.  In contrast, cutting expenses is entirely
under your control.  Having both spouses working means more expenses for
transportation, clothing, food, daycare and unwinding; are you <EM>sure</EM>
your net income is really higher than it would be without that second job?
What about the lost opportunity for the second spouse to pursue a hobby or
be a full-time volunteer?  I love my freedom in not having a car; it gives me
enough money to travel a couple times a year.  Sure it limits where I can
live and work, but those are the places I want to be anyway.

<P> Then there's the question of technology.  The Amish may be a bit too
luddite for most people's taste, but they have a good point: accept new
technology carefully, and only when it's proven its worth.  I love my cell
phone, but my stereo looks like it came from 1987 (which it did).  

<P> This feeds right into environmental sustainability, and the theory of
waste.  Why pay for stuff you don't want (and nobody wants)?  Did you buy
the applesauce for the applesauce itself or for the aluminum can it came in?
Did you buy it because it has an extra plastic seal at the top?  Did you 
buy it because of the energy used and effluent spent to produce the can?
I can't discuss all this properly here, but there's a book,
<I>Natural Capitalism</I> (Lovins, entire text online at
<A HREF="http://www.natcap.org/">www.natcap.org</A>), that's easily the
most important book of the 21st century so far.  It looks at the question
of waste from the individual's, businessman's, and policymaker's 
perspective, and how the (US) accounting and tax system allows companies to
externalize the cost of environmental cleanup, which falsely skews their
profit/loss statements and stock prices.  But it takes only a change in
business model to begin eliminating waste, work with the environment rather
than against it,
<EM>and</EM> turn a greater profit at the same time.  Good stuff, Maynard.

<P> What does all this have to do with simplicity in programming?  The
principles are the same.  Decide what you really want, and look for a 
tool that does <EM>that</EM>.  Maybe SQL is the cat's meow, but do you
really need all the features of MySQL or PosgreSQL?  Maybe you do, but it's
reassuring to have thought out exactly <EM>which</EM> features you need and
<EM>why</EM> you need them.  (Especially when Postgres segfaults and you're
wondering, why did I choose this?)  Or maybe SQL isn't the cat's meow, and
an object database like ZODB, or something even lighter weight like DBM or
pickle/shelve might do the job.

<H2>Brian's Marklarizing webproxy</H2>

To conclude this article, I have to mention Brian's funniest invention.
Here's the wiki entry describing it:

<BLOCKQUOTE>
Several of my friends have a running Marklar joke spawned from a South Park
episode. (<a
href="http://website.lineone.net/~csps/epi311/311_thisismarklar.wav">Example</a>.
For Marklar for the episode review <a class="external"
href="http://www.users.nac.net/msheff/southpark/spepisodes3.html">click
here</a> and search for 'Marklar'.) Short version: There are aliens who use the
word marklar for every noun - and no, it's not confusing. 
<IMG ALT=":)" SRC="../gx/dennis/smily.gif" WIDTH="20" HEIGHT="24"> After a long
weekend of Marklar overdose, I happened to run into a free word list which
included parts of speech (<a
href="http://www.dcs.shef.ac.uk/research/ilash/Moby/">Greg Ward's Moby</a>).
Something clicked in my brain and I decided I had to make a Marklarizing web
proxy, so that the entire internet could be seen as a Marklar would see it.
Eight hours and some pretty horrible code later (mostly on the proxy &amp; HTML
parsing side, but the libraries I used and the program I wrote are only a
couple pages each), I had something that mostly worked. Anyway, I demo'd it at
the meeting, and this was our favorite page.  It's an article from a newspaper.

<p><pre class="code">Two arrested for running marklar-end marklar marklar 
By Marklar Ko 
Marklar Marklars staff marklar 
         
 
Marklar County marklar's detectives have broken up a large marklar marklar  
in the Marklar marklar, and they said they've recovered a "black book" with  
the names of hundreds of marklars, including men who work for marklar marklars  
headquartered in Marklar. 
 
The two marklars of the marklar, a 49-marklar-old woman who lives in Marklar,  
and her 31-marklar-old marklar, who lives in Marklar, were arrested. They have  
not been charged. 
 
The two marklar used the Internet to advertise a marklar-end escort company  
called the "Marklar of Marklar." The Web site had pictures of available  
companions, a calendar of when they were available and marklars. Marklars  
could be made online. 
 
Some of the escorts were brought in from out of state "Las Marklars, New  
Marklar and Los Marklars" to work marklar, according to the Marklar's Marklar. 
 
Marklars were carefully screened, said marklar's Sgt. Marklar Marklar. 
 
For example, potential marklars had to leave a work number, and someone inside  
the marklar marklar would marklar the number, marklar it was to confirm a  
dental marklar, he said. 
 
This was done to make sure the marklar wasn't a police officer, Marklar said. 
 
The Marklar's Marklar marklar not release the names of the marklars the marklars  
worked for, to avoid tainting the companies, Marklar said. 
 
The men in the book, however, marklar likely be contacted soon, detectives said.  
They could face marklar charges of patronizing a prostitute. 
 
In the 1990s, the two marklar were involved in another marklar marklar in the  
marklar called "Affluent Marklars." 
 
An marklar tipped off the Marklar's Marklar about the Marklar of Marklar.
</pre> 

</BLOCKQUOTE>


</p>


<!-- *** BEGIN author bio *** -->
<P>&nbsp;
<P>
<!-- *** BEGIN bio *** -->
<P>
<IMG ALT="picture" SRC="../gx/2003/authors/orr.jpg" WIDTH="235" HEIGHT="333"
   ALIGN="left"  HSPACE="10" VSPACE="10">
<em>
Mike is the Editor-in-Chief of <I>Linux Gazette</I>.  You can read what he has
to say on the Back Page of many issues.  He has been a Linux enthusiast
since 1991 and a Debian user since 1995.  He was SSC's web technical
coordinator 1999-2003, which means he got to write a lot of Python scripts.
Now he's involved in three free software
projects for Python (<A HREF="http://www.cheetahtemplate.org/">Cheetah</A>,
<A HREF="http://webware.sourceforge.net/">Webware</A> and 
<A HREF="http://yaml.org/">YAML</A>), writes unittests and programs for a
Webware e-commerce site, and edits LG from his home.
Non-computer interests include wrestling, ska and oi! and ambient music, and
the international language Esperanto.  He's been known to listen to Dvorak,
Schubert, Mendelssohn, and Khachaturian too.
</em>
<br CLEAR="all">
<!-- *** END bio *** -->

<!-- *** END author bio *** -->

<div id="articlefooter">

<p>
Copyright &copy; 2004, Mike Orr (Sluggo). Copying license 
<a href="http://linuxgazette.net/copying.html">http://linuxgazette.net/copying.html</a>
</p>

<p>
Published in Issue 98 of Linux Gazette, January 2004
</p>

</div>


<div id="previousnextbottom">
<A HREF="moen.html" >&lt;-- prev</A> | <A HREF="pramode.html" >next --&gt;</A>
</div>


</div>






<div id="navigation">

<a href="../index.html">Home</a>
<a href="../faq/index.html">FAQ</a>
<a href="../lg_index.html">Site Map</a>
<a href="../mirrors.html">Mirrors</a>
<a href="../mirrors.html">Translations</a>
<a href="../search.html">Search</a>
<a href="../archives.html">Archives</a>
<a href="../authors/index.html">Authors</a>
<a href="../contact.html">Contact Us</a>

</div>



<div id="breadcrumbs">

<a href="../index.html">Home</a> &gt; 
<a href="index.html">January 2004 (#98)</a> &gt; 
Article

</div>





<img src="../gx/2003/sit3-shine.7-2.gif" id="tux" alt="Tux"/>




</body>
</html>