File: performance.html

package info (click to toggle)
lxml 6.0.2-1
links: PTS
area: main
in suites: forky, sid
size: 32,588 kB
sloc: python: 30,649; javascript: 1,640; ansic: 311; xml: 267; makefile: 243; sh: 68
file content (797 lines) | stat: -rw-r--r-- 64,636 bytes
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.21.2: https://docutils.sourceforge.io/" />
<meta content="Performance evaluation of lxml and ElementTree: fast operations, common pitfalls and optimisation hints." name="description" />
<meta content="Python XML parser performance, XML processing, performance comparison, lxml performance, lxml.etree, lxml.objectify, benchmarks, ElementTree" name="keywords" />
<meta name="author" content="Stefan Behnel" />
<title>Benchmarks and Speed</title>
<link rel="stylesheet" href="style.css" type="text/css" />
<script type="text/javascript">
function trigger_menu(event) {
    var sidemenu = document.getElementById("sidemenu");
    var classes = sidemenu.getAttribute("class");
    classes = (classes.indexOf(" visible") === -1) ? classes + " visible" : classes.replace(" visible", "");
    sidemenu.setAttribute("class", classes);
    event.preventDefault();
    event.stopPropagation();
}
function hide_menu() {
    var sidemenu = document.getElementById("sidemenu");
    var classes = sidemenu.getAttribute("class");
    if (classes.indexOf(" visible") !== -1) {
        sidemenu.setAttribute("class", classes.replace(" visible", ""));
    }
}
</script><meta name="viewport" content="width=device-width, initial-scale=1" /></head>
<body onclick="hide_menu()">
<div class="document" id="benchmarks-and-speed">
<div class="sidemenu" id="sidemenu"><div class="menutrigger" onclick="trigger_menu(event)">Menu</div><div class="menu"><div class="banner"><div class="banner_image"><img src="python-xml-title.png" /></div><div class="banner_link"><a href="index.html#support-the-project">Like the tool? <br class="first" />Help making it better! <br class="second" />Your donation helps!</a></div></div><ul id="lxml-section"><li><span class="section title">lxml</span><ul class="menu foreign" id="index-menu"><li class="menu title"><a href="index.html">lxml</a><ul class="submenu"><li class="menu item"><a href="index.html#introduction">Introduction</a></li><li class="menu item"><a href="index.html#support-the-project">Support the project</a></li><li class="menu item"><a href="index.html#documentation">Documentation</a></li><li class="menu item"><a href="index.html#download">Download</a></li><li class="menu item"><a href="index.html#mailing-list">Mailing list</a></li><li class="menu item"><a href="index.html#bug-tracker">Bug tracker</a></li><li class="menu item"><a href="index.html#license">License</a></li><li class="menu item"><a href="index.html#old-versions">Old Versions</a></li><li class="menu item"><a href="index.html#project-income-report">Project income report</a></li><li class="menu item"><a href="index.html#legal-notice-for-donations">Legal Notice for Donations</a></li></ul></li></ul><ul class="menu foreign" id="intro-menu"><li class="menu title"><a href="intro.html">Why lxml?</a><ul class="submenu"><li class="menu item"><a href="intro.html#motto">Motto</a></li><li class="menu item"><a href="intro.html#aims">Aims</a></li></ul></li></ul><ul class="menu foreign" id="installation-menu"><li class="menu title"><a href="installation.html">Installing lxml</a><ul class="submenu"><li class="menu item"><a href="installation.html#where-to-get-it">Where to get it</a></li><li class="menu item"><a href="installation.html#requirements">Requirements</a></li><li class="menu item"><a href="installation.html#installation">Installation</a></li><li class="menu item"><a href="installation.html#building-lxml-from-dev-sources">Building lxml from dev sources</a></li><li class="menu item"><a href="installation.html#using-lxml-with-python-libxml2">Using lxml with python-libxml2</a></li><li class="menu item"><a href="installation.html#source-builds-on-ms-windows">Source builds on MS Windows</a></li><li class="menu item"><a href="installation.html#source-builds-on-macos-x">Source builds on MacOS-X</a></li></ul></li></ul><ul class="menu current" id="performance-menu"><li class="menu title"><a href="performance.html">Benchmarks and Speed</a><ul class="submenu"><li class="menu item"><a href="performance.html#general-notes">General notes</a></li><li class="menu item"><a href="performance.html#how-to-read-the-timings">How to read the timings</a></li><li class="menu item"><a href="performance.html#parsing-and-serialising">Parsing and Serialising</a></li><li class="menu item"><a href="performance.html#the-elementtree-api">The ElementTree API</a></li><li class="menu item"><a href="performance.html#xpath">XPath</a></li><li class="menu item"><a href="performance.html#a-longer-example">A longer example</a></li><li class="menu item"><a href="performance.html#lxml-objectify">lxml.objectify</a></li></ul></li></ul><ul class="menu foreign" id="compatibility-menu"><li class="menu title"><a href="compatibility.html">ElementTree compatibility of lxml.etree</a></li></ul><ul class="menu foreign" id="FAQ-menu"><li class="menu title"><a href="FAQ.html">lxml FAQ - Frequently Asked Questions</a><ul class="submenu"><li class="menu item"><a href="FAQ.html#general-questions">General Questions</a></li><li class="menu item"><a href="FAQ.html#installation">Installation</a></li><li class="menu item"><a href="FAQ.html#contributing">Contributing</a></li><li class="menu item"><a href="FAQ.html#bugs">Bugs</a></li><li class="menu item"><a href="FAQ.html#threading-1">Threading</a></li><li class="menu item"><a href="FAQ.html#parsing-and-serialisation">Parsing and Serialisation</a></li><li class="menu item"><a href="FAQ.html#xpath-and-document-traversal">XPath and Document Traversal</a></li></ul></li></ul></li></ul><ul id="Developing with lxml-section"><li><span class="section title">Developing with lxml</span><ul class="menu foreign" id="tutorial-menu"><li class="menu title"><a href="tutorial.html">The lxml.etree Tutorial</a><ul class="submenu"><li class="menu item"><a href="tutorial.html#the-element-class">The Element class</a></li><li class="menu item"><a href="tutorial.html#the-elementtree-class">The ElementTree class</a></li><li class="menu item"><a href="tutorial.html#parsing-from-strings-and-files">Parsing from strings and files</a></li><li class="menu item"><a href="tutorial.html#namespaces">Namespaces</a></li><li class="menu item"><a href="tutorial.html#the-e-factory">The E-factory</a></li><li class="menu item"><a href="tutorial.html#elementpath">ElementPath</a></li></ul></li></ul><ul class="menu foreign" id="apidoc lxml-menu"><li class="menu title"><a href="apidoc/lxml.html">API reference</a></li></ul><ul class="menu foreign" id="api-menu"><li class="menu title"><a href="api.html">APIs specific to lxml.etree</a><ul class="submenu"><li class="menu item"><a href="api.html#lxml-etree">lxml.etree</a></li><li class="menu item"><a href="api.html#other-element-apis">Other Element APIs</a></li><li class="menu item"><a href="api.html#trees-and-documents">Trees and Documents</a></li><li class="menu item"><a href="api.html#iteration">Iteration</a></li><li class="menu item"><a href="api.html#error-handling-on-exceptions">Error handling on exceptions</a></li><li class="menu item"><a href="api.html#error-logging">Error logging</a></li><li class="menu item"><a href="api.html#serialisation">Serialisation</a></li><li class="menu item"><a href="api.html#incremental-xml-generation">Incremental XML generation</a></li><li class="menu item"><a href="api.html#cdata">CDATA</a></li><li class="menu item"><a href="api.html#xinclude-and-elementinclude">XInclude and ElementInclude</a></li></ul></li></ul><ul class="menu foreign" id="parsing-menu"><li class="menu title"><a href="parsing.html">Parsing XML and HTML with lxml</a><ul class="submenu"><li class="menu item"><a href="parsing.html#parsers">Parsers</a></li><li class="menu item"><a href="parsing.html#the-target-parser-interface">The target parser interface</a></li><li class="menu item"><a href="parsing.html#the-feed-parser-interface">The feed parser interface</a></li><li class="menu item"><a href="parsing.html#incremental-event-parsing">Incremental event parsing</a></li><li class="menu item"><a href="parsing.html#iterparse-and-iterwalk">iterparse and iterwalk</a></li><li class="menu item"><a href="parsing.html#python-unicode-strings">Python unicode strings</a></li></ul></li></ul><ul class="menu foreign" id="validation-menu"><li class="menu title"><a href="validation.html">Validation with lxml</a><ul class="submenu"><li class="menu item"><a href="validation.html#validation-at-parse-time">Validation at parse time</a></li><li class="menu item"><a href="validation.html#dtd-1">DTD</a></li><li class="menu item"><a href="validation.html#relaxng">RelaxNG</a></li><li class="menu item"><a href="validation.html#xmlschema">XMLSchema</a></li><li class="menu item"><a href="validation.html#schematron-1">Schematron</a></li><li class="menu item"><a href="validation.html#pre-iso-schematron-1">(Pre-ISO-Schematron)</a></li></ul></li></ul><ul class="menu foreign" id="xpathxslt-menu"><li class="menu title"><a href="xpathxslt.html">XPath and XSLT with lxml</a><ul class="submenu"><li class="menu item"><a href="xpathxslt.html#xpath">XPath</a></li><li class="menu item"><a href="xpathxslt.html#xslt">XSLT</a></li></ul></li></ul><ul class="menu foreign" id="objectify-menu"><li class="menu title"><a href="objectify.html">lxml.objectify</a><ul class="submenu"><li class="menu item"><a href="objectify.html#the-lxml-objectify-api">The lxml.objectify API</a></li><li class="menu item"><a href="objectify.html#asserting-a-schema">Asserting a Schema</a></li><li class="menu item"><a href="objectify.html#objectpath">ObjectPath</a></li><li class="menu item"><a href="objectify.html#python-data-types">Python data types</a></li><li class="menu item"><a href="objectify.html#how-data-types-are-matched">How data types are matched</a></li><li class="menu item"><a href="objectify.html#what-is-different-from-lxml-etree">What is different from lxml.etree?</a></li></ul></li></ul><ul class="menu foreign" id="lxmlhtml-menu"><li class="menu title"><a href="lxmlhtml.html">lxml.html</a><ul class="submenu"><li class="menu item"><a href="lxmlhtml.html#parsing-html">Parsing HTML</a></li><li class="menu item"><a href="lxmlhtml.html#html-element-methods">HTML Element Methods</a></li><li class="menu item"><a href="lxmlhtml.html#running-html-doctests">Running HTML doctests</a></li><li class="menu item"><a href="lxmlhtml.html#creating-html-with-the-e-factory">Creating HTML with the E-factory</a></li><li class="menu item"><a href="lxmlhtml.html#working-with-links">Working with links</a></li><li class="menu item"><a href="lxmlhtml.html#forms">Forms</a></li><li class="menu item"><a href="lxmlhtml.html#html-diff">HTML Diff</a></li><li class="menu item"><a href="lxmlhtml.html#examples">Examples</a></li></ul></li></ul><ul class="menu foreign" id="cssselect-menu"><li class="menu title"><a href="cssselect.html">lxml.cssselect</a><ul class="submenu"><li class="menu item"><a href="cssselect.html#the-cssselector-class">The CSSSelector class</a></li><li class="menu item"><a href="cssselect.html#the-cssselect-method">The cssselect method</a></li><li class="menu item"><a href="cssselect.html#supported-selectors">Supported Selectors</a></li><li class="menu item"><a href="cssselect.html#namespaces">Namespaces</a></li></ul></li></ul><ul class="menu foreign" id="elementsoup-menu"><li class="menu title"><a href="elementsoup.html">BeautifulSoup Parser</a><ul class="submenu"><li class="menu item"><a href="elementsoup.html#parsing-with-the-soupparser">Parsing with the soupparser</a></li><li class="menu item"><a href="elementsoup.html#entity-handling">Entity handling</a></li><li class="menu item"><a href="elementsoup.html#using-soupparser-as-a-fallback">Using soupparser as a fallback</a></li><li class="menu item"><a href="elementsoup.html#using-only-the-encoding-detection">Using only the encoding detection</a></li></ul></li></ul><ul class="menu foreign" id="html5parser-menu"><li class="menu title"><a href="html5parser.html">html5lib Parser</a><ul class="submenu"><li class="menu item"><a href="html5parser.html#differences-to-regular-html-parsing">Differences to regular HTML parsing</a></li><li class="menu item"><a href="html5parser.html#function-reference">Function Reference</a></li></ul></li></ul></li></ul><ul id="Extending lxml-section"><li><span class="section title">Extending lxml</span><ul class="menu foreign" id="resolvers-menu"><li class="menu title"><a href="resolvers.html">Document loading and URL resolving</a><ul class="submenu"><li class="menu item"><a href="resolvers.html#xml-catalogs">XML Catalogs</a></li><li class="menu item"><a href="resolvers.html#uri-resolvers">URI Resolvers</a></li><li class="menu item"><a href="resolvers.html#document-loading-in-context">Document loading in context</a></li><li class="menu item"><a href="resolvers.html#i-o-access-control-in-xslt">I/O access control in XSLT</a></li></ul></li></ul><ul class="menu foreign" id="extensions-menu"><li class="menu title"><a href="extensions.html">Python extensions for XPath and XSLT</a><ul class="submenu"><li class="menu item"><a href="extensions.html#xpath-extension-functions">XPath Extension functions</a></li><li class="menu item"><a href="extensions.html#xslt-extension-elements">XSLT extension elements</a></li></ul></li></ul><ul class="menu foreign" id="element classes-menu"><li class="menu title"><a href="element_classes.html">Using custom Element classes in lxml</a><ul class="submenu"><li class="menu item"><a href="element_classes.html#background-on-element-proxies">Background on Element proxies</a></li><li class="menu item"><a href="element_classes.html#element-initialization">Element initialization</a></li><li class="menu item"><a href="element_classes.html#setting-up-a-class-lookup-scheme">Setting up a class lookup scheme</a></li><li class="menu item"><a href="element_classes.html#generating-xml-with-custom-classes">Generating XML with custom classes</a></li><li class="menu item"><a href="element_classes.html#implementing-namespaces-1">Implementing namespaces</a></li></ul></li></ul><ul class="menu foreign" id="sax-menu"><li class="menu title"><a href="sax.html">Sax support</a><ul class="submenu"><li class="menu item"><a href="sax.html#building-a-tree-from-sax-events">Building a tree from SAX events</a></li><li class="menu item"><a href="sax.html#producing-sax-events-from-an-elementtree-or-element">Producing SAX events from an ElementTree or Element</a></li><li class="menu item"><a href="sax.html#interfacing-with-pulldom-minidom">Interfacing with pulldom/minidom</a></li></ul></li></ul><ul class="menu foreign" id="capi-menu"><li class="menu title"><a href="capi.html">The public C-API of lxml.etree</a><ul class="submenu"><li class="menu item"><a href="capi.html#passing-generated-trees-through-python">Passing generated trees through Python</a></li><li class="menu item"><a href="capi.html#writing-external-modules-in-cython">Writing external modules in Cython</a></li><li class="menu item"><a href="capi.html#writing-external-modules-in-c">Writing external modules in C</a></li></ul></li></ul></li></ul><ul id="Developing lxml-section"><li><span class="section title">Developing lxml</span><ul class="menu foreign" id="build-menu"><li class="menu title"><a href="build.html">How to build lxml from source</a><ul class="submenu"><li class="menu item"><a href="build.html#cython">Cython</a></li><li class="menu item"><a href="build.html#github-git-and-hg">Github, git and hg</a></li><li class="menu item"><a href="build.html#building-the-sources">Building the sources</a></li><li class="menu item"><a href="build.html#running-the-tests-and-reporting-errors">Running the tests and reporting errors</a></li><li class="menu item"><a href="build.html#building-an-egg-or-wheel">Building an egg or wheel</a></li><li class="menu item"><a href="build.html#building-lxml-on-macos-x">Building lxml on MacOS-X</a></li><li class="menu item"><a href="build.html#static-linking-on-windows">Static linking on Windows</a></li><li class="menu item"><a href="build.html#building-debian-packages-from-svn-sources">Building Debian packages from SVN sources</a></li></ul></li></ul><ul class="menu foreign" id="lxml source howto-menu"><li class="menu title"><a href="lxml-source-howto.html">How to read the source of lxml</a><ul class="submenu"><li class="menu item"><a href="lxml-source-howto.html#what-is-cython">What is Cython?</a></li><li class="menu item"><a href="lxml-source-howto.html#where-to-start">Where to start?</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-etree">lxml.etree</a></li><li class="menu item"><a href="lxml-source-howto.html#python-modules">Python modules</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-objectify">lxml.objectify</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-html">lxml.html</a></li></ul></li></ul><ul class="menu foreign" id="changes 6 0 2-menu"><li class="menu title"><a href="changes-6.0.2.html">Release Changelog</a></li></ul><ul class="menu foreign" id="credits-menu"><li class="menu title"><a href="credits.html">Credits</a><ul class="submenu"><li class="menu item"><a href="credits.html#main-contributors">Main contributors</a></li><li class="menu item"><a href="credits.html#special-thanks-goes-to">Special thanks goes to:</a></li></ul></li></ul></li><li><a href="/sitemap.html">Sitemap</a></li></ul></div></div><div class="banner"><div class="banner_image"><img src="python-xml-title.png" /></div><div class="banner_link"><a href="index.html#support-the-project">Like the tool? <br class="first" />Help making it better! <br class="second" />Your donation helps!</a></div></div><h1 class="title">Benchmarks and Speed</h1>
<table class="docinfo" frame="void" rules="none">
<col class="docinfo-name" />
<col class="docinfo-content" />
<tbody valign="top">
<tr><th class="docinfo-name">Author:</th>
<td>Stefan Behnel</td></tr>
</tbody>
</table>
<p>lxml.etree is a very fast XML library.  Most of this is due to the
speed of libxml2, e.g. the parser and serialiser, or the XPath engine.
Other areas of lxml were specifically written for high performance in
high-level operations, such as the tree iterators.</p>
<p>On the other hand, the simplicity of lxml sometimes hides internal
operations that are more costly than the API suggests.  If you are not
aware of these cases, lxml may not always perform as you expect.  A
common example in the Python world is the Python list type.  New users
often expect it to be a linked list, while it actually is implemented
as an array, which results in a completely different complexity for
common operations.</p>
<p>Similarly, the tree model of libxml2 is more complex than what lxml's
ElementTree API projects into Python space, so some operations may
show unexpected performance.  Rest assured that most lxml users will
not notice this in real life, as lxml is very fast in absolute
numbers.  It is definitely fast enough for most applications, so lxml
is probably somewhere between 'fast enough' and 'the best choice' for
yours.  Read some <a class="reference external" href="http://permalink.gmane.org/gmane.comp.python.lxml.devel/3250">messages</a> from <a class="reference external" href="http://article.gmane.org/gmane.comp.python.lxml.devel/3246">happy</a> <a class="reference external" href="http://thread.gmane.org/gmane.comp.python.lxml.devel/3244/focus=3244">users</a> to see what we mean.</p>
<p>This text describes where lxml.etree (abbreviated to 'lxe') excels, gives
hints on some performance traps and compares the overall performance to the
original <a class="reference external" href="http://effbot.org/zone/element-index.htm">ElementTree</a> (ET) and <a class="reference external" href="http://effbot.org/zone/celementtree.htm">cElementTree</a> (cET) libraries by Fredrik Lundh.
The cElementTree library is a fast C-implementation of the original
ElementTree.</p>
<div class="contents topic" id="contents">
<p class="topic-title">Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#general-notes" id="toc-entry-1">General notes</a></li>
<li><a class="reference internal" href="#how-to-read-the-timings" id="toc-entry-2">How to read the timings</a></li>
<li><a class="reference internal" href="#parsing-and-serialising" id="toc-entry-3">Parsing and Serialising</a></li>
<li><a class="reference internal" href="#the-elementtree-api" id="toc-entry-4">The ElementTree API</a><ul>
<li><a class="reference internal" href="#child-access" id="toc-entry-5">Child access</a></li>
<li><a class="reference internal" href="#element-creation" id="toc-entry-6">Element creation</a></li>
<li><a class="reference internal" href="#merging-different-sources" id="toc-entry-7">Merging different sources</a></li>
<li><a class="reference internal" href="#deepcopy" id="toc-entry-8">deepcopy</a></li>
<li><a class="reference internal" href="#tree-traversal" id="toc-entry-9">Tree traversal</a></li>
</ul>
</li>
<li><a class="reference internal" href="#xpath" id="toc-entry-10">XPath</a></li>
<li><a class="reference internal" href="#a-longer-example" id="toc-entry-11">A longer example</a></li>
<li><a class="reference internal" href="#lxml-objectify" id="toc-entry-12">lxml.objectify</a><ul>
<li><a class="reference internal" href="#objectpath" id="toc-entry-13">ObjectPath</a></li>
<li><a class="reference internal" href="#caching-elements" id="toc-entry-14">Caching Elements</a></li>
<li><a class="reference internal" href="#further-optimisations" id="toc-entry-15">Further optimisations</a></li>
</ul>
</li>
</ul>
</div>
<div class="section" id="general-notes">
<h1>General notes</h1>
<p>First thing to say: there <em>is</em> an overhead involved in having a DOM-like C
library mimic the ElementTree API.  As opposed to ElementTree, lxml has to
generate Python representations of tree nodes on the fly when asked for them,
and the internal tree structure of libxml2 results in a higher maintenance
overhead than the simpler top-down structure of ElementTree.  What this means
is: the more of your code runs in Python, the less you can benefit from the
speed of lxml and libxml2.  Note, however, that this is true for most
performance critical Python applications.  No one would implement Fourier
transformations in pure Python when you can use NumPy.</p>
<p>The up side then is that lxml provides powerful tools like tree iterators,
XPath and XSLT, that can handle complex operations at the speed of C.  Their
pythonic API in lxml makes them so flexible that most applications can easily
benefit from them.</p>
</div>
<div class="section" id="how-to-read-the-timings">
<h1>How to read the timings</h1>
<p>The statements made here are backed by the (micro-)benchmark scripts
<a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_etree.py">bench_etree.py</a>, <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_xpath.py">bench_xpath.py</a> and <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_objectify.py">bench_objectify.py</a> that come with
the lxml source distribution.  They are distributed under the same BSD license
as lxml itself, and the lxml project would like to promote them as a general
benchmarking suite for all ElementTree implementations.  New benchmarks are
very easy to add as tiny test methods, so if you write a performance test for
a specific part of the API yourself, please consider sending it to the lxml
mailing list.</p>
<p>The timings presented below compare lxml 4.6.3 (with libxml2 2.9.10) to the
latest released versions of ElementTree (with cElementTree as accelerator
module) in the standard library of CPython 3.8.10.  They were run
single-threaded on a 2.3GHz 64bit double core Intel i5 machine under
Ubuntu Linux 20.04 (Focal).</p>
<p>The scripts run a number of simple tests on the different libraries, using
different XML tree configurations: different tree sizes (T1-4), with or
without attributes (-/A), with or without ASCII string or unicode text
(-/S/U), and either against a tree or its serialised XML form (T/X).  In the
result extracts cited below, T1 refers to a 3-level tree with many children at
the third level, T2 is swapped around to have many children below the root
element, T3 is a deep tree with few children at each level and T4 is a small
tree, slightly broader than deep.  If repetition is involved, this usually
means running the benchmark in a loop over all children of the tree root,
otherwise, the operation is run on the root node (C/R).</p>
<p>As an example, the character code <tt class="docutils literal">(SATR T1)</tt> states that the benchmark was
running for tree T1, with plain string text (S) and attributes (A).  It was
run against the root element (R) in the tree structure of the data (T).</p>
<p>Note that very small operations are repeated in integer loops to make them
measurable.  It is therefore not always possible to compare the absolute
timings of, say, a single access benchmark (which usually loops) and a 'get
all in one step' benchmark, which already takes enough time to be measurable
and is therefore measured as is.  An example is the index access to a single
child, which cannot be compared to the timings for <tt class="docutils literal">getchildren()</tt>.  Take a
look at the concrete benchmarks in the scripts to understand how the numbers
compare.</p>
</div>
<div class="section" id="parsing-and-serialising">
<h1>Parsing and Serialising</h1>
<p>Serialisation is an area where lxml excels.  The reason is that it
executes entirely at the C level, without any interaction with Python
code.  The results are rather impressive, especially for UTF-8, which
is native to libxml2.  While 20 to 40 times faster than (c)ElementTree
1.2 (which was part of the standard library before Python 2.7/3.2),
lxml is still several times faster than the much improved
ElementTree 1.3 in recent Python versions:</p>
<pre class="literal-block">
lxe: tostring_utf16  (S-TR T1)    5.9340 msec/pass
cET: tostring_utf16  (S-TR T1)   38.3270 msec/pass

lxe: tostring_utf16  (UATR T1)    6.2032 msec/pass
cET: tostring_utf16  (UATR T1)   37.7944 msec/pass

lxe: tostring_utf16  (S-TR T2)    6.1841 msec/pass
cET: tostring_utf16  (S-TR T2)   40.2577 msec/pass

lxe: tostring_utf8   (S-TR T2)    4.6697 msec/pass
cET: tostring_utf8   (S-TR T2)   30.5173 msec/pass

lxe: tostring_utf8   (U-TR T3)    1.2085 msec/pass
cET: tostring_utf8   (U-TR T3)   9.0246 msec/pass
</pre>
<p>The difference is somewhat smaller for plain text serialisation:</p>
<pre class="literal-block">
lxe: tostring_text_ascii     (S-TR T1)    2.6727 msec/pass
cET: tostring_text_ascii     (S-TR T1)    2.9683 msec/pass

lxe: tostring_text_ascii     (S-TR T3)    0.6952 msec/pass
cET: tostring_text_ascii     (S-TR T3)    1.0073 msec/pass

lxe: tostring_text_utf16     (S-TR T1)    2.7366 msec/pass
cET: tostring_text_utf16     (S-TR T1)   7.3647 msec/pass

lxe: tostring_text_utf16     (U-TR T1)    3.0322 msec/pass
cET: tostring_text_utf16     (U-TR T1)   7.5922 msec/pass
</pre>
<p>The <tt class="docutils literal">tostring()</tt> function also supports serialisation to a Python
unicode string object, which is currently faster in ElementTree
under CPython 3.8:</p>
<pre class="literal-block">
lxe: tostring_text_unicode   (S-TR T1)    2.7645 msec/pass
cET: tostring_text_unicode   (S-TR T1)    1.1806 msec/pass

lxe: tostring_text_unicode   (U-TR T1)    2.9871 msec/pass
cET: tostring_text_unicode   (U-TR T1)    1.1659 msec/pass

lxe: tostring_text_unicode   (S-TR T3)    0.7446 msec/pass
cET: tostring_text_unicode   (S-TR T3)    0.4532 msec/pass

lxe: tostring_text_unicode   (U-TR T4)    0.0048 msec/pass
cET: tostring_text_unicode   (U-TR T4)    0.0134 msec/pass
</pre>
<p>For parsing, lxml.etree and cElementTree compete for the medal.
Depending on the input, either of the two can be faster.  The (c)ET
libraries use a very thin layer on top of the expat parser, which is
known to be very fast.  Here are some timings from the benchmarking
suite:</p>
<pre class="literal-block">
lxe: parse_bytesIO   (SAXR T1)   14.2074 msec/pass
cET: parse_bytesIO   (SAXR T1)    7.9336 msec/pass

lxe: parse_bytesIO   (S-XR T3)    1.4477 msec/pass
cET: parse_bytesIO   (S-XR T3)    2.1925 msec/pass

lxe: parse_bytesIO   (UAXR T3)    8.4128 msec/pass
cET: parse_bytesIO   (UAXR T3)   12.2926 msec/pass
</pre>
<p>And another couple of timings <a class="reference external" href="http://svn.effbot.org/public/elementtree-1.3/benchmark.py">from a benchmark</a> that Fredrik Lundh
<a class="reference external" href="http://effbot.org/zone/celementtree.htm#benchmarks">used to promote cElementTree</a>, comparing a number of different
parsers.  First, parsing a 274KB XML file containing Shakespeare's
Hamlet:</p>
<pre class="literal-block">
xml.etree.ElementTree.parse done in 0.006 seconds
xml.etree.cElementTree.parse done in 0.007 seconds
xml.etree.cElementTree.XMLParser.feed(): 6636 nodes read in 0.006 seconds
lxml.etree.parse done in 0.004 seconds
drop_whitespace.parse done in 0.004 seconds
lxml.etree.XMLParser.feed(): 6636 nodes read in 0.004 seconds
minidom tree read in 0.066 seconds
</pre>
<p>And a 3.4MB XML file containing the Old Testament:</p>
<pre class="literal-block">
xml.etree.ElementTree.parse done in 0.037 seconds
xml.etree.cElementTree.parse done in 0.036 seconds
xml.etree.cElementTree.XMLParser.feed(): 25317 nodes read in 0.036 seconds
lxml.etree.parse done in 0.025 seconds
drop_whitespace.parse done in 0.022 seconds
lxml.etree.XMLParser.feed(): 25317 nodes read in 0.026 seconds
minidom tree read in 0.194 seconds
</pre>
<p>Here are the same benchmarks again, but including the memory usage
of the process in KB before and after parsing (using os.fork() to
make sure we start from a clean state each time).  For the 274KB
hamlet.xml file:</p>
<pre class="literal-block">
Memory usage: 9256
xml.etree.ElementTree.parse done in 0.006 seconds
Memory usage: 12764 (+3508)
xml.etree.cElementTree.parse done in 0.007 seconds
Memory usage: 12764 (+3508)
xml.etree.cElementTree.XMLParser.feed(): 6636 nodes read in 0.006 seconds
Memory usage: 12720 (+3464)
lxml.etree.parse done in 0.004 seconds
Memory usage: 15052 (+5796)
drop_whitespace.parse done in 0.004 seconds
Memory usage: 14040 (+4784)
lxml.etree.XMLParser.feed(): 6636 nodes read in 0.004 seconds
Memory usage: 15812 (+6556)
minidom tree read in 0.066 seconds
Memory usage: 15332 (+6076)
</pre>
<p>And for the 3.4MB Old Testament XML file:</p>
<pre class="literal-block">
Memory usage: 12456
xml.etree.ElementTree.parse done in 0.037 seconds
Memory usage: 23288 (+10832)
xml.etree.cElementTree.parse done in 0.036 seconds
Memory usage: 23288 (+10832)
xml.etree.cElementTree.XMLParser.feed(): 25317 nodes read in 0.036 seconds
Memory usage: 23644 (+11220)
lxml.etree.parse done in 0.025 seconds
Memory usage: 31404 (+18948)
drop_whitespace.parse done in 0.022 seconds
Memory usage: 28752 (+16296)
lxml.etree.XMLParser.feed(): 25317 nodes read in 0.026 seconds
Memory usage: 33924 (+21500)
minidom tree read in 0.194 seconds
Memory usage: 31284 (+18828)
</pre>
<p>As can be seen from the sizes, both lxml.etree and cElementTree are
rather memory friendly and fast.  Comparing to older CPython
versions, the memory footprint of the minidom library was considerably
reduced in CPython 3.3, by about a factor of 4 in this case.</p>
<p>For plain parser performance, lxml.etree and cElementTree tend to stay
rather close to each other, usually within a factor of two, with
winners well distributed over both sides.  Similar timings can be
observed for the <tt class="docutils literal">iterparse()</tt> function:</p>
<pre class="literal-block">
lxe: iterparse_bytesIO   (SAXR T1)   20.3598 msec/pass
cET: iterparse_bytesIO   (SAXR T1)   10.8948 msec/pass

lxe: iterparse_bytesIO   (UAXR T3)    10.1640 msec/pass
cET: iterparse_bytesIO   (UAXR T3)   12.9926 msec/pass
</pre>
<p>However, if you benchmark the complete round-trip of a serialise-parse
cycle, the numbers will look similar to these:</p>
<pre class="literal-block">
lxe: write_utf8_parse_bytesIO   (S-TR T1)   18.9857 msec/pass
cET: write_utf8_parse_bytesIO   (S-TR T1)   35.7475 msec/pass

lxe: write_utf8_parse_bytesIO   (UATR T2)   22.4853 msec/pass
cET: write_utf8_parse_bytesIO   (UATR T2)   42.6254 msec/pass

lxe: write_utf8_parse_bytesIO   (S-TR T3)    3.3801 msec/pass
cET: write_utf8_parse_bytesIO   (S-TR T3)   11.2493 msec/pass

lxe: write_utf8_parse_bytesIO   (SATR T4)    0.4263 msec/pass
cET: write_utf8_parse_bytesIO   (SATR T4)    1.0326 msec/pass
</pre>
<p>For applications that require a high parser throughput of large files,
and that do little to no serialization, both cET and lxml.etree are a
good choice.  The cET library is particularly fast for iterparse
applications that extract small amounts of data or aggregate
information from large XML data sets that do not fit into memory.  If
it comes to round-trip performance, however, lxml is multiple times
faster in total.  So, whenever the input documents are not
considerably larger than the output, lxml is the clear winner.</p>
<p>Regarding HTML parsing, Ian Bicking has done some <a class="reference external" href="http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/">benchmarking on
lxml's HTML parser</a>, comparing it to a number of other famous HTML
parser tools for Python.  lxml wins this contest by quite a length.
To give an idea, the numbers suggest that lxml.html can run a couple
of parse-serialise cycles in the time that other tools need for
parsing alone.  The comparison even shows some very favourable results
regarding memory consumption.</p>
<p>Liza Daly has written an article that presents a couple of tweaks to
get the most out of lxml's parser for very large XML documents.  She
quite favourably positions <tt class="docutils literal">lxml.etree</tt> as a tool for
<a class="reference external" href="http://www.ibm.com/developerworks/xml/library/x-hiperfparse/">high-performance XML parsing</a>.</p>
<p>Finally, <a class="reference external" href="http://www.xml.com/">xml.com</a> has a couple of publications about XML parser
performance.  Farwick and Hafner have written two interesting articles
that compare the parser of libxml2 to some major Java based XML
parsers.  One deals with <a class="reference external" href="http://www.xml.com/lpt/a/1702">event-driven parser performance</a>, the other
one presents <a class="reference external" href="http://www.xml.com/lpt/a/1703">benchmark results comparing DOM parsers</a>.  Both
comparisons suggest that libxml2's parser performance is largely
superior to all commonly used Java parsers in almost all cases.  Note
that the C parser benchmark results are based on <a class="reference external" href="http://xmlbench.sourceforge.net/">xmlbench</a>, which uses
a simpler setup for libxml2 than lxml does.</p>
</div>
<div class="section" id="the-elementtree-api">
<h1>The ElementTree API</h1>
<p>Since all three libraries implement the same API, their performance is
easy to compare in this area.  A major disadvantage for lxml's
performance is the different tree model that underlies libxml2.  It
allows lxml to provide parent pointers for elements and full XPath
support, but also increases the overhead of tree building and
restructuring.  This can be seen from the tree setup times of the
benchmark (given in seconds):</p>
<pre class="literal-block">
lxe:       --     S-     U-     -A     SA     UA
     T1: 0.0219 0.0254 0.0257 0.0216 0.0259 0.0259
     T2: 0.0234 0.0279 0.0283 0.0271 0.0318 0.0307
     T3: 0.0051 0.0050 0.0058 0.0218 0.0233 0.0231
     T4: 0.0001 0.0001 0.0001 0.0004 0.0004 0.0004
cET:       --     S-     U-     -A     SA     UA
     T1: 0.0035 0.0029 0.0078 0.0031 0.0031 0.0029
     T2: 0.0047 0.0051 0.0053 0.0046 0.0055 0.0048
     T3: 0.0016 0.0216 0.0027 0.0021 0.0023 0.0026
     T4: 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
</pre>
<p>The timings are somewhat close to each other, although cET can be
several times faster than lxml for larger trees.  One of the
reasons is that lxml must encode incoming string data and tag names
into UTF-8, and additionally discard the created Python elements
after their use, when they are no longer referenced.  ElementTree
represents the tree itself through these objects, which reduces
the overhead in creating them.</p>
<div class="section" id="child-access">
<h2>Child access</h2>
<p>The same tree overhead makes operations like collecting children as in
<tt class="docutils literal">list(element)</tt> more costly in lxml.  Where cET can quickly create
a shallow copy of their list of children, lxml has to create a Python
object for each child and collect them in a list:</p>
<pre class="literal-block">
lxe: root_list_children        (--TR T1)    0.0036 msec/pass
cET: root_list_children        (--TR T1)    0.0005 msec/pass

lxe: root_list_children        (--TR T2)    0.0634 msec/pass
cET: root_list_children        (--TR T2)    0.0086 msec/pass
</pre>
<p>This handicap is also visible when accessing single children:</p>
<pre class="literal-block">
lxe: first_child               (--TR T2)    0.0601 msec/pass
cET: first_child               (--TR T2)    0.0548 msec/pass

lxe: last_child                (--TR T1)    0.0570 msec/pass
cET: last_child                (--TR T1)    0.0534 msec/pass
</pre>
<p>... unless you also add the time to find a child index in a bigger
list.  ET and cET use Python lists here, which are based on arrays.
The data structure used by libxml2 is a linked tree, and thus, a
linked list of children:</p>
<pre class="literal-block">
lxe: middle_child              (--TR T1)    0.0892 msec/pass
cET: middle_child              (--TR T1)    0.0510 msec/pass

lxe: middle_child              (--TR T2)    2.3038 msec/pass
cET: middle_child              (--TR T2)    0.0508 msec/pass
</pre>
</div>
<div class="section" id="element-creation">
<h2>Element creation</h2>
<p>As opposed to ET, libxml2 has a notion of documents that each element must be
in.  This results in a major performance difference for creating independent
Elements that end up in independently created documents:</p>
<pre class="literal-block">
lxe: create_elements           (--TC T2)    0.8032 msec/pass
cET: create_elements           (--TC T2)    0.0675 msec/pass
</pre>
<p>Therefore, it is always preferable to create Elements for the document they
are supposed to end up in, either as SubElements of an Element or using the
explicit <tt class="docutils literal">Element.makeelement()</tt> call:</p>
<pre class="literal-block">
lxe: makeelement               (--TC T2)    0.8030 msec/pass
cET: makeelement               (--TC T2)    0.0625 msec/pass

lxe: create_subelements        (--TC T2)    0.8621 msec/pass
cET: create_subelements        (--TC T2)    0.0923 msec/pass
</pre>
<p>So, if the main performance bottleneck of an application is creating large XML
trees in memory through calls to Element and SubElement, cET is the best
choice.  Note, however, that the serialisation performance may even out this
advantage, especially for smaller trees and trees with many attributes.</p>
</div>
<div class="section" id="merging-different-sources">
<h2>Merging different sources</h2>
<p>A critical action for lxml is moving elements between document contexts.  It
requires lxml to do recursive adaptations throughout the moved tree structure.</p>
<p>The following benchmark appends all root children of the second tree to the
root of the first tree:</p>
<pre class="literal-block">
lxe: append_from_document      (--TR T1,T2)    1.3800 msec/pass
cET: append_from_document      (--TR T1,T2)    0.0513 msec/pass

lxe: append_from_document      (--TR T3,T4)    0.0150 msec/pass
cET: append_from_document      (--TR T3,T4)    0.0026 msec/pass
</pre>
<p>Although these are fairly small numbers compared to parsing, this easily shows
the different performance classes for lxml and (c)ET.  Where the latter do not
have to care about parent pointers and tree structures, lxml has to deep
traverse the appended tree.  The performance difference therefore increases
with the size of the tree that is moved.</p>
<p>This difference is not always as visible, but applies to most parts of the
API, like inserting newly created elements:</p>
<pre class="literal-block">
lxe: insert_from_document         (--TR T1,T2)    5.2345 msec/pass
cET: insert_from_document         (--TR T1,T2)    0.0732 msec/pass
</pre>
<p>or replacing the child slice by a newly created element:</p>
<pre class="literal-block">
lxe: replace_children_element   (--TC T1)    0.0720 msec/pass
cET: replace_children_element   (--TC T1)    0.0105 msec/pass
</pre>
<p>as opposed to replacing the slice with an existing element from the
same document:</p>
<pre class="literal-block">
lxe: replace_children           (--TC T1)    0.0060 msec/pass
cET: replace_children           (--TC T1)    0.0050 msec/pass
</pre>
<p>While these numbers are too small to provide a major performance
impact in practice, you should keep this difference in mind when you
merge very large trees.  Note that Elements have a <tt class="docutils literal">makeelement()</tt>
method that allows to create an Element within the same document,
thus avoiding the merge overhead when inserting it into that tree.</p>
</div>
<div class="section" id="deepcopy">
<h2>deepcopy</h2>
<p>Deep copying a tree is fast in lxml:</p>
<pre class="literal-block">
lxe: deepcopy_all              (--TR T1)    4.1246 msec/pass
cET: deepcopy_all              (--TR T1)   2.5451 msec/pass

lxe: deepcopy_all              (-ATR T2)    4.7867 msec/pass
cET: deepcopy_all              (-ATR T2)   2.7504 msec/pass

lxe: deepcopy_all              (S-TR T3)    1.0097 msec/pass
cET: deepcopy_all              (S-TR T3)   0.6278 msec/pass
</pre>
<p>So, for example, if you have a database-like scenario where you parse in a
large tree and then search and copy independent subtrees from it for further
processing, lxml is by far the best choice here.</p>
</div>
<div class="section" id="tree-traversal">
<h2>Tree traversal</h2>
<p>Another important area in XML processing is iteration for tree
traversal.  If your algorithms can benefit from step-by-step
traversal of the XML tree and especially if few elements are of
interest or the target element tag name is known, the <tt class="docutils literal">.iter()</tt>
method is a good choice:</p>
<pre class="literal-block">
lxe: iter_all             (--TR T1)    1.3661 msec/pass
cET: iter_all             (--TR T1)    0.2670 msec/pass

lxe: iter_islice          (--TR T2)    0.0122 msec/pass
cET: iter_islice          (--TR T2)    0.0033 msec/pass

lxe: iter_tag             (--TR T2)    0.0098 msec/pass
cET: iter_tag             (--TR T2)    0.0086 msec/pass

lxe: iter_tag_all         (--TR T2)    0.6840 msec/pass
cET: iter_tag_all         (--TR T2)    0.4323 msec/pass
</pre>
<p>This translates directly into similar timings for <tt class="docutils literal">Element.findall()</tt>:</p>
<pre class="literal-block">
lxe: findall              (--TR T2)    3.9611 msec/pass
cET: findall              (--TR T2)    0.9227 msec/pass

lxe: findall              (--TR T3)    0.3989 msec/pass
cET: findall              (--TR T3)    0.2670 msec/pass

lxe: findall_tag          (--TR T2)    0.7420 msec/pass
cET: findall_tag          (--TR T2)    0.4942 msec/pass

lxe: findall_tag          (--TR T3)    0.1099 msec/pass
cET: findall_tag          (--TR T3)    0.1748 msec/pass
</pre>
<p>Note that all three libraries currently use the same Python
implementation for <tt class="docutils literal">.findall()</tt>, except for their native tree
iterator (<tt class="docutils literal">element.iter()</tt>).  In general, lxml is very fast
for iteration, but loses ground against cET when many Elements
are found and need to be instantiated.  So, the more selective
your search is, the faster lxml will run.</p>
</div>
</div>
<div class="section" id="xpath">
<h1>XPath</h1>
<p>The following timings are based on the benchmark script <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_xpath.py">bench_xpath.py</a>.</p>
<p>This part of lxml does not have an equivalent in ElementTree.  However, lxml
provides more than one way of accessing it and you should take care which part
of the lxml API you use.  The most straight forward way is to call the
<tt class="docutils literal">xpath()</tt> method on an Element or ElementTree:</p>
<pre class="literal-block">
lxe: xpath_method         (--TC T1)    0.2828 msec/pass
lxe: xpath_method         (--TC T2)    5.4705 msec/pass
lxe: xpath_method         (--TC T3)    0.0324 msec/pass
lxe: xpath_method         (--TC T4)    0.2804 msec/pass
</pre>
<p>This is well suited for testing and when the XPath expressions are as diverse
as the trees they are called on.  However, if you have a single XPath
expression that you want to apply to a larger number of different elements,
the <tt class="docutils literal">XPath</tt> class is the most efficient way to do it:</p>
<pre class="literal-block">
lxe: xpath_class          (--TC T1)    0.0570 msec/pass
lxe: xpath_class          (--TC T2)    0.6924 msec/pass
lxe: xpath_class          (--TC T3)    0.0148 msec/pass
lxe: xpath_class          (--TC T4)    0.0446 msec/pass
</pre>
<p>Note that this still allows you to use variables in the expression, so you can
parse it once and then adapt it through variables at call time.  In other
cases, where you have a fixed Element or ElementTree and want to run different
expressions on it, you should consider the <tt class="docutils literal">XPathEvaluator</tt>:</p>
<pre class="literal-block">
lxe: xpath_element        (--TR T1)    0.0684 msec/pass
lxe: xpath_element        (--TR T2)    1.0865 msec/pass
lxe: xpath_element        (--TR T3)    0.0174 msec/pass
lxe: xpath_element        (--TR T4)    0.0665 msec/pass
</pre>
<p>While it looks slightly slower, creating an XPath object for each of the
expressions generates a much higher overhead here:</p>
<pre class="literal-block">
lxe: xpath_class_repeat           (--TC T1   )    0.2813 msec/pass
lxe: xpath_class_repeat           (--TC T2   )    5.4042 msec/pass
lxe: xpath_class_repeat           (--TC T3   )    0.0339 msec/pass
lxe: xpath_class_repeat           (--TC T4   )    0.2706 msec/pass
</pre>
<p>Note that tree iteration can be substantially faster than XPath if
your code short-circuits after the first couple of elements were
found.  The XPath engine will always return the complete result set,
regardless of how much of it will actually be used.</p>
<p>Here is an example where only the first matching element is being
searched, a case for which XPath has syntax support as well:</p>
<pre class="literal-block">
lxe: find_single                (--TR T2)    0.0031 msec/pass
cET: find_single                (--TR T2)    0.0026 msec/pass

lxe: iter_single                (--TR T2)    0.0019 msec/pass
cET: iter_single                (--TR T2)    0.0002 msec/pass

lxe: xpath_single               (--TR T2)    0.0861 msec/pass
</pre>
<p>When looking for the first two elements out of many, the numbers
explode for XPath, as restricting the result subset requires a
more complex expression:</p>
<pre class="literal-block">
lxe: iterfind_two               (--TR T2)    0.0050 msec/pass
cET: iterfind_two               (--TR T2)    0.0036 msec/pass

lxe: iter_two                   (--TR T2)    0.0021 msec/pass
cET: iter_two                   (--TR T2)    0.0014 msec/pass

lxe: xpath_two                  (--TR T2)    0.0916 msec/pass
</pre>
</div>
<div class="section" id="a-longer-example">
<h1>A longer example</h1>
<p>... based on lxml 1.3.</p>
<p>A while ago, Uche Ogbuji posted a <a class="reference external" href="http://www.onlamp.com/pub/wlg/6291">benchmark proposal</a> that would
read in a 3MB XML version of the <a class="reference external" href="http://www.ibiblio.org/bosak/xml/eg/religion.2.00.xml.zip">Old Testament</a> of the Bible and
look for the word <em>begat</em> in all verses.  Apparently, it is contained
in 120 out of almost 24000 verses.  This is easy to implement in
ElementTree using <tt class="docutils literal">findall()</tt>.  However, the fastest and most memory
friendly way to do this is obviously <tt class="docutils literal">iterparse()</tt>, as most of the
data is not of any interest.</p>
<p>Now, Uche's original proposal was more or less the following:</p>
<div class="syntax"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">bench_ET</span><span class="p">():</span>
    <span class="n">tree</span> <span class="o">=</span> <span class="n">ElementTree</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s2">"ot.xml"</span><span class="p">)</span>
    <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">tree</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s2">"//v"</span><span class="p">):</span>
        <span class="n">text</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">text</span>
        <span class="k">if</span> <span class="s1">'begat'</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
            <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
    <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
</pre></div>
<p>which takes about one second on my machine today.  The faster <tt class="docutils literal">iterparse()</tt>
variant looks like this:</p>
<div class="syntax"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">bench_ET_iterparse</span><span class="p">():</span>
    <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="k">for</span> <span class="n">event</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">ElementTree</span><span class="o">.</span><span class="n">iterparse</span><span class="p">(</span><span class="s2">"ot.xml"</span><span class="p">):</span>
        <span class="k">if</span> <span class="n">v</span><span class="o">.</span><span class="n">tag</span> <span class="o">==</span> <span class="s1">'v'</span><span class="p">:</span>
            <span class="n">text</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">text</span>
            <span class="k">if</span> <span class="s1">'begat'</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
                <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
        <span class="n">v</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span>
    <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
</pre></div>
<p>The improvement is about 10%.  At the time I first tried (early 2006), lxml
didn't have <tt class="docutils literal">iterparse()</tt> support, but the <tt class="docutils literal">findall()</tt> variant was already
faster than ElementTree.  This changes immediately when you switch to
cElementTree.  The latter only needs 0.17 seconds to do the trick today and
only some impressive 0.10 seconds when running the iterparse version.  And
even back then, it was quite a bit faster than what lxml could achieve.</p>
<p>Since then, lxml has matured a lot and has gotten much faster.  The iterparse
variant now runs in 0.14 seconds, and if you remove the <tt class="docutils literal">v.clear()</tt>, it is
even a little faster (which isn't the case for cElementTree).</p>
<p>One of the many great tools in lxml is XPath, a Swiss army knife for finding
things in XML documents.  It is possible to move the whole thing to a pure
XPath implementation, which looks like this:</p>
<div class="syntax"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">bench_lxml_xpath_all</span><span class="p">():</span>
    <span class="n">tree</span> <span class="o">=</span> <span class="n">etree</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s2">"ot.xml"</span><span class="p">)</span>
    <span class="n">result</span> <span class="o">=</span> <span class="n">tree</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">"//v[contains(., 'begat')]/text()"</span><span class="p">)</span>
    <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
</pre></div>
<p>This runs in about 0.13 seconds and is about the shortest possible
implementation (in lines of Python code) that I could come up with.  Now, this
is already a rather complex XPath expression compared to the simple "//v"
ElementPath expression we started with.  Since this is also valid XPath, let's
try this instead:</p>
<div class="syntax"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">bench_lxml_xpath</span><span class="p">():</span>
    <span class="n">tree</span> <span class="o">=</span> <span class="n">etree</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s2">"ot.xml"</span><span class="p">)</span>
    <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">tree</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">"//v"</span><span class="p">):</span>
        <span class="n">text</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">text</span>
        <span class="k">if</span> <span class="s1">'begat'</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
            <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
    <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
</pre></div>
<p>This gets us down to 0.12 seconds, thus showing that a generic XPath
evaluation engine cannot always compete with a simpler, tailored solution.
However, since this is not much different from the original findall variant,
we can remove the complexity of the XPath call completely and just go with
what we had in the beginning.  Under lxml, this runs in the same 0.12 seconds.</p>
<p>But there is one thing left to try.  We can replace the simple ElementPath
expression with a native tree iterator:</p>
<div class="syntax"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">bench_lxml_getiterator</span><span class="p">():</span>
    <span class="n">tree</span> <span class="o">=</span> <span class="n">etree</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s2">"ot.xml"</span><span class="p">)</span>
    <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">tree</span><span class="o">.</span><span class="n">getiterator</span><span class="p">(</span><span class="s2">"v"</span><span class="p">):</span>
        <span class="n">text</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">text</span>
        <span class="k">if</span> <span class="s1">'begat'</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
            <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
    <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
</pre></div>
<p>This implements the same thing, just without the overhead of parsing and
evaluating a path expression.  And this makes it another bit faster, down to
0.11 seconds.  For comparison, cElementTree runs this version in 0.17 seconds.</p>
<p>So, what have we learned?</p>
<ul class="simple">
<li>Python code is not slow.  The pure XPath solution was not even as fast as
the first shot Python implementation.  In general, a few more lines in
Python make things more readable, which is much more important than the last
5% of performance.</li>
<li>It's important to know the available options - and it's worth starting with
the most simple one.  In this case, a programmer would then probably have
started with <tt class="docutils literal"><span class="pre">getiterator("v")</span></tt> or <tt class="docutils literal">iterparse()</tt>.  Either of them would
already have been the most efficient, depending on which library is used.</li>
<li>It's important to know your tool.  lxml and cElementTree are both very fast
libraries, but they do not have the same performance characteristics.  The
fastest solution in one library can be comparatively slow in the other.  If
you optimise, optimise for the specific target platform.</li>
<li>It's not always worth optimising.  After all that hassle we got from 0.12
seconds for the initial implementation to 0.11 seconds.  Switching over to
cElementTree and writing an <tt class="docutils literal">iterparse()</tt> based version would have given
us 0.10 seconds - not a big difference for 3MB of XML.</li>
<li>Take care what operation is really dominating in your use case.  If we split
up the operations, we can see that lxml is slightly slower than cElementTree
on <tt class="docutils literal">parse()</tt> (both about 0.06 seconds), but more visibly slower on
<tt class="docutils literal">iterparse()</tt>: 0.07 versus 0.10 seconds.  However, tree iteration in lxml
is incredibly fast, so it can be better to parse the whole tree and then
iterate over it rather than using <tt class="docutils literal">iterparse()</tt> to do both in one step.
Or, you can just wait for the lxml developers to optimise iterparse in one
of the next releases...</li>
</ul>
</div>
<div class="section" id="lxml-objectify">
<h1>lxml.objectify</h1>
<p>The following timings are based on the benchmark script <a class="reference external" href="https://github.com/lxml/lxml/blob/master/benchmark/bench_objectify.py">bench_objectify.py</a>.</p>
<p>Objectify is a data-binding API for XML based on lxml.etree, that was added in
version 1.1.  It uses standard Python attribute access to traverse the XML
tree.  It also features ObjectPath, a fast path language based on the same
meme.</p>
<p>Just like lxml.etree, lxml.objectify creates Python representations of
elements on the fly.  To save memory, the normal Python garbage collection
mechanisms will discard them when their last reference is gone.  In cases
where deeply nested elements are frequently accessed through the objectify
API, the create-discard cycles can become a bottleneck, as elements have to be
instantiated over and over again.</p>
<div class="section" id="objectpath">
<h2>ObjectPath</h2>
<p>ObjectPath can be used to speed up the access to elements that are deep in the
tree.  It avoids step-by-step Python element instantiations along the path,
which can substantially improve the access time:</p>
<pre class="literal-block">
lxe: attribute                  (--TR T1)    2.4018 msec/pass
lxe: attribute                  (--TR T2)   16.3755 msec/pass
lxe: attribute                  (--TR T4)    2.3725 msec/pass

lxe: objectpath                 (--TR T1)    1.1816 msec/pass
lxe: objectpath                 (--TR T2)   14.4675 msec/pass
lxe: objectpath                 (--TR T4)    1.2276 msec/pass

lxe: attributes_deep            (--TR T1)    3.7086 msec/pass
lxe: attributes_deep            (--TR T2)   17.5436 msec/pass
lxe: attributes_deep            (--TR T4)    3.8407 msec/pass

lxe: objectpath_deep            (--TR T1)    1.4980 msec/pass
lxe: objectpath_deep            (--TR T2)   14.7266 msec/pass
lxe: objectpath_deep            (--TR T4)    1.4834 msec/pass
</pre>
<p>Note, however, that parsing ObjectPath expressions is not for free either, so
this is most effective for frequently accessing the same element.</p>
</div>
<div class="section" id="caching-elements">
<h2>Caching Elements</h2>
<p>A way to improve the normal attribute access time is static instantiation of
the Python objects, thus trading memory for speed.  Just create a cache
dictionary and run:</p>
<div class="syntax"><pre><span></span><span class="n">cache</span><span class="p">[</span><span class="n">root</span><span class="p">]</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">root</span><span class="o">.</span><span class="n">iter</span><span class="p">())</span>
</pre></div>
<p>after parsing and:</p>
<div class="syntax"><pre><span></span><span class="k">del</span> <span class="n">cache</span><span class="p">[</span><span class="n">root</span><span class="p">]</span>
</pre></div>
<p>when you are done with the tree.  This will keep the Python element
representations of all elements alive and thus avoid the overhead of repeated
Python object creation.  You can also consider using filters or generator
expressions to be more selective.  By choosing the right trees (or even
subtrees and elements) to cache, you can trade memory usage against access
speed:</p>
<pre class="literal-block">
lxe: attribute_cached           (--TR T1)    1.9207 msec/pass
lxe: attribute_cached           (--TR T2)   15.6903 msec/pass
lxe: attribute_cached           (--TR T4)    1.8718 msec/pass

lxe: attributes_deep_cached     (--TR T1)    2.6512 msec/pass
lxe: attributes_deep_cached     (--TR T2)   16.7937 msec/pass
lxe: attributes_deep_cached     (--TR T4)    2.5539 msec/pass

lxe: objectpath_deep_cached     (--TR T1)    0.8519 msec/pass
lxe: objectpath_deep_cached     (--TR T2)   13.9337 msec/pass
lxe: objectpath_deep_cached     (--TR T4)    0.8645 msec/pass
</pre>
<p>Things to note: you cannot currently use <tt class="docutils literal">weakref.WeakKeyDictionary</tt> objects
for this as lxml's element objects do not support weak references (which are
costly in terms of memory).  Also note that new element objects that you add
to these trees will not turn up in the cache automatically and will therefore
still be garbage collected when all their Python references are gone, so this
is most effective for largely immutable trees.  You should consider using a
set instead of a list in this case and add new elements by hand.</p>
</div>
<div class="section" id="further-optimisations">
<h2>Further optimisations</h2>
<p>Here are some more things to try if optimisation is required:</p>
<ul class="simple">
<li>A lot of time is usually spent in tree traversal to find the addressed
elements in the tree.  If you often work in subtrees, do what you would also
do with deep Python objects: assign the parent of the subtree to a variable
or pass it into functions instead of starting at the root.  This allows
accessing its descendants more directly.</li>
<li>Try assigning data values directly to attributes instead of passing them
through DataElement.</li>
<li>If you use custom data types that are costly to parse, try running
<tt class="docutils literal">objectify.annotate()</tt> over read-only trees to speed up the attribute type
inference on read access.</li>
</ul>
<p>Note that none of these measures is guaranteed to speed up your application.
As usual, you should prefer readable code over premature optimisations and
profile your expected use cases before bothering to apply optimisations at
random.</p>
</div>
</div>
</div>
<div class="footer">
<hr class="footer" />
Generated on: 2025-09-21.

</div>
</body>
</html>