File: Tutorial.html

package info (click to toggle)
python-biopython 1.64%2Bdfsg-5
links: PTS, VCS
area: main
in suites: jessie, jessie-kfreebsd
size: 44,416 kB
ctags: 12,472
sloc: python: 153,759; xml: 67,286; ansic: 9,003; sql: 1,488; makefile: 144; sh: 59
file content (11938 lines) | stat: -rw-r--r-- 1,018,828 bytes
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
<meta name="generator" content="hevea 2.09">
<style type="text/css">
.li-itemize{margin:1ex 0ex;}
.li-enumerate{margin:1ex 0ex;}
.dd-description{margin:0ex 0ex 1ex 4ex;}
.dt-description{margin:0ex;}
.toc{list-style:none;}
.footnotetext{margin:0ex; padding:0ex;}
div.footnotetext P{margin:0px; text-indent:1em;}
.thefootnotes{text-align:left;margin:0ex;}
.dt-thefootnotes{margin:0em;}
.dd-thefootnotes{margin:0em 0em 0em 2em;}
.footnoterule{margin:1em auto 1em 0px;width:50%;}
.caption{padding-left:2ex; padding-right:2ex; margin-left:auto; margin-right:auto}
.title{margin:2ex auto;text-align:center}
.titlemain{margin:1ex 2ex 2ex 1ex;}
.titlerest{margin:0ex 2ex;}
.center{text-align:center;margin-left:auto;margin-right:auto;}
.flushleft{text-align:left;margin-left:0ex;margin-right:auto;}
.flushright{text-align:right;margin-left:auto;margin-right:0ex;}
div table{margin-left:inherit;margin-right:inherit;margin-bottom:2px;margin-top:2px}
td table{margin:auto;}
table{border-collapse:collapse;}
td{padding:0;}
.cellpadding0 tr td{padding:0;}
.cellpadding1 tr td{padding:1px;}
pre{text-align:left;margin-left:0ex;margin-right:auto;}
blockquote{margin-left:4ex;margin-right:4ex;text-align:left;}
td p{margin:0px;}
.boxed{border:1px solid black}
.textboxed{border:1px solid black}
.vbar{border:none;width:2px;background-color:black;}
.hbar{border:none;height:2px;width:100%;background-color:black;}
.hfill{border:none;height:1px;width:200%;background-color:black;}
.vdisplay{border-collapse:separate;border-spacing:2px;width:auto; empty-cells:show; border:2px solid red;}
.vdcell{white-space:nowrap;padding:0px; border:2px solid green;}
.display{border-collapse:separate;border-spacing:2px;width:auto; border:none;}
.dcell{white-space:nowrap;padding:0px; border:none;}
.dcenter{margin:0ex auto;}
.vdcenter{border:solid #FF8000 2px; margin:0ex auto;}
.minipage{text-align:left; margin-left:0em; margin-right:auto;}
.marginpar{border:solid thin black; width:20%; text-align:left;}
.marginparleft{float:left; margin-left:0ex; margin-right:1ex;}
.marginparright{float:right; margin-left:1ex; margin-right:0ex;}
.theorem{text-align:left;margin:1ex auto 1ex 0ex;}
.part{margin:2ex auto;text-align:center}
</style>
<title>Biopython Tutorial and Cookbook
</title>
</head>
<body >
<!--HEVEA command line is: /usr/bin/hevea -fix Tutorial.tex -->
<!--CUT STYLE book--><!--CUT DEF chapter 1 --><p>
<P ALIGN="center">
<IMG ALIGN="center" SRC="images/biopython.jpg" TITLE="Biopython Logo" ALT="[Biopython Logo]" width="1024" height="288" />
</p>
</p><table class="title"><tr><td style="padding:1ex"><h1 class="titlemain">Biopython Tutorial and Cookbook</h1><h3 class="titlerest">Jeff Chang, Brad Chapman, Iddo Friedberg, Thomas Hamelryck, <br>
Michiel de Hoon, Peter Cock, Tiago Antao, Eric Talevich, Bartek Wilczy&#X144;ski</h3><h3 class="titlerest">Last Update &#X2013; 29 May 2014 (Biopython 1.64)</h3></td></tr>
</table><!--TOC chapter id="sec1" Contents-->
<h1 id="sec1" class="chapter">Contents</h1><!--SEC END --><ul class="toc"><li class="li-toc">
<a href="#sec2">Chapter&#XA0;1&#XA0;&#XA0;Introduction</a>
<ul class="toc"><li class="li-toc">
<a href="#sec3">1.1&#XA0;&#XA0;What is Biopython?</a>
</li><li class="li-toc"><a href="#sec4">1.2&#XA0;&#XA0;What can I find in the Biopython package</a>
</li><li class="li-toc"><a href="#sec5">1.3&#XA0;&#XA0;Installing Biopython</a>
</li><li class="li-toc"><a href="#sec6">1.4&#XA0;&#XA0;Frequently Asked Questions (FAQ)</a>
</li></ul>
</li><li class="li-toc"><a href="#sec7">Chapter&#XA0;2&#XA0;&#XA0;Quick Start &#X2013; What can you do with Biopython?</a>
<ul class="toc"><li class="li-toc">
<a href="#sec8">2.1&#XA0;&#XA0;General overview of what Biopython provides</a>
</li><li class="li-toc"><a href="#sec9">2.2&#XA0;&#XA0;Working with sequences</a>
</li><li class="li-toc"><a href="#sec10">2.3&#XA0;&#XA0;A usage example</a>
</li><li class="li-toc"><a href="#sec11">2.4&#XA0;&#XA0;Parsing sequence file formats</a>
<ul class="toc"><li class="li-toc">
<a href="#sec12">2.4.1&#XA0;&#XA0;Simple FASTA parsing example</a>
</li><li class="li-toc"><a href="#sec13">2.4.2&#XA0;&#XA0;Simple GenBank parsing example</a>
</li><li class="li-toc"><a href="#sec14">2.4.3&#XA0;&#XA0;I love parsing &#X2013; please don&#X2019;t stop talking about it!</a>
</li></ul>
</li><li class="li-toc"><a href="#sec15">2.5&#XA0;&#XA0;Connecting with biological databases</a>
</li><li class="li-toc"><a href="#sec16">2.6&#XA0;&#XA0;What to do next</a>
</li></ul>
</li><li class="li-toc"><a href="#sec17">Chapter&#XA0;3&#XA0;&#XA0;Sequence objects</a>
<ul class="toc"><li class="li-toc">
<a href="#sec18">3.1&#XA0;&#XA0;Sequences and Alphabets</a>
</li><li class="li-toc"><a href="#sec19">3.2&#XA0;&#XA0;Sequences act like strings</a>
</li><li class="li-toc"><a href="#sec20">3.3&#XA0;&#XA0;Slicing a sequence</a>
</li><li class="li-toc"><a href="#sec21">3.4&#XA0;&#XA0;Turning Seq objects into strings</a>
</li><li class="li-toc"><a href="#sec22">3.5&#XA0;&#XA0;Concatenating or adding sequences</a>
</li><li class="li-toc"><a href="#sec23">3.6&#XA0;&#XA0;Changing case</a>
</li><li class="li-toc"><a href="#sec24">3.7&#XA0;&#XA0;Nucleotide sequences and (reverse) complements</a>
</li><li class="li-toc"><a href="#sec25">3.8&#XA0;&#XA0;Transcription</a>
</li><li class="li-toc"><a href="#sec26">3.9&#XA0;&#XA0;Translation</a>
</li><li class="li-toc"><a href="#sec27">3.10&#XA0;&#XA0;Translation Tables</a>
</li><li class="li-toc"><a href="#sec28">3.11&#XA0;&#XA0;Comparing Seq objects</a>
</li><li class="li-toc"><a href="#sec29">3.12&#XA0;&#XA0;MutableSeq objects</a>
</li><li class="li-toc"><a href="#sec30">3.13&#XA0;&#XA0;UnknownSeq objects</a>
</li><li class="li-toc"><a href="#sec31">3.14&#XA0;&#XA0;Working with strings directly</a>
</li></ul>
</li><li class="li-toc"><a href="#sec32">Chapter&#XA0;4&#XA0;&#XA0;Sequence annotation objects</a>
<ul class="toc"><li class="li-toc">
<a href="#sec33">4.1&#XA0;&#XA0;The SeqRecord object</a>
</li><li class="li-toc"><a href="#sec34">4.2&#XA0;&#XA0;Creating a SeqRecord</a>
<ul class="toc"><li class="li-toc">
<a href="#sec35">4.2.1&#XA0;&#XA0;SeqRecord objects from scratch</a>
</li><li class="li-toc"><a href="#sec36">4.2.2&#XA0;&#XA0;SeqRecord objects from FASTA files</a>
</li><li class="li-toc"><a href="#sec37">4.2.3&#XA0;&#XA0;SeqRecord objects from GenBank files</a>
</li></ul>
</li><li class="li-toc"><a href="#sec38">4.3&#XA0;&#XA0;Feature, location and position objects</a>
<ul class="toc"><li class="li-toc">
<a href="#sec39">4.3.1&#XA0;&#XA0;SeqFeature objects</a>
</li><li class="li-toc"><a href="#sec40">4.3.2&#XA0;&#XA0;Positions and locations</a>
</li><li class="li-toc"><a href="#sec45">4.3.3&#XA0;&#XA0;Sequence described by a feature or location</a>
</li></ul>
</li><li class="li-toc"><a href="#sec46">4.4&#XA0;&#XA0;References</a>
</li><li class="li-toc"><a href="#sec47">4.5&#XA0;&#XA0;The format method</a>
</li><li class="li-toc"><a href="#sec48">4.6&#XA0;&#XA0;Slicing a SeqRecord</a>
</li><li class="li-toc"><a href="#sec49">4.7&#XA0;&#XA0;Adding SeqRecord objects</a>
</li><li class="li-toc"><a href="#sec50">4.8&#XA0;&#XA0;Reverse-complementing SeqRecord objects</a>
</li></ul>
</li><li class="li-toc"><a href="#sec51">Chapter&#XA0;5&#XA0;&#XA0;Sequence Input/Output</a>
<ul class="toc"><li class="li-toc">
<a href="#sec52">5.1&#XA0;&#XA0;Parsing or Reading Sequences</a>
<ul class="toc"><li class="li-toc">
<a href="#sec53">5.1.1&#XA0;&#XA0;Reading Sequence Files</a>
</li><li class="li-toc"><a href="#sec54">5.1.2&#XA0;&#XA0;Iterating over the records in a sequence file</a>
</li><li class="li-toc"><a href="#sec55">5.1.3&#XA0;&#XA0;Getting a list of the records in a sequence file</a>
</li><li class="li-toc"><a href="#sec56">5.1.4&#XA0;&#XA0;Extracting data</a>
</li></ul>
</li><li class="li-toc"><a href="#sec57">5.2&#XA0;&#XA0;Parsing sequences from compressed files</a>
</li><li class="li-toc"><a href="#sec58">5.3&#XA0;&#XA0;Parsing sequences from the net</a>
<ul class="toc"><li class="li-toc">
<a href="#sec59">5.3.1&#XA0;&#XA0;Parsing GenBank records from the net</a>
</li><li class="li-toc"><a href="#sec60">5.3.2&#XA0;&#XA0;Parsing SwissProt sequences from the net</a>
</li></ul>
</li><li class="li-toc"><a href="#sec61">5.4&#XA0;&#XA0;Sequence files as Dictionaries</a>
<ul class="toc"><li class="li-toc">
<a href="#sec62">5.4.1&#XA0;&#XA0;Sequence files as Dictionaries &#X2013; In memory</a>
</li><li class="li-toc"><a href="#sec65">5.4.2&#XA0;&#XA0;Sequence files as Dictionaries &#X2013; Indexed files</a>
</li><li class="li-toc"><a href="#sec68">5.4.3&#XA0;&#XA0;Sequence files as Dictionaries &#X2013; Database indexed files</a>
</li><li class="li-toc"><a href="#sec70">5.4.4&#XA0;&#XA0;Indexing compressed files</a>
</li><li class="li-toc"><a href="#sec71">5.4.5&#XA0;&#XA0;Discussion</a>
</li></ul>
</li><li class="li-toc"><a href="#sec72">5.5&#XA0;&#XA0;Writing Sequence Files</a>
<ul class="toc"><li class="li-toc">
<a href="#sec73">5.5.1&#XA0;&#XA0;Round trips</a>
</li><li class="li-toc"><a href="#sec74">5.5.2&#XA0;&#XA0;Converting between sequence file formats</a>
</li><li class="li-toc"><a href="#sec75">5.5.3&#XA0;&#XA0;Converting a file of sequences to their reverse complements</a>
</li><li class="li-toc"><a href="#sec76">5.5.4&#XA0;&#XA0;Getting your SeqRecord objects as formatted strings</a>
</li></ul>
</li></ul>
</li><li class="li-toc"><a href="#sec77">Chapter&#XA0;6&#XA0;&#XA0;Multiple Sequence Alignment objects</a>
<ul class="toc"><li class="li-toc">
<a href="#sec78">6.1&#XA0;&#XA0;Parsing or Reading Sequence Alignments</a>
<ul class="toc"><li class="li-toc">
<a href="#sec79">6.1.1&#XA0;&#XA0;Single Alignments</a>
</li><li class="li-toc"><a href="#sec80">6.1.2&#XA0;&#XA0;Multiple Alignments</a>
</li><li class="li-toc"><a href="#sec81">6.1.3&#XA0;&#XA0;Ambiguous Alignments</a>
</li></ul>
</li><li class="li-toc"><a href="#sec82">6.2&#XA0;&#XA0;Writing Alignments</a>
<ul class="toc"><li class="li-toc">
<a href="#sec83">6.2.1&#XA0;&#XA0;Converting between sequence alignment file formats</a>
</li><li class="li-toc"><a href="#sec84">6.2.2&#XA0;&#XA0;Getting your alignment objects as formatted strings</a>
</li></ul>
</li><li class="li-toc"><a href="#sec85">6.3&#XA0;&#XA0;Manipulating Alignments</a>
<ul class="toc"><li class="li-toc">
<a href="#sec86">6.3.1&#XA0;&#XA0;Slicing alignments</a>
</li><li class="li-toc"><a href="#sec87">6.3.2&#XA0;&#XA0;Alignments as arrays</a>
</li></ul>
</li><li class="li-toc"><a href="#sec88">6.4&#XA0;&#XA0;Alignment Tools</a>
<ul class="toc"><li class="li-toc">
<a href="#sec89">6.4.1&#XA0;&#XA0;ClustalW</a>
</li><li class="li-toc"><a href="#sec90">6.4.2&#XA0;&#XA0;MUSCLE</a>
</li><li class="li-toc"><a href="#sec91">6.4.3&#XA0;&#XA0;MUSCLE using stdout</a>
</li><li class="li-toc"><a href="#sec92">6.4.4&#XA0;&#XA0;MUSCLE using stdin and stdout</a>
</li><li class="li-toc"><a href="#sec93">6.4.5&#XA0;&#XA0;EMBOSS needle and water</a>
</li></ul>
</li></ul>
</li><li class="li-toc"><a href="#sec94">Chapter&#XA0;7&#XA0;&#XA0;BLAST</a>
<ul class="toc"><li class="li-toc">
<a href="#sec95">7.1&#XA0;&#XA0;Running BLAST over the Internet</a>
</li><li class="li-toc"><a href="#sec96">7.2&#XA0;&#XA0;Running BLAST locally</a>
<ul class="toc"><li class="li-toc">
<a href="#sec97">7.2.1&#XA0;&#XA0;Introduction</a>
</li><li class="li-toc"><a href="#sec98">7.2.2&#XA0;&#XA0;Standalone NCBI BLAST+</a>
</li><li class="li-toc"><a href="#sec99">7.2.3&#XA0;&#XA0;Other versions of BLAST</a>
</li></ul>
</li><li class="li-toc"><a href="#sec100">7.3&#XA0;&#XA0;Parsing BLAST output</a>
</li><li class="li-toc"><a href="#sec101">7.4&#XA0;&#XA0;The BLAST record class</a>
</li><li class="li-toc"><a href="#sec102">7.5&#XA0;&#XA0;Deprecated BLAST parsers</a>
<ul class="toc"><li class="li-toc">
<a href="#sec103">7.5.1&#XA0;&#XA0;Parsing plain-text BLAST output</a>
</li><li class="li-toc"><a href="#sec104">7.5.2&#XA0;&#XA0;Parsing a plain-text BLAST file full of BLAST runs</a>
</li><li class="li-toc"><a href="#sec105">7.5.3&#XA0;&#XA0;Finding a bad record somewhere in a huge plain-text BLAST file</a>
</li></ul>
</li><li class="li-toc"><a href="#sec106">7.6&#XA0;&#XA0;Dealing with PSI-BLAST</a>
</li><li class="li-toc"><a href="#sec107">7.7&#XA0;&#XA0;Dealing with RPS-BLAST</a>
</li></ul>
</li><li class="li-toc"><a href="#sec108">Chapter&#XA0;8&#XA0;&#XA0;BLAST and other sequence search tools (<span style="font-style:italic">experimental code</span>)</a>
<ul class="toc"><li class="li-toc">
<a href="#sec109">8.1&#XA0;&#XA0;The SearchIO object model</a>
<ul class="toc"><li class="li-toc">
<a href="#sec110">8.1.1&#XA0;&#XA0;QueryResult</a>
</li><li class="li-toc"><a href="#sec111">8.1.2&#XA0;&#XA0;Hit</a>
</li><li class="li-toc"><a href="#sec112">8.1.3&#XA0;&#XA0;HSP</a>
</li><li class="li-toc"><a href="#sec113">8.1.4&#XA0;&#XA0;HSPFragment</a>
</li></ul>
</li><li class="li-toc"><a href="#sec114">8.2&#XA0;&#XA0;A note about standards and conventions</a>
</li><li class="li-toc"><a href="#sec115">8.3&#XA0;&#XA0;Reading search output files</a>
</li><li class="li-toc"><a href="#sec116">8.4&#XA0;&#XA0;Dealing with large search output files with indexing</a>
</li><li class="li-toc"><a href="#sec117">8.5&#XA0;&#XA0;Writing and converting search output files</a>
</li></ul>
</li><li class="li-toc"><a href="#sec118">Chapter&#XA0;9&#XA0;&#XA0;Accessing NCBI&#X2019;s Entrez databases</a>
<ul class="toc"><li class="li-toc">
<a href="#sec119">9.1&#XA0;&#XA0;Entrez Guidelines</a>
</li><li class="li-toc"><a href="#sec120">9.2&#XA0;&#XA0;EInfo: Obtaining information about the Entrez databases</a>
</li><li class="li-toc"><a href="#sec121">9.3&#XA0;&#XA0;ESearch: Searching the Entrez databases</a>
</li><li class="li-toc"><a href="#sec122">9.4&#XA0;&#XA0;EPost: Uploading a list of identifiers</a>
</li><li class="li-toc"><a href="#sec123">9.5&#XA0;&#XA0;ESummary: Retrieving summaries from primary IDs</a>
</li><li class="li-toc"><a href="#sec124">9.6&#XA0;&#XA0;EFetch: Downloading full records from Entrez</a>
</li><li class="li-toc"><a href="#sec125">9.7&#XA0;&#XA0;ELink: Searching for related items in NCBI Entrez</a>
</li><li class="li-toc"><a href="#sec126">9.8&#XA0;&#XA0;EGQuery: Global Query - counts for search terms</a>
</li><li class="li-toc"><a href="#sec127">9.9&#XA0;&#XA0;ESpell: Obtaining spelling suggestions</a>
</li><li class="li-toc"><a href="#sec128">9.10&#XA0;&#XA0;Parsing huge Entrez XML files</a>
</li><li class="li-toc"><a href="#sec129">9.11&#XA0;&#XA0;Handling errors</a>
</li><li class="li-toc"><a href="#sec130">9.12&#XA0;&#XA0;Specialized parsers</a>
<ul class="toc"><li class="li-toc">
<a href="#sec131">9.12.1&#XA0;&#XA0;Parsing Medline records</a>
</li><li class="li-toc"><a href="#sec132">9.12.2&#XA0;&#XA0;Parsing GEO records</a>
</li><li class="li-toc"><a href="#sec133">9.12.3&#XA0;&#XA0;Parsing UniGene records</a>
</li></ul>
</li><li class="li-toc"><a href="#sec134">9.13&#XA0;&#XA0;Using a proxy</a>
</li><li class="li-toc"><a href="#sec135">9.14&#XA0;&#XA0;Examples</a>
<ul class="toc"><li class="li-toc">
<a href="#sec136">9.14.1&#XA0;&#XA0;PubMed and Medline</a>
</li><li class="li-toc"><a href="#sec137">9.14.2&#XA0;&#XA0;Searching, downloading, and parsing Entrez Nucleotide records</a>
</li><li class="li-toc"><a href="#sec138">9.14.3&#XA0;&#XA0;Searching, downloading, and parsing GenBank records</a>
</li><li class="li-toc"><a href="#sec139">9.14.4&#XA0;&#XA0;Finding the lineage of an organism</a>
</li></ul>
</li><li class="li-toc"><a href="#sec140">9.15&#XA0;&#XA0;Using the history and WebEnv</a>
<ul class="toc"><li class="li-toc">
<a href="#sec141">9.15.1&#XA0;&#XA0;Searching for and downloading sequences using the history</a>
</li><li class="li-toc"><a href="#sec142">9.15.2&#XA0;&#XA0;Searching for and downloading abstracts using the history</a>
</li><li class="li-toc"><a href="#sec143">9.15.3&#XA0;&#XA0;Searching for citations</a>
</li></ul>
</li></ul>
</li><li class="li-toc"><a href="#sec144">Chapter&#XA0;10&#XA0;&#XA0;Swiss-Prot and ExPASy</a>
<ul class="toc"><li class="li-toc">
<a href="#sec145">10.1&#XA0;&#XA0;Parsing Swiss-Prot files</a>
<ul class="toc"><li class="li-toc">
<a href="#sec146">10.1.1&#XA0;&#XA0;Parsing Swiss-Prot records</a>
</li><li class="li-toc"><a href="#sec147">10.1.2&#XA0;&#XA0;Parsing the Swiss-Prot keyword and category list</a>
</li></ul>
</li><li class="li-toc"><a href="#sec148">10.2&#XA0;&#XA0;Parsing Prosite records</a>
</li><li class="li-toc"><a href="#sec149">10.3&#XA0;&#XA0;Parsing Prosite documentation records</a>
</li><li class="li-toc"><a href="#sec150">10.4&#XA0;&#XA0;Parsing Enzyme records</a>
</li><li class="li-toc"><a href="#sec151">10.5&#XA0;&#XA0;Accessing the ExPASy server</a>
<ul class="toc"><li class="li-toc">
<a href="#sec152">10.5.1&#XA0;&#XA0;Retrieving a Swiss-Prot record</a>
</li><li class="li-toc"><a href="#sec153">10.5.2&#XA0;&#XA0;Searching Swiss-Prot</a>
</li><li class="li-toc"><a href="#sec154">10.5.3&#XA0;&#XA0;Retrieving Prosite and Prosite documentation records</a>
</li></ul>
</li><li class="li-toc"><a href="#sec155">10.6&#XA0;&#XA0;Scanning the Prosite database</a>
</li></ul>
</li><li class="li-toc"><a href="#sec156">Chapter&#XA0;11&#XA0;&#XA0;Going 3D: The PDB module</a>
<ul class="toc"><li class="li-toc">
<a href="#sec157">11.1&#XA0;&#XA0;Reading and writing crystal structure files</a>
<ul class="toc"><li class="li-toc">
<a href="#sec158">11.1.1&#XA0;&#XA0;Reading a PDB file</a>
</li><li class="li-toc"><a href="#sec159">11.1.2&#XA0;&#XA0;Reading an mmCIF file</a>
</li><li class="li-toc"><a href="#sec160">11.1.3&#XA0;&#XA0;Reading files in the PDB XML format</a>
</li><li class="li-toc"><a href="#sec161">11.1.4&#XA0;&#XA0;Writing PDB files</a>
</li></ul>
</li><li class="li-toc"><a href="#sec162">11.2&#XA0;&#XA0;Structure representation</a>
<ul class="toc"><li class="li-toc">
<a href="#sec163">11.2.1&#XA0;&#XA0;Structure</a>
</li><li class="li-toc"><a href="#sec164">11.2.2&#XA0;&#XA0;Model</a>
</li><li class="li-toc"><a href="#sec165">11.2.3&#XA0;&#XA0;Chain</a>
</li><li class="li-toc"><a href="#sec166">11.2.4&#XA0;&#XA0;Residue</a>
</li><li class="li-toc"><a href="#sec167">11.2.5&#XA0;&#XA0;Atom</a>
</li><li class="li-toc"><a href="#sec168">11.2.6&#XA0;&#XA0;Extracting a specific <span style="font-family:monospace">Atom/Residue/Chain/Model</span>
from a Structure</a>
</li></ul>
</li><li class="li-toc"><a href="#sec169">11.3&#XA0;&#XA0;Disorder</a>
<ul class="toc"><li class="li-toc">
<a href="#disorder%20problems">11.3.1&#XA0;&#XA0;General approach</a>
</li><li class="li-toc"><a href="#disordered%20atoms">11.3.2&#XA0;&#XA0;Disordered atoms</a>
</li><li class="li-toc"><a href="#sec172">11.3.3&#XA0;&#XA0;Disordered residues</a>
</li></ul>
</li><li class="li-toc"><a href="#sec175">11.4&#XA0;&#XA0;Hetero residues</a>
<ul class="toc"><li class="li-toc">
<a href="#hetero%20problems">11.4.1&#XA0;&#XA0;Associated problems</a>
</li><li class="li-toc"><a href="#sec177">11.4.2&#XA0;&#XA0;Water residues</a>
</li><li class="li-toc"><a href="#sec178">11.4.3&#XA0;&#XA0;Other hetero residues</a>
</li></ul>
</li><li class="li-toc"><a href="#sec179">11.5&#XA0;&#XA0;Navigating through a Structure object</a>
</li><li class="li-toc"><a href="#sec190">11.6&#XA0;&#XA0;Analyzing structures</a>
<ul class="toc"><li class="li-toc">
<a href="#sec191">11.6.1&#XA0;&#XA0;Measuring distances</a>
</li><li class="li-toc"><a href="#sec192">11.6.2&#XA0;&#XA0;Measuring angles</a>
</li><li class="li-toc"><a href="#sec193">11.6.3&#XA0;&#XA0;Measuring torsion angles</a>
</li><li class="li-toc"><a href="#sec194">11.6.4&#XA0;&#XA0;Determining atom-atom contacts</a>
</li><li class="li-toc"><a href="#sec195">11.6.5&#XA0;&#XA0;Superimposing two structures</a>
</li><li class="li-toc"><a href="#sec196">11.6.6&#XA0;&#XA0;Mapping the residues of two related structures onto each other</a>
</li><li class="li-toc"><a href="#sec197">11.6.7&#XA0;&#XA0;Calculating the Half Sphere Exposure</a>
</li><li class="li-toc"><a href="#sec198">11.6.8&#XA0;&#XA0;Determining the secondary structure</a>
</li><li class="li-toc"><a href="#subsec%3Aresidue_depth">11.6.9&#XA0;&#XA0;Calculating the residue depth</a>
</li></ul>
</li><li class="li-toc"><a href="#sec200">11.7&#XA0;&#XA0;Common problems in PDB files</a>
<ul class="toc"><li class="li-toc">
<a href="#problem%20structures">11.7.1&#XA0;&#XA0;Examples</a>
</li><li class="li-toc"><a href="#sec204">11.7.2&#XA0;&#XA0;Automatic correction</a>
</li><li class="li-toc"><a href="#sec207">11.7.3&#XA0;&#XA0;Fatal errors</a>
</li></ul>
</li><li class="li-toc"><a href="#sec210">11.8&#XA0;&#XA0;Accessing the Protein Data Bank</a>
<ul class="toc"><li class="li-toc">
<a href="#sec211">11.8.1&#XA0;&#XA0;Downloading structures from the Protein Data Bank</a>
</li><li class="li-toc"><a href="#sec212">11.8.2&#XA0;&#XA0;Downloading the entire PDB</a>
</li><li class="li-toc"><a href="#sec213">11.8.3&#XA0;&#XA0;Keeping a local copy of the PDB up to date</a>
</li></ul>
</li><li class="li-toc"><a href="#sec214">11.9&#XA0;&#XA0;General questions</a>
<ul class="toc"><li class="li-toc">
<a href="#sec215">11.9.1&#XA0;&#XA0;How well tested is Bio.PDB?</a>
</li><li class="li-toc"><a href="#sec216">11.9.2&#XA0;&#XA0;How fast is it?</a>
</li><li class="li-toc"><a href="#sec217">11.9.3&#XA0;&#XA0;Is there support for molecular graphics?</a>
</li><li class="li-toc"><a href="#sec218">11.9.4&#XA0;&#XA0;Who&#X2019;s using Bio.PDB?</a>
</li></ul>
</li></ul>
</li><li class="li-toc"><a href="#sec219">Chapter&#XA0;12&#XA0;&#XA0;Bio.PopGen: Population genetics</a>
<ul class="toc"><li class="li-toc">
<a href="#sec220">12.1&#XA0;&#XA0;GenePop</a>
</li><li class="li-toc"><a href="#sec221">12.2&#XA0;&#XA0;Coalescent simulation</a>
<ul class="toc"><li class="li-toc">
<a href="#sec222">12.2.1&#XA0;&#XA0;Creating scenarios</a>
</li><li class="li-toc"><a href="#sec225">12.2.2&#XA0;&#XA0;Running Fastsimcoal2</a>
</li></ul>
</li><li class="li-toc"><a href="#sec226">12.3&#XA0;&#XA0;Other applications</a>
<ul class="toc"><li class="li-toc">
<a href="#sec227">12.3.1&#XA0;&#XA0;FDist: Detecting selection and molecular adaptation</a>
</li></ul>
</li><li class="li-toc"><a href="#sec230">12.4&#XA0;&#XA0;Future Developments</a>
</li></ul>
</li><li class="li-toc"><a href="#sec231">Chapter&#XA0;13&#XA0;&#XA0;Phylogenetics with Bio.Phylo</a>
<ul class="toc"><li class="li-toc">
<a href="#sec232">13.1&#XA0;&#XA0;Demo: What&#X2019;s in a Tree?</a>
<ul class="toc"><li class="li-toc">
<a href="#sec233">13.1.1&#XA0;&#XA0;Coloring branches within a tree</a>
</li></ul>
</li><li class="li-toc"><a href="#sec234">13.2&#XA0;&#XA0;I/O functions</a>
</li><li class="li-toc"><a href="#sec235">13.3&#XA0;&#XA0;View and export trees</a>
</li><li class="li-toc"><a href="#sec236">13.4&#XA0;&#XA0;Using Tree and Clade objects</a>
<ul class="toc"><li class="li-toc">
<a href="#sec237">13.4.1&#XA0;&#XA0;Search and traversal methods</a>
</li><li class="li-toc"><a href="#sec238">13.4.2&#XA0;&#XA0;Information methods</a>
</li><li class="li-toc"><a href="#sec239">13.4.3&#XA0;&#XA0;Modification methods</a>
</li><li class="li-toc"><a href="#sec240">13.4.4&#XA0;&#XA0;Features of PhyloXML trees</a>
</li></ul>
</li><li class="li-toc"><a href="#sec241">13.5&#XA0;&#XA0;Running external applications</a>
</li><li class="li-toc"><a href="#sec242">13.6&#XA0;&#XA0;PAML integration</a>
</li><li class="li-toc"><a href="#sec243">13.7&#XA0;&#XA0;Future plans</a>
</li></ul>
</li><li class="li-toc"><a href="#sec244">Chapter&#XA0;14&#XA0;&#XA0;Sequence motif analysis using Bio.motifs</a>
<ul class="toc"><li class="li-toc">
<a href="#sec245">14.1&#XA0;&#XA0;Motif objects</a>
<ul class="toc"><li class="li-toc">
<a href="#sec246">14.1.1&#XA0;&#XA0;Creating a motif from instances</a>
</li><li class="li-toc"><a href="#sec247">14.1.2&#XA0;&#XA0;Creating a sequence logo</a>
</li></ul>
</li><li class="li-toc"><a href="#sec248">14.2&#XA0;&#XA0;Reading motifs</a>
<ul class="toc"><li class="li-toc">
<a href="#sec249">14.2.1&#XA0;&#XA0;JASPAR</a>
</li><li class="li-toc"><a href="#sec255">14.2.2&#XA0;&#XA0;MEME</a>
</li><li class="li-toc"><a href="#sec257">14.2.3&#XA0;&#XA0;TRANSFAC</a>
</li></ul>
</li><li class="li-toc"><a href="#sec258">14.3&#XA0;&#XA0;Writing motifs</a>
</li><li class="li-toc"><a href="#sec259">14.4&#XA0;&#XA0;Position-Weight Matrices</a>
</li><li class="li-toc"><a href="#sec260">14.5&#XA0;&#XA0;Position-Specific Scoring Matrices</a>
</li><li class="li-toc"><a href="#sec261">14.6&#XA0;&#XA0;Searching for instances</a>
<ul class="toc"><li class="li-toc">
<a href="#sec262">14.6.1&#XA0;&#XA0;Searching for exact matches</a>
</li><li class="li-toc"><a href="#sec263">14.6.2&#XA0;&#XA0;Searching for matches using the PSSM score</a>
</li><li class="li-toc"><a href="#sec264">14.6.3&#XA0;&#XA0;Selecting a score threshold</a>
</li></ul>
</li><li class="li-toc"><a href="#sec265">14.7&#XA0;&#XA0;Each motif object has an associated Position-Specific Scoring Matrix</a>
</li><li class="li-toc"><a href="#sec266">14.8&#XA0;&#XA0;Comparing motifs</a>
</li><li class="li-toc"><a href="#sec267">14.9&#XA0;&#XA0;<em>De novo</em> motif finding</a>
<ul class="toc"><li class="li-toc">
<a href="#sec268">14.9.1&#XA0;&#XA0;MEME</a>
</li><li class="li-toc"><a href="#sec269">14.9.2&#XA0;&#XA0;AlignAce</a>
</li></ul>
</li><li class="li-toc"><a href="#sec270">14.10&#XA0;&#XA0;Useful links </a>
</li></ul>
</li><li class="li-toc"><a href="#sec271">Chapter&#XA0;15&#XA0;&#XA0;Cluster analysis</a>
<ul class="toc"><li class="li-toc">
<a href="#sec275">15.1&#XA0;&#XA0;Distance functions</a>
</li><li class="li-toc"><a href="#sec286">15.2&#XA0;&#XA0;Calculating cluster properties</a>
</li><li class="li-toc"><a href="#sec289">15.3&#XA0;&#XA0;Partitioning algorithms</a>
</li><li class="li-toc"><a href="#sec292">15.4&#XA0;&#XA0;Hierarchical clustering</a>
</li><li class="li-toc"><a href="#sec295">15.5&#XA0;&#XA0;Self-Organizing Maps</a>
</li><li class="li-toc"><a href="#sec296">15.6&#XA0;&#XA0;Principal Component Analysis</a>
</li><li class="li-toc"><a href="#sec297">15.7&#XA0;&#XA0;Handling Cluster/TreeView-type files</a>
</li><li class="li-toc"><a href="#sec305">15.8&#XA0;&#XA0;Example calculation</a>
</li><li class="li-toc"><a href="#sec306">15.9&#XA0;&#XA0;Auxiliary functions</a>
</li></ul>
</li><li class="li-toc"><a href="#sec307">Chapter&#XA0;16&#XA0;&#XA0;Supervised learning methods</a>
<ul class="toc"><li class="li-toc">
<a href="#sec308">16.1&#XA0;&#XA0;The Logistic Regression Model</a>
<ul class="toc"><li class="li-toc">
<a href="#sec309">16.1.1&#XA0;&#XA0;Background and Purpose</a>
</li><li class="li-toc"><a href="#sec310">16.1.2&#XA0;&#XA0;Training the logistic regression model</a>
</li><li class="li-toc"><a href="#sec311">16.1.3&#XA0;&#XA0;Using the logistic regression model for classification</a>
</li><li class="li-toc"><a href="#sec312">16.1.4&#XA0;&#XA0;Logistic Regression, Linear Discriminant Analysis, and Support Vector Machines</a>
</li></ul>
</li><li class="li-toc"><a href="#sec313">16.2&#XA0;&#XA0;<span style="font-style:italic">k</span>-Nearest Neighbors</a>
<ul class="toc"><li class="li-toc">
<a href="#sec314">16.2.1&#XA0;&#XA0;Background and purpose</a>
</li><li class="li-toc"><a href="#sec315">16.2.2&#XA0;&#XA0;Initializing a <span style="font-style:italic">k</span>-nearest neighbors model</a>
</li><li class="li-toc"><a href="#sec316">16.2.3&#XA0;&#XA0;Using a <span style="font-style:italic">k</span>-nearest neighbors model for classification</a>
</li></ul>
</li><li class="li-toc"><a href="#sec317">16.3&#XA0;&#XA0;Na&#XEF;ve Bayes</a>
</li><li class="li-toc"><a href="#sec318">16.4&#XA0;&#XA0;Maximum Entropy</a>
</li><li class="li-toc"><a href="#sec319">16.5&#XA0;&#XA0;Markov Models</a>
</li></ul>
</li><li class="li-toc"><a href="#sec320">Chapter&#XA0;17&#XA0;&#XA0;Graphics including GenomeDiagram</a>
<ul class="toc"><li class="li-toc">
<a href="#sec321">17.1&#XA0;&#XA0;GenomeDiagram</a>
<ul class="toc"><li class="li-toc">
<a href="#sec322">17.1.1&#XA0;&#XA0;Introduction</a>
</li><li class="li-toc"><a href="#sec323">17.1.2&#XA0;&#XA0;Diagrams, tracks, feature-sets and features</a>
</li><li class="li-toc"><a href="#sec324">17.1.3&#XA0;&#XA0;A top down example</a>
</li><li class="li-toc"><a href="#sec325">17.1.4&#XA0;&#XA0;A bottom up example</a>
</li><li class="li-toc"><a href="#sec326">17.1.5&#XA0;&#XA0;Features without a SeqFeature</a>
</li><li class="li-toc"><a href="#sec327">17.1.6&#XA0;&#XA0;Feature captions</a>
</li><li class="li-toc"><a href="#sec328">17.1.7&#XA0;&#XA0;Feature sigils</a>
</li><li class="li-toc"><a href="#sec329">17.1.8&#XA0;&#XA0;Arrow sigils</a>
</li><li class="li-toc"><a href="#sec330">17.1.9&#XA0;&#XA0;A nice example</a>
</li><li class="li-toc"><a href="#sec331">17.1.10&#XA0;&#XA0;Multiple tracks</a>
</li><li class="li-toc"><a href="#sec332">17.1.11&#XA0;&#XA0;Cross-Links between tracks</a>
</li><li class="li-toc"><a href="#sec333">17.1.12&#XA0;&#XA0;Further options</a>
</li><li class="li-toc"><a href="#sec334">17.1.13&#XA0;&#XA0;Converting old code</a>
</li></ul>
</li><li class="li-toc"><a href="#sec335">17.2&#XA0;&#XA0;Chromosomes</a>
<ul class="toc"><li class="li-toc">
<a href="#sec336">17.2.1&#XA0;&#XA0;Simple Chromosomes</a>
</li><li class="li-toc"><a href="#sec337">17.2.2&#XA0;&#XA0;Annotated Chromosomes</a>
</li></ul>
</li></ul>
</li><li class="li-toc"><a href="#sec338">Chapter&#XA0;18&#XA0;&#XA0;Cookbook &#X2013; Cool things to do with it</a>
<ul class="toc"><li class="li-toc">
<a href="#sec339">18.1&#XA0;&#XA0;Working with sequence files</a>
<ul class="toc"><li class="li-toc">
<a href="#sec340">18.1.1&#XA0;&#XA0;Filtering a sequence file</a>
</li><li class="li-toc"><a href="#sec341">18.1.2&#XA0;&#XA0;Producing randomised genomes</a>
</li><li class="li-toc"><a href="#sec342">18.1.3&#XA0;&#XA0;Translating a FASTA file of CDS entries</a>
</li><li class="li-toc"><a href="#sec343">18.1.4&#XA0;&#XA0;Making the sequences in a FASTA file upper case</a>
</li><li class="li-toc"><a href="#sec344">18.1.5&#XA0;&#XA0;Sorting a sequence file</a>
</li><li class="li-toc"><a href="#sec345">18.1.6&#XA0;&#XA0;Simple quality filtering for FASTQ files</a>
</li><li class="li-toc"><a href="#sec346">18.1.7&#XA0;&#XA0;Trimming off primer sequences</a>
</li><li class="li-toc"><a href="#sec347">18.1.8&#XA0;&#XA0;Trimming off adaptor sequences</a>
</li><li class="li-toc"><a href="#sec348">18.1.9&#XA0;&#XA0;Converting FASTQ files</a>
</li><li class="li-toc"><a href="#sec349">18.1.10&#XA0;&#XA0;Converting FASTA and QUAL files into FASTQ files</a>
</li><li class="li-toc"><a href="#sec350">18.1.11&#XA0;&#XA0;Indexing a FASTQ file</a>
</li><li class="li-toc"><a href="#sec351">18.1.12&#XA0;&#XA0;Converting SFF files</a>
</li><li class="li-toc"><a href="#sec352">18.1.13&#XA0;&#XA0;Identifying open reading frames</a>
</li></ul>
</li><li class="li-toc"><a href="#sec353">18.2&#XA0;&#XA0;Sequence parsing plus simple plots</a>
<ul class="toc"><li class="li-toc">
<a href="#sec354">18.2.1&#XA0;&#XA0;Histogram of sequence lengths</a>
</li><li class="li-toc"><a href="#sec355">18.2.2&#XA0;&#XA0;Plot of sequence GC%</a>
</li><li class="li-toc"><a href="#sec356">18.2.3&#XA0;&#XA0;Nucleotide dot plots</a>
</li><li class="li-toc"><a href="#sec357">18.2.4&#XA0;&#XA0;Plotting the quality scores of sequencing read data</a>
</li></ul>
</li><li class="li-toc"><a href="#sec358">18.3&#XA0;&#XA0;Dealing with alignments</a>
<ul class="toc"><li class="li-toc">
<a href="#sec359">18.3.1&#XA0;&#XA0;Calculating summary information</a>
</li><li class="li-toc"><a href="#sec360">18.3.2&#XA0;&#XA0;Calculating a quick consensus sequence</a>
</li><li class="li-toc"><a href="#sec361">18.3.3&#XA0;&#XA0;Position Specific Score Matrices</a>
</li><li class="li-toc"><a href="#sec362">18.3.4&#XA0;&#XA0;Information Content</a>
</li></ul>
</li><li class="li-toc"><a href="#sec363">18.4&#XA0;&#XA0;Substitution Matrices</a>
<ul class="toc"><li class="li-toc">
<a href="#sec364">18.4.1&#XA0;&#XA0;Using common substitution matrices</a>
</li><li class="li-toc"><a href="#sec365">18.4.2&#XA0;&#XA0;Creating your own substitution matrix from an alignment</a>
</li></ul>
</li><li class="li-toc"><a href="#sec366">18.5&#XA0;&#XA0;BioSQL &#X2013; storing sequences in a relational database</a>
</li></ul>
</li><li class="li-toc"><a href="#sec367">Chapter&#XA0;19&#XA0;&#XA0;The Biopython testing framework</a>
<ul class="toc"><li class="li-toc">
<a href="#sec368">19.1&#XA0;&#XA0;Running the tests</a>
</li><li class="li-toc"><a href="#sec369">19.2&#XA0;&#XA0;Writing tests</a>
<ul class="toc"><li class="li-toc">
<a href="#sec370">19.2.1&#XA0;&#XA0;Writing a print-and-compare test</a>
</li><li class="li-toc"><a href="#sec371">19.2.2&#XA0;&#XA0;Writing a unittest-based test</a>
</li></ul>
</li><li class="li-toc"><a href="#sec372">19.3&#XA0;&#XA0;Writing doctests</a>
</li></ul>
</li><li class="li-toc"><a href="#sec373">Chapter&#XA0;20&#XA0;&#XA0;Advanced</a>
<ul class="toc"><li class="li-toc">
<a href="#sec374">20.1&#XA0;&#XA0;Parser Design</a>
</li><li class="li-toc"><a href="#sec375">20.2&#XA0;&#XA0;Substitution Matrices</a>
<ul class="toc"><li class="li-toc">
<a href="#sec376">20.2.1&#XA0;&#XA0;SubsMat</a>
</li><li class="li-toc"><a href="#sec377">20.2.2&#XA0;&#XA0;FreqTable</a>
</li></ul>
</li></ul>
</li><li class="li-toc"><a href="#sec378">Chapter&#XA0;21&#XA0;&#XA0;Where to go from here &#X2013; contributing to Biopython</a>
<ul class="toc"><li class="li-toc">
<a href="#sec379">21.1&#XA0;&#XA0;Bug Reports + Feature Requests</a>
</li><li class="li-toc"><a href="#sec380">21.2&#XA0;&#XA0;Mailing lists and helping newcomers</a>
</li><li class="li-toc"><a href="#sec381">21.3&#XA0;&#XA0;Contributing Documentation</a>
</li><li class="li-toc"><a href="#sec382">21.4&#XA0;&#XA0;Contributing cookbook examples</a>
</li><li class="li-toc"><a href="#sec383">21.5&#XA0;&#XA0;Maintaining a distribution for a platform</a>
</li><li class="li-toc"><a href="#sec384">21.6&#XA0;&#XA0;Contributing Unit Tests</a>
</li><li class="li-toc"><a href="#sec385">21.7&#XA0;&#XA0;Contributing Code</a>
</li></ul>
</li><li class="li-toc"><a href="#sec386">Chapter&#XA0;22&#XA0;&#XA0;Appendix: Useful stuff about Python</a>
<ul class="toc"><li class="li-toc">
<a href="#sec387">22.1&#XA0;&#XA0;What the heck is a handle?</a>
<ul class="toc"><li class="li-toc">
<a href="#sec388">22.1.1&#XA0;&#XA0;Creating a handle from a string</a>
</li></ul>
</li></ul>
</li></ul>
<!--TOC chapter id="sec2" Introduction-->
<h1 id="sec2" class="chapter">Chapter&#XA0;1&#XA0;&#XA0;Introduction</h1><!--SEC END --><p>
<a id="chapter:introduction"></a></p>
<!--TOC section id="sec3" What is Biopython?-->
<h2 id="sec3" class="section">1.1&#XA0;&#XA0;What is Biopython?</h2><!--SEC END --><p>The Biopython Project is an international association of developers of freely available Python (<a href="http://www.python.org"><span style="font-family:monospace">http://www.python.org</span></a>) tools for computational molecular biology. Python is an object oriented, interpreted, flexible language that is becoming increasingly popular for scientific computing. Python is easy to learn, has a very clear syntax and can easily be extended with modules written in C, C++ or FORTRAN.</p><p>The Biopython web site (<a href="http://www.biopython.org"><span style="font-family:monospace">http://www.biopython.org</span></a>) provides
an online resource for modules, scripts, and web links for developers
of Python-based software for bioinformatics use and research. Basically,
the goal of Biopython is to make it as easy as possible to use Python
for bioinformatics by creating high-quality, reusable modules and
classes. Biopython features include parsers for various Bioinformatics
file formats (BLAST, Clustalw, FASTA, Genbank,...), access to online
services (NCBI, Expasy,...), interfaces to common and not-so-common
programs (Clustalw, DSSP, MSMS...), a standard sequence class, various
clustering modules, a KD tree data structure etc. and even documentation. </p><p>Basically, we just like to program in Python and want to make it as easy as possible to use Python for bioinformatics by creating high-quality, reusable modules and scripts.</p>
<!--TOC section id="sec4" What can I find in the Biopython package-->
<h2 id="sec4" class="section">1.2&#XA0;&#XA0;What can I find in the Biopython package</h2><!--SEC END --><p>The main Biopython releases have lots of functionality, including:</p><ul class="itemize"><li class="li-itemize">
The ability to parse bioinformatics files into Python utilizable data structures, including support for the following formats:<ul class="itemize"><li class="li-itemize">
Blast output &#X2013; both from standalone and WWW Blast
</li><li class="li-itemize">Clustalw
</li><li class="li-itemize">FASTA
</li><li class="li-itemize">GenBank
</li><li class="li-itemize">PubMed and Medline
</li><li class="li-itemize">ExPASy files, like Enzyme and Prosite
</li><li class="li-itemize">SCOP, including &#X2018;dom&#X2019; and &#X2018;lin&#X2019; files
</li><li class="li-itemize">UniGene
</li><li class="li-itemize">SwissProt
</li></ul></li><li class="li-itemize">Files in the supported formats can be iterated over record by record or indexed and accessed via a Dictionary interface.</li><li class="li-itemize">Code to deal with popular on-line bioinformatics destinations such as:<ul class="itemize"><li class="li-itemize">
NCBI &#X2013; Blast, Entrez and PubMed services
</li><li class="li-itemize">ExPASy &#X2013; Swiss-Prot and Prosite entries, as well as Prosite searches
</li></ul></li><li class="li-itemize">Interfaces to common bioinformatics programs such as:<ul class="itemize"><li class="li-itemize">
Standalone Blast from NCBI
</li><li class="li-itemize">Clustalw alignment program
</li><li class="li-itemize">EMBOSS command line tools
</li></ul></li><li class="li-itemize">A standard sequence class that deals with sequences, ids on sequences, and sequence features.</li><li class="li-itemize">Tools for performing common operations on sequences, such as translation, transcription and weight calculations.</li><li class="li-itemize">Code to perform classification of data using k Nearest Neighbors, Naive Bayes or Support Vector Machines.</li><li class="li-itemize">Code for dealing with alignments, including a standard way to create and deal with substitution matrices.</li><li class="li-itemize">Code making it easy to split up parallelizable tasks into separate processes.</li><li class="li-itemize">GUI-based programs to do basic sequence manipulations, translations, BLASTing, etc.</li><li class="li-itemize">Extensive documentation and help with using the modules, including this file, on-line wiki documentation, the web site, and the mailing list.</li><li class="li-itemize">Integration with BioSQL, a sequence database schema also supported by the BioPerl and BioJava projects.</li></ul><p>We hope this gives you plenty of reasons to download and start using Biopython!</p>
<!--TOC section id="sec5" Installing Biopython-->
<h2 id="sec5" class="section">1.3&#XA0;&#XA0;Installing Biopython</h2><!--SEC END --><p>All of the installation information for Biopython was separated from
this document to make it easier to keep updated.</p><p>The short version is go to our downloads page (<a href="http://biopython.org/wiki/Download"><span style="font-family:monospace">http://biopython.org/wiki/Download</span></a>),
download and install the listed dependencies, then download and install Biopython.
Biopython runs on many platforms (Windows, Mac, and on the various flavors of Linux and Unix).
For Windows we provide pre-compiled click-and-run installers, while for Unix and other
operating systems you must install from source as described in the included README file.
This is usually as simple as the standard commands:</p><pre class="verbatim">python setup.py build
python setup.py test
sudo python setup.py install
</pre><p>(You can in fact skip the build and test, and go straight to the install &#X2013;
but its better to make sure everything seems to be working.)</p><p>The longer version of our installation instructions covers
installation of Python, Biopython dependencies and Biopython itself.
It is available in PDF
(<a href="http://biopython.org/DIST/docs/install/Installation.pdf"><span style="font-family:monospace">http://biopython.org/DIST/docs/install/Installation.pdf</span></a>)
and HTML formats
(<a href="http://biopython.org/DIST/docs/install/Installation.html"><span style="font-family:monospace">http://biopython.org/DIST/docs/install/Installation.html</span></a>).</p>
<!--TOC section id="sec6" Frequently Asked Questions (FAQ)-->
<h2 id="sec6" class="section">1.4&#XA0;&#XA0;Frequently Asked Questions (FAQ)</h2><!--SEC END --><ol class="enumerate" type=1><li class="li-enumerate"><em>How do I cite Biopython in a scientific publication?</em> <br>
 Please cite our application note [<a href="#cock2009">1</a>, Cock <span style="font-style:italic">et al.</span>, 2009]
as the main Biopython reference.
In addition, please cite any publications from the following list if appropriate, in particular as a reference for specific modules within Biopython (more information can be found on our website):
<ul class="itemize"><li class="li-itemize">
For the official project announcement: [<a href="#chapman2000">13</a>, Chapman and Chang, 2000];
</li><li class="li-itemize">For <code>Bio.PDB</code>: [<a href="#hamelryck2003a">18</a>, Hamelryck and Manderick, 2003];
</li><li class="li-itemize">For <code>Bio.Cluster</code>: [<a href="#dehoon2004">14</a>, De Hoon <span style="font-style:italic">et al.</span>, 2004];
</li><li class="li-itemize">For <code>Bio.Graphics.GenomeDiagram</code>: [<a href="#pritchard2006">2</a>, Pritchard <span style="font-style:italic">et al.</span>, 2006];
</li><li class="li-itemize">For <code>Bio.Phylo</code> and <code>Bio.Phylo.PAML</code>: [<a href="#talevich2012">9</a>, Talevich <span style="font-style:italic">et al.</span>, 2012];
</li><li class="li-itemize">For the FASTQ file format as supported in Biopython, BioPerl, BioRuby, BioJava, and EMBOSS: [<a href="#cock2010">7</a>, Cock <span style="font-style:italic">et al.</span>, 2010].
</li></ul></li><li class="li-enumerate"><em>How should I capitalize &#X201C;Biopython&#X201D;? Is &#X201C;BioPython&#X201D; OK?</em> <br>
 The correct capitalization is &#X201C;Biopython&#X201D;, not &#X201C;BioPython&#X201D; (even though
that would have matched BioPerl, BioJava and BioRuby).</li><li class="li-enumerate"><em>What is going wrong with my print commands?</em> <br>
 This tutorial now uses the Python 3 style print <em>function</em>.
As of Biopython 1.62, we support both Python 2 and Python 3.
The most obvious language difference is the print <em>statement</em>
in Python 2 became a print <em>function</em> in Python 3.<p>For example, this will only work under Python 2:</p><pre class="verbatim">&gt;&gt;&gt; print "Hello World!"
Hello World!
</pre><p>If you try that on Python 3 you&#X2019;ll get a <code>SyntaxError</code>.
Under Python 3 you must write:</p><pre class="verbatim">&gt;&gt;&gt; print("Hello World!")
Hello World!
</pre><p>Surprisingly that will also work on Python 2 &#X2013; but only for simple
examples printing one thing. In general you need to add this magic
line to the start of your Python scripts to use the print function
under Python 2.6 and 2.7:</p><pre class="verbatim">from __future__ import print_function
</pre><p>If you forget to add this magic import, under Python 2 you&#X2019;ll see
extra brackets produced by trying to use the print function when
Python 2 is interpreting it as a print statement and a tuple.</p></li><li class="li-enumerate"><em>How do I find out what version of Biopython I have installed?</em> <br>
 Use this:
<pre class="verbatim">  &gt;&gt;&gt; import Bio
  &gt;&gt;&gt; print(Bio.__version__)
  ...
  </pre>If the &#X201C;<code>import Bio</code>&#X201D; line fails, Biopython is not installed.
If the second line fails, your version is very out of date.
If the version string ends with a plus, you don&#X2019;t have an official
release, but a snapshot of the in development code. </li><li class="li-enumerate"><em>Where is the latest version of this document?</em><br>
 If you download a Biopython source code archive, it will include the
relevant version in both HTML and PDF formats. The latest published
version of this document (updated at each release) is online:
<ul class="itemize"><li class="li-itemize">
<a href="http://biopython.org/DIST/docs/tutorial/Tutorial.html"><span style="font-family:monospace">http://biopython.org/DIST/docs/tutorial/Tutorial.html</span></a>
</li><li class="li-itemize"><a href="http://biopython.org/DIST/docs/tutorial/Tutorial.pdf"><span style="font-family:monospace">http://biopython.org/DIST/docs/tutorial/Tutorial.pdf</span></a>
</li></ul>
If you are using the very latest unreleased code from our repository
you can find copies of the in-progress tutorial here:
<ul class="itemize"><li class="li-itemize">
<a href="http://biopython.org/DIST/docs/tutorial/Tutorial-dev.html"><span style="font-family:monospace">http://biopython.org/DIST/docs/tutorial/Tutorial-dev.html</span></a>
</li><li class="li-itemize"><a href="http://biopython.org/DIST/docs/tutorial/Tutorial-dev.pdf"><span style="font-family:monospace">http://biopython.org/DIST/docs/tutorial/Tutorial-dev.pdf</span></a>
</li></ul></li><li class="li-enumerate"><em>Why is the</em> <code>Seq</code> <em>object missing the upper &amp; lower methods described in this Tutorial?</em> <br>
 You need Biopython 1.53 or later. Alternatively, use <code>str(my_seq).upper()</code> to get an upper case string.
If you need a Seq object, try <code>Seq(str(my_seq).upper())</code> but be careful about blindly re-using the same alphabet.</li><li class="li-enumerate"><em>Why doesn&#X2019;t the</em> <code>Seq</code> <em>object translation method support the</em> <code>cds</code> <em>option described in this Tutorial?</em> <br>
 You need Biopython 1.51 or later.</li><li class="li-enumerate"><em>What file formats do</em> <code>Bio.SeqIO</code> <em>and</em> <code>Bio.AlignIO</code> <em>read and write?</em> <br>
 Check the built in docstrings (<span style="font-family:monospace">from Bio import SeqIO</span>, then <span style="font-family:monospace">help(SeqIO)</span>), or see <a href="http://biopython.org/wiki/SeqIO"><span style="font-family:monospace">http://biopython.org/wiki/SeqIO</span></a> and <a href="http://biopython.org/wiki/AlignIO"><span style="font-family:monospace">http://biopython.org/wiki/AlignIO</span></a> on the wiki for the latest listing.</li><li class="li-enumerate"><em>Why won&#X2019;t the </em> <code>Bio.SeqIO</code> <em>and</em> <code>Bio.AlignIO</code> <em>functions</em> <code>parse</code><em>,</em> <code>read</code> <em>and</em> <code>write</code> <em>take filenames? They insist on handles!</em> <br>
 You need Biopython 1.54 or later, or just use handles explicitly (see Section&#XA0;<a href="#sec%3Aappendix-handles">22.1</a>).
It is especially important to remember to close output handles explicitly after writing your data.</li><li class="li-enumerate"><em>Why won&#X2019;t the </em> <code>Bio.SeqIO.write()</code> <em>and</em> <code>Bio.AlignIO.write()</code> <em>functions accept a single record or alignment? They insist on a list or iterator!</em> <br>
 You need Biopython 1.54 or later, or just wrap the item with <code>[...]</code> to create a list of one element.</li><li class="li-enumerate"><em>Why doesn&#X2019;t</em> <code>str(...)</code> <em>give me the full sequence of a</em> <code>Seq</code> <em>object?</em> <br>
 You need Biopython 1.45 or later.</li><li class="li-enumerate"><em>Why doesn&#X2019;t</em> <code>Bio.Blast</code> <em>work with the latest plain text NCBI blast output?</em> <br>
 The NCBI keep tweaking the plain text output from the BLAST tools, and keeping our parser up to date is/was an ongoing struggle.
If you aren&#X2019;t using the latest version of Biopython, you could try upgrading.
However, we (and the NCBI) recommend you use the XML output instead, which is designed to be read by a computer program.</li><li class="li-enumerate"><em>Why doesn&#X2019;t</em> <code>Bio.Entrez.parse()</code> <em>work? The module imports fine but there is no parse function!</em> <br>
 You need Biopython 1.52 or later.</li><li class="li-enumerate"><em>Why has my script using</em> <code>Bio.Entrez.efetch()</code> <em>stopped working?</em> <br>
 This could be due to NCBI changes in February 2012 introducing EFetch 2.0.
First, they changed the default return modes - you probably want to add <code>retmode="text"</code> to
your call.
Second, they are now stricter about how to provide a list of IDs &#X2013; Biopython 1.59 onwards
turns a list into a comma separated string automatically.</li><li class="li-enumerate"><em>Why doesn&#X2019;t</em> <code>Bio.Blast.NCBIWWW.qblast()</code> <em>give the same results as the NCBI BLAST website?</em> <br>
 You need to specify the same options &#X2013; the NCBI often adjust the default settings on the website,
and they do not match the QBLAST defaults anymore. Check things like the gap penalties and expectation threshold.</li><li class="li-enumerate"><em>Why doesn&#X2019;t</em> <code>Bio.Blast.NCBIXML.read()</code> <em>work? The module imports but there is no read function!</em> <br>
 You need Biopython 1.50 or later. Or, use <span style="font-family:monospace">next(Bio.Blast.NCBIXML.parse(...))</span> instead.</li><li class="li-enumerate"><em>Why doesn&#X2019;t my</em> <code>SeqRecord</code> <em>object have a</em> <code>letter_annotations</code> <em>attribute?</em> <br>
 Per-letter-annotation support was added in Biopython 1.50.</li><li class="li-enumerate"><em>Why can&#X2019;t I slice my</em> <code>SeqRecord</code> <em>to get a sub-record?</em> <br>
 You need Biopython 1.50 or later.</li><li class="li-enumerate"><em>Why can&#X2019;t I add</em> <code>SeqRecord</code> <em>objects together?</em> <br>
 You need Biopython 1.53 or later.</li><li class="li-enumerate"><em>Why doesn&#X2019;t</em> <code>Bio.SeqIO.convert()</code> <em>or</em> <code>Bio.AlignIO.convert()</code> <em>work? The modules import fine but there is no convert function!</em> <br>
 You need Biopython 1.52 or later. Alternatively, combine the <code>parse</code> and <code>write</code>
functions as described in this tutorial (see Sections&#XA0;<a href="#sec%3ASeqIO-conversion">5.5.2</a> and&#XA0;<a href="#sec%3Aconverting-alignments">6.2.1</a>).</li><li class="li-enumerate"><em>Why doesn&#X2019;t</em> <code>Bio.SeqIO.index()</code> <em>work? The module imports fine but there is no index function!</em> <br>
 You need Biopython 1.52 or later.</li><li class="li-enumerate"><em>Why doesn&#X2019;t</em> <code>Bio.SeqIO.index_db()</code> <em>work? The module imports fine but there is no </em><em><span style="font-family:monospace">index_db</span></em><em> function!</em> <br>
 You need Biopython 1.57 or later (and a Python with SQLite3 support).</li><li class="li-enumerate"><em>Where is the</em> <code>MultipleSeqAlignment</code> <em>object? The</em> <code>Bio.Align</code> <em>module imports fine but this class isn&#X2019;t there!</em> <br>
 You need Biopython 1.54 or later. Alternatively, the older <code>Bio.Align.Generic.Alignment</code> class supports some of its functionality, but using this is now discouraged.</li><li class="li-enumerate"><em>Why can&#X2019;t I run command line tools directly from the application wrappers?</em> <br>
 You need Biopython 1.55 or later. Alternatively, use the Python <code>subprocess</code> module directly.</li><li class="li-enumerate"><em>I looked in a directory for code, but I couldn&#X2019;t find the code that does something. Where&#X2019;s it hidden?</em> <br>
 One thing to know is that we put code in <code>__init__.py</code> files. If you are not used to looking for code in this file this can be confusing. The reason we do this is to make the imports easier for users. For instance, instead of having to do a &#X201C;repetitive&#X201D; import like <code>from Bio.GenBank import GenBank</code>, you can just use <code>from Bio import GenBank</code>.</li><li class="li-enumerate"><em>Why does the code from CVS seem out of date?</em> <br>
 In late September 2009, just after the release of Biopython 1.52, we switched from using CVS to git, a distributed version control system. The old CVS server will remain available as a static and read only backup, but if you want to grab the latest code, you&#X2019;ll need to use git instead. See our website for more details.
</li></ol><p>For more general questions, the Python FAQ pages <a href="http://www.python.org/doc/faq/"><span style="font-family:monospace">http://www.python.org/doc/faq/</span></a> may be useful.</p>
<!--TOC chapter id="sec7" Quick Start &#X2013; What can you do with Biopython?-->
<h1 id="sec7" class="chapter">Chapter&#XA0;2&#XA0;&#XA0;Quick Start &#X2013; What can you do with Biopython?</h1><!--SEC END --><p>
<a id="chapter:quick-start"></a></p><p>This section is designed to get you started quickly with Biopython, and to give a general overview of what is available and how to use it. All of the examples in this section assume that you have some general working knowledge of Python, and that you have successfully installed Biopython on your system. If you think you need to brush up on your Python, the main Python web site provides quite a bit of free documentation to get started with (<a href="http://www.python.org/doc/"><span style="font-family:monospace">http://www.python.org/doc/</span></a>).</p><p>Since much biological work on the computer involves connecting with databases on the internet, some of the examples will also require a working internet connection in order to run.</p><p>Now that that is all out of the way, let&#X2019;s get into what we can do with Biopython.</p>
<!--TOC section id="sec8" General overview of what Biopython provides-->
<h2 id="sec8" class="section">2.1&#XA0;&#XA0;General overview of what Biopython provides</h2><!--SEC END --><p>As mentioned in the introduction, Biopython is a set of libraries to provide the ability to deal with &#X201C;things&#X201D; of interest to biologists working on the computer. In general this means that you will need to have at least some programming experience (in Python, of course!) or at least an interest in learning to program. Biopython&#X2019;s job is to make your job easier as a programmer by supplying reusable libraries so that you can focus on answering your specific question of interest, instead of focusing on the internals of parsing a particular file format (of course, if you want to help by writing a parser that doesn&#X2019;t exist and contributing it to Biopython, please go ahead!). So Biopython&#X2019;s job is to make you happy!</p><p>One thing to note about Biopython is that it often provides multiple ways of &#X201C;doing the same thing.&#X201D; Things have improved in recent releases, but this can still be frustrating as in Python there should ideally be one right way to do something. However, this can also be a real benefit because it gives you lots of flexibility and control over the libraries. The tutorial helps to show you the common or easy ways to do things so that you can just make things work. To learn more about the alternative possibilities, look in the Cookbook (Chapter&#XA0;<a href="#chapter%3Acookbook">18</a>, this has some cools tricks and tips), the Advanced section (Chapter&#XA0;<a href="#chapter%3Aadvanced">20</a>), the built in &#X201C;docstrings&#X201D; (via the Python help command, or the <a href="http://biopython.org/DIST/docs/api/">API documentation</a>) or ultimately the code itself.</p>
<!--TOC section id="sec9" Working with sequences-->
<h2 id="sec9" class="section">2.2&#XA0;&#XA0;Working with sequences</h2><!--SEC END --><p>
<a id="sec:sequences"></a></p><p>Disputably (of course!), the central object in bioinformatics is the sequence. Thus, we&#X2019;ll start with a quick introduction to the Biopython mechanisms for dealing with sequences, the <code>Seq</code> object, which we&#X2019;ll discuss in more detail in Chapter&#XA0;<a href="#chapter%3ABio.Seq">3</a>.</p><p>Most of the time when we think about sequences we have in my mind a string of letters like &#X2018;<code>AGTACACTGGT</code>&#X2019;. You can create such <code>Seq</code> object with this sequence as follows - the &#X201C;<code>&gt;&gt;&gt;</code>&#X201D; represents the Python prompt followed by what you would type in:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; my_seq = Seq("AGTACACTGGT")
&gt;&gt;&gt; my_seq
Seq('AGTACACTGGT', Alphabet())
&gt;&gt;&gt; print(my_seq)
AGTACACTGGT
&gt;&gt;&gt; my_seq.alphabet
Alphabet()
</pre><p>What we have here is a sequence object with a <em>generic</em> alphabet - reflecting the fact we have <em>not</em> specified if this is a DNA or protein sequence (okay, a protein with a lot of Alanines, Glycines, Cysteines and Threonines!). We&#X2019;ll talk more about alphabets in Chapter&#XA0;<a href="#chapter%3ABio.Seq">3</a>.</p><p>In addition to having an alphabet, the <code>Seq</code> object differs from the Python string in the methods it supports. You can&#X2019;t do this with a plain string:</p><pre class="verbatim">&gt;&gt;&gt; my_seq
Seq('AGTACACTGGT', Alphabet())
&gt;&gt;&gt; my_seq.complement()
Seq('TCATGTGACCA', Alphabet())
&gt;&gt;&gt; my_seq.reverse_complement()
Seq('ACCAGTGTACT', Alphabet())
</pre><p>The next most important class is the <code>SeqRecord</code> or Sequence Record. This holds a sequence (as a <code>Seq</code> object) with additional annotation including an identifier, name and description. The <code>Bio.SeqIO</code> module for reading and writing sequence file formats works with <code>SeqRecord</code> objects, which will be introduced below and covered in more detail by Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>.</p><p>This covers the basic features and uses of the Biopython sequence class.
Now that you&#X2019;ve got some idea of what it is like to interact with the Biopython libraries, it&#X2019;s time to delve into the fun, fun world of dealing with biological file formats!</p>
<!--TOC section id="sec10" A usage example-->
<h2 id="sec10" class="section">2.3&#XA0;&#XA0;A usage example</h2><!--SEC END --><p>
<a id="sec:orchids"></a></p><p>Before we jump right into parsers and everything else to do with Biopython, let&#X2019;s set up an example to motivate everything we do and make life more interesting. After all, if there wasn&#X2019;t any biology in this tutorial, why would you want you read it?</p><p>Since I love plants, I think we&#X2019;re just going to have to have a plant based example (sorry to all the fans of other organisms out there!). Having just completed a recent trip to our local greenhouse, we&#X2019;ve suddenly developed an incredible obsession with Lady Slipper Orchids (if you wonder why, have a look at some <a href="http://www.flickr.com/search/?q=lady+slipper+orchid&s=int&z=t">Lady Slipper Orchids photos on Flickr</a>, or try a <a href="http://images.google.com/images?q=lady%20slipper%20orchid">Google Image Search</a>).</p><p>Of course, orchids are not only beautiful to look at, they are also extremely interesting for people studying evolution and systematics. So let&#X2019;s suppose we&#X2019;re thinking about writing a funding proposal to do a molecular study of Lady Slipper evolution, and would like to see what kind of research has already been done and how we can add to that.</p><p>After a little bit of reading up we discover that the Lady Slipper Orchids are in the Orchidaceae family and the Cypripedioideae sub-family and are made up of 5 genera: <em>Cypripedium</em>, <em>Paphiopedilum</em>, <em>Phragmipedium</em>, <em>Selenipedium</em> and <em>Mexipedium</em>.</p><p>That gives us enough to get started delving for more information. So, let&#X2019;s look at how the Biopython tools can help us. We&#X2019;ll start with sequence parsing in Section&#XA0;<a href="#sec%3Asequence-parsing">2.4</a>, but the orchids will be back later on as well - for example we&#X2019;ll search PubMed for papers about orchids and extract sequence data from GenBank in Chapter&#XA0;<a href="#chapter%3Aentrez">9</a>, extract data from Swiss-Prot from certain orchid proteins in Chapter&#XA0;<a href="#chapter%3Aswiss_prot">10</a>, and work with ClustalW multiple sequence alignments of orchid proteins in Section&#XA0;<a href="#sec%3Aalign_clustal">6.4.1</a>.</p>
<!--TOC section id="sec11" Parsing sequence file formats-->
<h2 id="sec11" class="section">2.4&#XA0;&#XA0;Parsing sequence file formats</h2><!--SEC END --><p>
<a id="sec:sequence-parsing"></a></p><p>A large part of much bioinformatics work involves dealing with the many types of file formats designed to hold biological data. These files are loaded with interesting biological data, and a special challenge is parsing these files into a format so that you can manipulate them with some kind of programming language. However the task of parsing these files can be frustrated by the fact that the formats can change quite regularly, and that formats may contain small subtleties which can break even the most well designed parsers.</p><p>We are now going to briefly introduce the <code>Bio.SeqIO</code> module &#X2013; you can find out more in Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>. We&#X2019;ll start with an online search for our friends, the lady slipper orchids. To keep this introduction simple, we&#X2019;re just using the NCBI website by hand. Let&#X2019;s just take a look through the nucleotide databases at NCBI, using an Entrez online search (<a href="http://www.ncbi.nlm.nih.gov:80/entrez/query.fcgi?db=Nucleotide"><span style="font-family:monospace">http://www.ncbi.nlm.nih.gov:80/entrez/query.fcgi?db=Nucleotide</span></a>) for everything mentioning the text Cypripedioideae (this is the subfamily of lady slipper orchids). </p><p>When this tutorial was originally written, this search gave us only 94 hits, which we saved as a FASTA formatted text file and as a GenBank formatted text file (files <a href="http://biopython.org/DIST/docs/tutorial/examples/ls_orchid.fasta"><span style="font-family:monospace">ls_orchid.fasta</span></a> and <a href="http://biopython.org/DIST/docs/tutorial/examples/ls_orchid.gbk"><span style="font-family:monospace">ls_orchid.gbk</span></a>, also included with the Biopython source code under <span style="font-family:monospace">docs/tutorial/examples/</span>).</p><p>If you run the search today, you&#X2019;ll get hundreds of results! When following the tutorial, if you want to see the same list of genes, just download the two files above or copy them from <code>docs/examples/</code> in the Biopython source code. In Section&#XA0;<a href="#sec%3Aconnecting-with-biological-databases">2.5</a> we will look at how to do a search like this from within Python.</p>
<!--TOC subsection id="sec12" Simple FASTA parsing example-->
<h3 id="sec12" class="subsection">2.4.1&#XA0;&#XA0;Simple FASTA parsing example</h3><!--SEC END --><p>
<a id="sec:fasta-parsing"></a></p><p>If you open the lady slipper orchids FASTA file <a href="http://biopython.org/DIST/docs/tutorial/examples/ls_orchid.fasta"><span style="font-family:monospace">ls_orchid.fasta</span></a> in your favourite text editor, you&#X2019;ll see that the file starts like this:</p><pre class="verbatim">&gt;gi|2765658|emb|Z78533.1|CIZ78533 C.irapeanum 5.8S rRNA gene and ITS1 and ITS2 DNA
CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGATGAGACCGTGGAATAAACGATCGAGTG
AATCCGGAGGACCGGTGTACTCAGCTCACCGGGGGCATTGCTCCCGTGGTGACCCTGATTTGTTGTTGGG
...
</pre><p>It contains 94 records, each has a line starting with &#X201C;<code>&gt;</code>&#X201D; (greater-than symbol) followed by the sequence on one or more lines. Now try this in Python:</p><pre class="verbatim">from Bio import SeqIO
for seq_record in SeqIO.parse("ls_orchid.fasta", "fasta"):
    print(seq_record.id)
    print(repr(seq_record.seq))
    print(len(seq_record))
</pre><p>You should get something like this on your screen:</p><pre class="verbatim">gi|2765658|emb|Z78533.1|CIZ78533
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGATGAGACCGTGG...CGC', SingleLetterAlphabet())
740
...
gi|2765564|emb|Z78439.1|PBZ78439
Seq('CATTGTTGAGATCACATAATAATTGATCGAGTTAATCTGGAGGATCTGTTTACT...GCC', SingleLetterAlphabet())
592
</pre><p>Notice that the FASTA format does not specify the alphabet, so <code>Bio.SeqIO</code> has defaulted to the rather generic <code>SingleLetterAlphabet()</code> rather than something DNA specific.</p>
<!--TOC subsection id="sec13" Simple GenBank parsing example-->
<h3 id="sec13" class="subsection">2.4.2&#XA0;&#XA0;Simple GenBank parsing example</h3><!--SEC END --><p>Now let&#X2019;s load the GenBank file <a href="http://biopython.org/DIST/docs/tutorial/examples/ls_orchid.gbk"><span style="font-family:monospace">ls_orchid.gbk</span></a> instead - notice that the code to do this is almost identical to the snippet used above for the FASTA file - the only difference is we change the filename and the format string:</p><pre class="verbatim">from Bio import SeqIO
for seq_record in SeqIO.parse("ls_orchid.gbk", "genbank"):
    print(seq_record.id)
    print(repr(seq_record.seq))
    print(len(seq_record))
</pre><p>This should give:</p><pre class="verbatim">Z78533.1
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGATGAGACCGTGG...CGC', IUPACAmbiguousDNA())
740
...
Z78439.1
Seq('CATTGTTGAGATCACATAATAATTGATCGAGTTAATCTGGAGGATCTGTTTACT...GCC', IUPACAmbiguousDNA())
592
</pre><p>This time <code>Bio.SeqIO</code> has been able to choose a sensible alphabet, IUPAC Ambiguous DNA. You&#X2019;ll also notice that a shorter string has been used as the <code>seq_record.id</code> in this case.</p>
<!--TOC subsection id="sec14" I love parsing &#X2013; please don&#X2019;t stop talking about it!-->
<h3 id="sec14" class="subsection">2.4.3&#XA0;&#XA0;I love parsing &#X2013; please don&#X2019;t stop talking about it!</h3><!--SEC END --><p>Biopython has a lot of parsers, and each has its own little special niches based on the sequence format it is parsing and all of that. Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a> covers <code>Bio.SeqIO</code> in more detail, while Chapter&#XA0;<a href="#chapter%3ABio.AlignIO">6</a> introduces <code>Bio.AlignIO</code> for sequence alignments.</p><p>While the most popular file formats have parsers integrated into <code>Bio.SeqIO</code> and/or <code>Bio.AlignIO</code>, for some of the rarer and unloved file formats there is either no parser at all, or an old parser which has not been linked in yet.
Please also check the wiki pages <a href="http://biopython.org/wiki/SeqIO"><span style="font-family:monospace">http://biopython.org/wiki/SeqIO</span></a> and <a href="http://biopython.org/wiki/AlignIO"><span style="font-family:monospace">http://biopython.org/wiki/AlignIO</span></a> for the latest information, or ask on the mailing list. The wiki pages should include an up to date list of supported file types, and some additional examples.</p><p>The next place to look for information about specific parsers and how to do cool things with them is in the Cookbook (Chapter&#XA0;<a href="#chapter%3Acookbook">18</a> of this Tutorial). If you don&#X2019;t find the information you are looking for, please consider helping out your poor overworked documentors and submitting a cookbook entry about it! (once you figure out how to do it, that is!)</p>
<!--TOC section id="sec15" Connecting with biological databases-->
<h2 id="sec15" class="section">2.5&#XA0;&#XA0;Connecting with biological databases</h2><!--SEC END --><p>
<a id="sec:connecting-with-biological-databases"></a></p><p>One of the very common things that you need to do in bioinformatics is extract information from biological databases. It can be quite tedious to access these databases manually, especially if you have a lot of repetitive work to do. Biopython attempts to save you time and energy by making some on-line databases available from Python scripts. Currently, Biopython has code to extract information from the following databases:</p><ul class="itemize"><li class="li-itemize">
<a href="http://www.ncbi.nlm.nih.gov/Entrez/">Entrez</a> (and <a href="http://www.ncbi.nlm.nih.gov/PubMed/">PubMed</a>) from the NCBI &#X2013; See Chapter&#XA0;<a href="#chapter%3Aentrez">9</a>.
</li><li class="li-itemize"><a href="http://www.expasy.org/">ExPASy</a> &#X2013; See Chapter&#XA0;<a href="#chapter%3Aswiss_prot">10</a>.
</li><li class="li-itemize"><a href="http://scop.mrc-lmb.cam.ac.uk/scop/">SCOP</a> &#X2013; See the <code>Bio.SCOP.search()</code> function.
</li></ul><p>The code in these modules basically makes it easy to write Python code that interact with the CGI scripts on these pages, so that you can get results in an easy to deal with format. In some cases, the results can be tightly integrated with the Biopython parsers to make it even easier to extract information.</p>
<!--TOC section id="sec16" What to do next-->
<h2 id="sec16" class="section">2.6&#XA0;&#XA0;What to do next</h2><!--SEC END --><p>Now that you&#X2019;ve made it this far, you hopefully have a good understanding of the basics of Biopython and are ready to start using it for doing useful work. The best thing to do now is finish reading this tutorial, and then if you want start snooping around in the source code, and looking at the automatically generated documentation.</p><p>Once you get a picture of what you want to do, and what libraries in Biopython will do it, you should take a peak at the Cookbook (Chapter&#XA0;<a href="#chapter%3Acookbook">18</a>), which may have example code to do something similar to what you want to do.</p><p>If you know what you want to do, but can&#X2019;t figure out how to do it, please feel free to post questions to the main Biopython list (see <a href="http://biopython.org/wiki/Mailing_lists"><span style="font-family:monospace">http://biopython.org/wiki/Mailing_lists</span></a>). This will not only help us answer your question, it will also allow us to improve the documentation so it can help the next person do what you want to do.</p><p>Enjoy the code!</p>
<!--TOC chapter id="sec17" Sequence objects-->
<h1 id="sec17" class="chapter">Chapter&#XA0;3&#XA0;&#XA0;Sequence objects</h1><!--SEC END --><p>
<a id="chapter:Bio.Seq"></a></p><p>Biological sequences are arguably the central object in Bioinformatics, and in this chapter we&#X2019;ll introduce the Biopython mechanism for dealing with sequences, the <code>Seq</code> object.
Chapter&#XA0;<a href="#chapter%3ASeqRecord">4</a> will introduce the related <code>SeqRecord</code> object, which combines the sequence information with any annotation, used again in Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a> for Sequence Input/Output.</p><p>Sequences are essentially strings of letters like <code>AGTACACTGGT</code>, which seems very natural since this is the most common way that sequences are seen in biological file formats.</p><p>There are two important differences between <code>Seq</code> objects and standard Python strings.
First of all, they have different methods. Although the <code>Seq</code> object supports many of the same methods as a plain string, its <code>translate()</code> method differs by doing biological translation, and there are also additional biologically relevant methods like <code>reverse_complement()</code>.
Secondly, the <code>Seq</code> object has an important attribute, <code>alphabet</code>, which is an object describing what the individual characters making up the sequence string &#X201C;mean&#X201D;, and how they should be interpreted. For example, is <code>AGTACACTGGT</code> a DNA sequence, or just a protein sequence that happens to be rich in Alanines, Glycines, Cysteines
and Threonines?</p>
<!--TOC section id="sec18" Sequences and Alphabets-->
<h2 id="sec18" class="section">3.1&#XA0;&#XA0;Sequences and Alphabets</h2><!--SEC END --><p>The alphabet object is perhaps the important thing that makes the <code>Seq</code> object more than just a string. The currently available alphabets for Biopython are defined in the <code>Bio.Alphabet</code> module. We&#X2019;ll use the IUPAC alphabets (<a href="http://www.chem.qmw.ac.uk/iupac/"><span style="font-family:monospace">http://www.chem.qmw.ac.uk/iupac/</span></a>) here to deal with some of our favorite objects: DNA, RNA and Proteins.</p><p><code>Bio.Alphabet.IUPAC</code> provides basic definitions for proteins, DNA and RNA, but additionally provides the ability to extend and customize the basic definitions. For instance, for proteins, there is a basic IUPACProtein class, but there is an additional ExtendedIUPACProtein class providing for the additional elements &#X201C;U&#X201D; (or &#X201C;Sec&#X201D; for selenocysteine) and &#X201C;O&#X201D; (or &#X201C;Pyl&#X201D; for pyrrolysine), plus the ambiguous symbols &#X201C;B&#X201D; (or &#X201C;Asx&#X201D; for asparagine or aspartic acid), &#X201C;Z&#X201D; (or &#X201C;Glx&#X201D; for glutamine or glutamic acid), &#X201C;J&#X201D; (or &#X201C;Xle&#X201D; for leucine isoleucine) and &#X201C;X&#X201D; (or &#X201C;Xxx&#X201D; for an unknown amino acid). For DNA you&#X2019;ve got choices of IUPACUnambiguousDNA, which provides for just the basic letters, IUPACAmbiguousDNA (which provides for ambiguity letters for every possible situation) and ExtendedIUPACDNA, which allows letters for modified bases. Similarly, RNA can be represented by IUPACAmbiguousRNA or IUPACUnambiguousRNA.</p><p>The advantages of having an alphabet class are two fold. First, this gives an idea of the type of information the Seq object contains. Secondly, this provides a means of constraining the information, as a means of type checking.</p><p>Now that we know what we are dealing with, let&#X2019;s look at how to utilize this class to do interesting work.
You can create an ambiguous sequence with the default generic alphabet like this:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; my_seq = Seq("AGTACACTGGT")
&gt;&gt;&gt; my_seq
Seq('AGTACACTGGT', Alphabet())
&gt;&gt;&gt; my_seq.alphabet
Alphabet()
</pre><p>However, where possible you should specify the alphabet explicitly when creating your sequence objects - in this case an unambiguous DNA alphabet object:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; my_seq = Seq("AGTACACTGGT", IUPAC.unambiguous_dna)
&gt;&gt;&gt; my_seq
Seq('AGTACACTGGT', IUPACUnambiguousDNA())
&gt;&gt;&gt; my_seq.alphabet
IUPACUnambiguousDNA()
</pre><p>Unless of course, this really is an amino acid sequence:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; my_prot = Seq("AGTACACTGGT", IUPAC.protein)
&gt;&gt;&gt; my_prot
Seq('AGTACACTGGT', IUPACProtein())
&gt;&gt;&gt; my_prot.alphabet
IUPACProtein()
</pre>
<!--TOC section id="sec19" Sequences act like strings-->
<h2 id="sec19" class="section">3.2&#XA0;&#XA0;Sequences act like strings</h2><!--SEC END --><p>In many ways, we can deal with Seq objects as if they were normal Python strings, for example getting the length, or iterating over the elements:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; my_seq = Seq("GATCG", IUPAC.unambiguous_dna)
&gt;&gt;&gt; for index, letter in enumerate(my_seq):
...     print("%i %s" % (index, letter))
0 G
1 A
2 T
3 C
4 G
&gt;&gt;&gt; print(len(my_seq))
5
</pre><p>You can access elements of the sequence in the same way as for strings (but remember, Python counts from zero!):</p><pre class="verbatim">&gt;&gt;&gt; print(my_seq[0]) #first letter
G
&gt;&gt;&gt; print(my_seq[2]) #third letter
T
&gt;&gt;&gt; print(my_seq[-1]) #last letter
G
</pre><p>The <code>Seq</code> object has a <code>.count()</code> method, just like a string.
Note that this means that like a Python string, this gives a
<em>non-overlapping</em> count:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; "AAAA".count("AA")
2
&gt;&gt;&gt; Seq("AAAA").count("AA")
2
</pre><p>For some biological uses, you may actually want an overlapping count
(i.e. 3 in this trivial example). When searching for single letters, this
makes no difference:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; my_seq = Seq('GATCGATGGGCCTATATAGGATCGAAAATCGC', IUPAC.unambiguous_dna)
&gt;&gt;&gt; len(my_seq)
32
&gt;&gt;&gt; my_seq.count("G")
9
&gt;&gt;&gt; 100 * float(my_seq.count("G") + my_seq.count("C")) / len(my_seq)
46.875
</pre><p>While you could use the above snippet of code to calculate a GC%, note that the <code>Bio.SeqUtils</code> module has several GC functions already built. For example:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; from Bio.SeqUtils import GC
&gt;&gt;&gt; my_seq = Seq('GATCGATGGGCCTATATAGGATCGAAAATCGC', IUPAC.unambiguous_dna)
&gt;&gt;&gt; GC(my_seq)
46.875
</pre><p>Note that using the <code>Bio.SeqUtils.GC()</code> function should automatically cope with mixed case sequences and the ambiguous nucleotide S which means G or C.</p><p>Also note that just like a normal Python string, the <code>Seq</code> object is in some ways &#X201C;read-only&#X201D;. If you need to edit your sequence, for example simulating a point mutation, look at the Section&#XA0;<a href="#sec%3Amutable-seq">3.12</a> below which talks about the <code>MutableSeq</code> object.</p>
<!--TOC section id="sec20" Slicing a sequence-->
<h2 id="sec20" class="section">3.3&#XA0;&#XA0;Slicing a sequence</h2><!--SEC END --><p>A more complicated example, let&#X2019;s get a slice of the sequence:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; my_seq = Seq("GATCGATGGGCCTATATAGGATCGAAAATCGC", IUPAC.unambiguous_dna)
&gt;&gt;&gt; my_seq[4:12]
Seq('GATGGGCC', IUPACUnambiguousDNA())
</pre><p>Two things are interesting to note. First, this follows the normal conventions for Python strings. So the first element of the sequence is 0 (which is normal for computer science, but not so normal for biology). When you do a slice the first item is included (i.e.&#XA0;4 in this case) and the last is excluded (12 in this case), which is the way things work in Python, but of course not necessarily the way everyone in the world would expect. The main goal is to stay consistent with what Python does.</p><p>The second thing to notice is that the slice is performed on the sequence data string, but the new object produced is another <code>Seq</code> object which retains the alphabet information from the original <code>Seq</code> object.</p><p>Also like a Python string, you can do slices with a start, stop and <em>stride</em> (the step size, which defaults to one). For example, we can get the first, second and third codon positions of this DNA sequence:</p><pre class="verbatim">&gt;&gt;&gt; my_seq[0::3]
Seq('GCTGTAGTAAG', IUPACUnambiguousDNA())
&gt;&gt;&gt; my_seq[1::3]
Seq('AGGCATGCATC', IUPACUnambiguousDNA())
&gt;&gt;&gt; my_seq[2::3]
Seq('TAGCTAAGAC', IUPACUnambiguousDNA())
</pre><p>Another stride trick you might have seen with a Python string is the use of a -1 stride to reverse the string. You can do this with a <code>Seq</code> object too:</p><pre class="verbatim">&gt;&gt;&gt; my_seq[::-1]
Seq('CGCTAAAAGCTAGGATATATCCGGGTAGCTAG', IUPACUnambiguousDNA())
</pre>
<!--TOC section id="sec21" Turning Seq objects into strings-->
<h2 id="sec21" class="section">3.4&#XA0;&#XA0;Turning Seq objects into strings</h2><!--SEC END --><p>
<a id="sec:seq-to-string"></a></p><p>If you really do just need a plain string, for example to write to a file, or insert into a database, then this is very easy to get:
</p><pre class="verbatim">&gt;&gt;&gt; str(my_seq)
'GATCGATGGGCCTATATAGGATCGAAAATCGC'
</pre><p>Since calling <code>str()</code> on a <code>Seq</code> object returns the full sequence as a string,
you often don&#X2019;t actually have to do this conversion explicitly.
Python does this automatically in the print function
(and the print statement under Python 2):
</p><pre class="verbatim">&gt;&gt;&gt; print(my_seq)
GATCGATGGGCCTATATAGGATCGAAAATCGC
</pre><p>You can also use the <code>Seq</code> object directly with a <code>%s</code> placeholder when using the Python string formatting or interpolation operator (<code>%</code>):
</p><pre class="verbatim">&gt;&gt;&gt; fasta_format_string = "&gt;Name\n%s\n" % my_seq
&gt;&gt;&gt; print(fasta_format_string)
&gt;Name
GATCGATGGGCCTATATAGGATCGAAAATCGC
&lt;BLANKLINE&gt;
</pre><p>This line of code constructs a simple FASTA format record (without worrying about line wrapping).
Section&#XA0;<a href="#sec%3ASeqRecord-format">4.5</a> describes a neat way to get a FASTA formatted
string from a <code>SeqRecord</code> object, while the more general topic of reading and
writing FASTA format sequence files is covered in Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>.</p><pre class="verbatim">&gt;&gt;&gt; str(my_seq)
'GATCGATGGGCCTATATAGGATCGAAAATCGC'
</pre>
<!--TOC section id="sec22" Concatenating or adding sequences-->
<h2 id="sec22" class="section">3.5&#XA0;&#XA0;Concatenating or adding sequences</h2><!--SEC END --><p>Naturally, you can in principle add any two Seq objects together - just like you can with Python strings to concatenate them. However, you can&#X2019;t add sequences with incompatible alphabets, such as a protein sequence and a DNA sequence:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; protein_seq = Seq("EVRNAK", IUPAC.protein)
&gt;&gt;&gt; dna_seq = Seq("ACGT", IUPAC.unambiguous_dna)
&gt;&gt;&gt; protein_seq + dna_seq
Traceback (most recent call last):
...
TypeError: Incompatible alphabets IUPACProtein() and IUPACUnambiguousDNA()
</pre><p>If you <em>really</em> wanted to do this, you&#X2019;d have to first give both sequences generic alphabets:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Alphabet import generic_alphabet
&gt;&gt;&gt; protein_seq.alphabet = generic_alphabet
&gt;&gt;&gt; dna_seq.alphabet = generic_alphabet
&gt;&gt;&gt; protein_seq + dna_seq
Seq('EVRNAKACGT', Alphabet())
</pre><p>Here is an example of adding a generic nucleotide sequence to an unambiguous IUPAC DNA sequence, resulting in an ambiguous nucleotide sequence:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import generic_nucleotide
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; nuc_seq = Seq("GATCGATGC", generic_nucleotide)
&gt;&gt;&gt; dna_seq = Seq("ACGT", IUPAC.unambiguous_dna)
&gt;&gt;&gt; nuc_seq
Seq('GATCGATGC', NucleotideAlphabet())
&gt;&gt;&gt; dna_seq
Seq('ACGT', IUPACUnambiguousDNA())
&gt;&gt;&gt; nuc_seq + dna_seq
Seq('GATCGATGCACGT', NucleotideAlphabet())
</pre><p>You may often have many sequences to add together, which can be done with a for loop like this:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import generic_dna
&gt;&gt;&gt; list_of_seqs = [Seq("ACGT", generic_dna), Seq("AACC", generic_dna), Seq("GGTT", generic_dna)]
&gt;&gt;&gt; concatenated = Seq("", generic_dna)
&gt;&gt;&gt; for s in list_of_seqs:
...      concatenated += s
...
&gt;&gt;&gt; concatenated
Seq('ACGTAACCGGTT', DNAAlphabet())
</pre><p>Or, a more elegant approach is to the use built in <code>sum</code> function with its optional start value argument (which otherwise defaults to zero):</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import generic_dna
&gt;&gt;&gt; list_of_seqs = [Seq("ACGT", generic_dna), Seq("AACC", generic_dna), Seq("GGTT", generic_dna)]
&gt;&gt;&gt; sum(list_of_seqs, Seq("", generic_dna))
Seq('ACGTAACCGGTT', DNAAlphabet())
</pre><p>Unlike the Python string, the Biopython <code>Seq</code> does not (currently) have a <code>.join</code> method.</p>
<!--TOC section id="sec23" Changing case-->
<h2 id="sec23" class="section">3.6&#XA0;&#XA0;Changing case</h2><!--SEC END --><p>Python strings have very useful <code>upper</code> and <code>lower</code> methods for changing the case.
As of Biopython 1.53, the <code>Seq</code> object gained similar methods which are alphabet aware.
For example,</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import generic_dna
&gt;&gt;&gt; dna_seq = Seq("acgtACGT", generic_dna)
&gt;&gt;&gt; dna_seq
Seq('acgtACGT', DNAAlphabet())
&gt;&gt;&gt; dna_seq.upper()
Seq('ACGTACGT', DNAAlphabet())
&gt;&gt;&gt; dna_seq.lower()
Seq('acgtacgt', DNAAlphabet())
</pre><p>These are useful for doing case insensitive matching:</p><pre class="verbatim">&gt;&gt;&gt; "GTAC" in dna_seq
False
&gt;&gt;&gt; "GTAC" in dna_seq.upper()
True
</pre><p>Note that strictly speaking the IUPAC alphabets are for upper case
sequences only, thus:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; dna_seq = Seq("ACGT", IUPAC.unambiguous_dna)
&gt;&gt;&gt; dna_seq
Seq('ACGT', IUPACUnambiguousDNA())
&gt;&gt;&gt; dna_seq.lower()
Seq('acgt', DNAAlphabet())
</pre>
<!--TOC section id="sec24" Nucleotide sequences and (reverse) complements-->
<h2 id="sec24" class="section">3.7&#XA0;&#XA0;Nucleotide sequences and (reverse) complements</h2><!--SEC END --><p>
<a id="sec:seq-reverse-complement"></a></p><p>For nucleotide sequences, you can easily obtain the complement or reverse
complement of a <code>Seq</code> object using its built-in methods:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; my_seq = Seq("GATCGATGGGCCTATATAGGATCGAAAATCGC", IUPAC.unambiguous_dna)
&gt;&gt;&gt; my_seq
Seq('GATCGATGGGCCTATATAGGATCGAAAATCGC', IUPACUnambiguousDNA())
&gt;&gt;&gt; my_seq.complement()
Seq('CTAGCTACCCGGATATATCCTAGCTTTTAGCG', IUPACUnambiguousDNA())
&gt;&gt;&gt; my_seq.reverse_complement()
Seq('GCGATTTTCGATCCTATATAGGCCCATCGATC', IUPACUnambiguousDNA())
</pre><p>As mentioned earlier, an easy way to just reverse a <code>Seq</code> object (or a
Python string) is slice it with -1 step:</p><pre class="verbatim">&gt;&gt;&gt; my_seq[::-1]
Seq('CGCTAAAAGCTAGGATATATCCGGGTAGCTAG', IUPACUnambiguousDNA())
</pre><p>In all of these operations, the alphabet property is maintained. This is very
useful in case you accidentally end up trying to do something weird like take
the (reverse)complement of a protein sequence:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; protein_seq = Seq("EVRNAK", IUPAC.protein)
&gt;&gt;&gt; protein_seq.complement()
Traceback (most recent call last):
...
ValueError: Proteins do not have complements!
</pre><p>The example in Section&#XA0;<a href="#sec%3ASeqIO-reverse-complement">5.5.3</a> combines the <code>Seq</code>
object&#X2019;s reverse complement method with <code>Bio.SeqIO</code> for sequence input/output.</p>
<!--TOC section id="sec25" Transcription-->
<h2 id="sec25" class="section">3.8&#XA0;&#XA0;Transcription</h2><!--SEC END --><p>
Before talking about transcription, I want to try and clarify the strand issue.
Consider the following (made up) stretch of double stranded DNA which
encodes a short peptide:</p><table style="border-spacing:6px;border-collapse:separate;" class="cellpading0"><tr><td style="text-align:right;white-space:nowrap" >&nbsp;</td></tr>
<tr><td style="text-align:right;white-space:nowrap" >&nbsp;</td><td style="text-align:center;white-space:nowrap" ><span style="font-size:small">DNA coding strand (aka Crick strand, strand </span><span style="font-size:small">+1</span><span style="font-size:small">)</span></td><td style="text-align:left;white-space:nowrap" >&nbsp;</td></tr>
<tr><td style="text-align:right;white-space:nowrap" >5&#X2019;</td><td style="text-align:center;white-space:nowrap" ><span style="font-family:monospace">ATGGCCATTGTAATGGGCCGCTGAAAGGGTGCCCGATAG</span></td><td style="text-align:left;white-space:nowrap" >3&#X2019; </td></tr>
<tr><td style="text-align:right;white-space:nowrap" >&nbsp;</td><td style="text-align:center;white-space:nowrap" ><span style="font-family:monospace">|||||||||||||||||||||||||||||||||||||||</span></td><td style="text-align:left;white-space:nowrap" >&nbsp;</td></tr>
<tr><td style="text-align:right;white-space:nowrap" >3&#X2019;</td><td style="text-align:center;white-space:nowrap" ><span style="font-family:monospace">TACCGGTAACATTACCCGGCGACTTTCCCACGGGCTATC</span></td><td style="text-align:left;white-space:nowrap" >5&#X2019; </td></tr>
<tr><td style="text-align:right;white-space:nowrap" >&nbsp;</td><td style="text-align:center;white-space:nowrap" ><span style="font-size:small">DNA template strand (aka Watson strand, strand </span><span style="font-size:small">&#X2212;1</span><span style="font-size:small">)</span></td><td style="text-align:left;white-space:nowrap" >&nbsp;</td></tr>
<tr><td style="text-align:right;white-space:nowrap" >&nbsp;</td></tr>
<tr><td style="text-align:right;white-space:nowrap" >&nbsp;</td><td style="text-align:center;white-space:nowrap" ><span style="font-size:x-large">|</span></td><td style="text-align:left;white-space:nowrap" >&nbsp;</td></tr>
<tr><td style="text-align:right;white-space:nowrap" >&nbsp;</td><td style="text-align:center;white-space:nowrap" >Transcription</td><td style="text-align:left;white-space:nowrap" >&nbsp;</td></tr>
<tr><td style="text-align:right;white-space:nowrap" >&nbsp;</td><td style="text-align:center;white-space:nowrap" ><span style="font-size:x-large">&#X2193;</span></td><td style="text-align:left;white-space:nowrap" >&nbsp;</td></tr>
<tr><td style="text-align:right;white-space:nowrap" >&nbsp;</td></tr>
<tr><td style="text-align:right;white-space:nowrap" >5&#X2019;</td><td style="text-align:center;white-space:nowrap" ><span style="font-family:monospace">AUGGCCAUUGUAAUGGGCCGCUGAAAGGGUGCCCGAUAG</span></td><td style="text-align:left;white-space:nowrap" >3&#X2019; </td></tr>
<tr><td style="text-align:right;white-space:nowrap" >&nbsp;</td><td style="text-align:center;white-space:nowrap" ><span style="font-size:small">Single stranded messenger RNA</span></td><td style="text-align:left;white-space:nowrap" >&nbsp;</td></tr>
<tr><td style="text-align:right;white-space:nowrap" >&nbsp;</td></tr>
</table><p>The actual biological transcription process works from the template strand, doing a reverse complement (TCAG &#X2192; CUGA) to give the mRNA. However, in Biopython and bioinformatics in general, we typically work directly with the coding strand because this means we can get the mRNA sequence just by switching T &#X2192; U.</p><p>Now let&#X2019;s actually get down to doing a transcription in Biopython. First, let&#X2019;s create <code>Seq</code> objects for the coding and template DNA strands:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; coding_dna = Seq("ATGGCCATTGTAATGGGCCGCTGAAAGGGTGCCCGATAG", IUPAC.unambiguous_dna)
&gt;&gt;&gt; coding_dna
Seq('ATGGCCATTGTAATGGGCCGCTGAAAGGGTGCCCGATAG', IUPACUnambiguousDNA())
&gt;&gt;&gt; template_dna = coding_dna.reverse_complement()
&gt;&gt;&gt; template_dna
Seq('CTATCGGGCACCCTTTCAGCGGCCCATTACAATGGCCAT', IUPACUnambiguousDNA())
</pre><p>These should match the figure above - remember by convention nucleotide sequences are normally read from the 5&#X2019; to 3&#X2019; direction, while in the figure the template strand is shown reversed.</p><p>Now let&#X2019;s transcribe the coding strand into the corresponding mRNA, using the <code>Seq</code> object&#X2019;s built in <code>transcribe</code> method:
</p><pre class="verbatim">&gt;&gt;&gt; coding_dna
Seq('ATGGCCATTGTAATGGGCCGCTGAAAGGGTGCCCGATAG', IUPACUnambiguousDNA())
&gt;&gt;&gt; messenger_rna = coding_dna.transcribe()
&gt;&gt;&gt; messenger_rna
Seq('AUGGCCAUUGUAAUGGGCCGCUGAAAGGGUGCCCGAUAG', IUPACUnambiguousRNA())
</pre><p>As you can see, all this does is switch T &#X2192; U, and adjust the alphabet.</p><p>If you do want to do a true biological transcription starting with the template strand, then this becomes a two-step process:
</p><pre class="verbatim">&gt;&gt;&gt; template_dna.reverse_complement().transcribe()
Seq('AUGGCCAUUGUAAUGGGCCGCUGAAAGGGUGCCCGAUAG', IUPACUnambiguousRNA())
</pre><p>The <code>Seq</code> object also includes a back-transcription method for going from the mRNA to the coding strand of the DNA. Again, this is a simple U &#X2192; T substitution and associated change of alphabet:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; messenger_rna = Seq("AUGGCCAUUGUAAUGGGCCGCUGAAAGGGUGCCCGAUAG", IUPAC.unambiguous_rna)
&gt;&gt;&gt; messenger_rna
Seq('AUGGCCAUUGUAAUGGGCCGCUGAAAGGGUGCCCGAUAG', IUPACUnambiguousRNA())
&gt;&gt;&gt; messenger_rna.back_transcribe()
Seq('ATGGCCATTGTAATGGGCCGCTGAAAGGGTGCCCGATAG', IUPACUnambiguousDNA())
</pre><p><em>Note:</em> The <code>Seq</code> object&#X2019;s <code>transcribe</code> and <code>back_transcribe</code> methods
were added in Biopython 1.49. For older releases you would have to use the <code>Bio.Seq</code>
module&#X2019;s functions instead, see Section&#XA0;<a href="#sec%3Aseq-module-functions">3.14</a>.</p>
<!--TOC section id="sec26" Translation-->
<h2 id="sec26" class="section">3.9&#XA0;&#XA0;Translation</h2><!--SEC END --><p>
<a id="sec:translation"></a>
Sticking with the same example discussed in the transcription section above,
now let&#X2019;s translate this mRNA into the corresponding protein sequence - again taking
advantage of one of the <code>Seq</code> object&#X2019;s biological methods:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; messenger_rna = Seq("AUGGCCAUUGUAAUGGGCCGCUGAAAGGGUGCCCGAUAG", IUPAC.unambiguous_rna)
&gt;&gt;&gt; messenger_rna
Seq('AUGGCCAUUGUAAUGGGCCGCUGAAAGGGUGCCCGAUAG', IUPACUnambiguousRNA())
&gt;&gt;&gt; messenger_rna.translate()
Seq('MAIVMGR*KGAR*', HasStopCodon(IUPACProtein(), '*'))
</pre><p>You can also translate directly from the coding strand DNA sequence:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; coding_dna = Seq("ATGGCCATTGTAATGGGCCGCTGAAAGGGTGCCCGATAG", IUPAC.unambiguous_dna)
&gt;&gt;&gt; coding_dna
Seq('ATGGCCATTGTAATGGGCCGCTGAAAGGGTGCCCGATAG', IUPACUnambiguousDNA())
&gt;&gt;&gt; coding_dna.translate()
Seq('MAIVMGR*KGAR*', HasStopCodon(IUPACProtein(), '*'))
</pre><p>You should notice in the above protein sequences that in addition to the end stop character, there is an internal stop as well. This was a deliberate choice of example, as it gives an excuse to talk about some optional arguments, including different translation tables (Genetic Codes).</p><p>The translation tables available in Biopython are based on those <a href="http://www.ncbi.nlm.nih.gov/Taxonomy/Utils/wprintgc.cgi">from the NCBI</a> (see the next section of this tutorial). By default, translation will use the <em>standard</em> genetic code (NCBI table id 1).
Suppose we are dealing with a mitochondrial sequence. We need to tell the translation function to use the relevant genetic code instead:
</p><pre class="verbatim">&gt;&gt;&gt; coding_dna.translate(table="Vertebrate Mitochondrial")
Seq('MAIVMGRWKGAR*', HasStopCodon(IUPACProtein(), '*'))
</pre><p>You can also specify the table using the NCBI table number which is shorter, and often included in the feature annotation of GenBank files:
</p><pre class="verbatim">&gt;&gt;&gt; coding_dna.translate(table=2)
Seq('MAIVMGRWKGAR*', HasStopCodon(IUPACProtein(), '*'))
</pre><p>Now, you may want to translate the nucleotides up to the first in frame stop codon,
and then stop (as happens in nature):
</p><pre class="verbatim">&gt;&gt;&gt; coding_dna.translate()
Seq('MAIVMGR*KGAR*', HasStopCodon(IUPACProtein(), '*'))
&gt;&gt;&gt; coding_dna.translate(to_stop=True)
Seq('MAIVMGR', IUPACProtein())
&gt;&gt;&gt; coding_dna.translate(table=2)
Seq('MAIVMGRWKGAR*', HasStopCodon(IUPACProtein(), '*'))
&gt;&gt;&gt; coding_dna.translate(table=2, to_stop=True)
Seq('MAIVMGRWKGAR', IUPACProtein())
</pre><p>Notice that when you use the <code>to_stop</code> argument, the stop codon itself
is not translated - and the stop symbol is not included at the end of your protein
sequence.</p><p>You can even specify the stop symbol if you don&#X2019;t like the default asterisk:
</p><pre class="verbatim">&gt;&gt;&gt; coding_dna.translate(table=2, stop_symbol="@")
Seq('MAIVMGRWKGAR@', HasStopCodon(IUPACProtein(), '@'))
</pre><p>Now, suppose you have a complete coding sequence CDS, which is to say a
nucleotide sequence (e.g. mRNA &#X2013; after any splicing) which is a whole number
of codons (i.e. the length is a multiple of three), commences with a start
codon, ends with a stop codon, and has no internal in-frame stop codons.
In general, given a complete CDS, the default translate method will do what
you want (perhaps with the <code>to_stop</code> option). However, what if your
sequence uses a non-standard start codon? This happens a lot in bacteria &#X2013;
for example the gene yaaX in <span style="font-family:monospace">E. coli</span> K12:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import generic_dna
&gt;&gt;&gt; gene = Seq("GTGAAAAAGATGCAATCTATCGTACTCGCACTTTCCCTGGTTCTGGTCGCTCCCATGGCA" + \
...            "GCACAGGCTGCGGAAATTACGTTAGTCCCGTCAGTAAAATTACAGATAGGCGATCGTGAT" + \
...            "AATCGTGGCTATTACTGGGATGGAGGTCACTGGCGCGACCACGGCTGGTGGAAACAACAT" + \
...            "TATGAATGGCGAGGCAATCGCTGGCACCTACACGGACCGCCGCCACCGCCGCGCCACCAT" + \
...            "AAGAAAGCTCCTCATGATCATCACGGCGGTCATGGTCCAGGCAAACATCACCGCTAA",
...            generic_dna)
&gt;&gt;&gt; gene.translate(table="Bacterial")
Seq('VKKMQSIVLALSLVLVAPMAAQAAEITLVPSVKLQIGDRDNRGYYWDGGHWRDH...HR*',
HasStopCodon(ExtendedIUPACProtein(), '*')
&gt;&gt;&gt; gene.translate(table="Bacterial", to_stop=True)
Seq('VKKMQSIVLALSLVLVAPMAAQAAEITLVPSVKLQIGDRDNRGYYWDGGHWRDH...HHR',
ExtendedIUPACProtein())
</pre><p>In the bacterial genetic code <span style="font-family:monospace">GTG</span> is a valid start codon,
and while it does <em>normally</em> encode Valine, if used as a start codon it
should be translated as methionine. This happens if you tell Biopython your
sequence is a complete CDS:</p><pre class="verbatim">&gt;&gt;&gt; gene.translate(table="Bacterial", cds=True)
Seq('MKKMQSIVLALSLVLVAPMAAQAAEITLVPSVKLQIGDRDNRGYYWDGGHWRDH...HHR',
ExtendedIUPACProtein())
</pre><p>In addition to telling Biopython to translate an alternative start codon as
methionine, using this option also makes sure your sequence really is a valid
CDS (you&#X2019;ll get an exception if not).</p><p>The example in Section&#XA0;<a href="#sec%3ASeqIO-translate">18.1.3</a> combines the <code>Seq</code> object&#X2019;s
translate method with <code>Bio.SeqIO</code> for sequence input/output.</p>
<!--TOC section id="sec27" Translation Tables-->
<h2 id="sec27" class="section">3.10&#XA0;&#XA0;Translation Tables</h2><!--SEC END --><p>In the previous sections we talked about the <code>Seq</code> object translation method (and mentioned the equivalent function in the <code>Bio.Seq</code> module &#X2013; see
Section&#XA0;<a href="#sec%3Aseq-module-functions">3.14</a>).
Internally these use codon table objects derived from the NCBI information at
<a href="ftp://ftp.ncbi.nlm.nih.gov/entrez/misc/data/gc.prt"><span style="font-family:monospace">ftp://ftp.ncbi.nlm.nih.gov/entrez/misc/data/gc.prt</span></a>, also shown on
<a href="http://www.ncbi.nlm.nih.gov/Taxonomy/Utils/wprintgc.cgi"><span style="font-family:monospace">http://www.ncbi.nlm.nih.gov/Taxonomy/Utils/wprintgc.cgi</span></a> in a much more readable layout.</p><p>As before, let&#X2019;s just focus on two choices: the Standard translation table, and the
translation table for Vertebrate Mitochondrial DNA. </p><pre class="verbatim">&gt;&gt;&gt; from Bio.Data import CodonTable
&gt;&gt;&gt; standard_table = CodonTable.unambiguous_dna_by_name["Standard"]
&gt;&gt;&gt; mito_table = CodonTable.unambiguous_dna_by_name["Vertebrate Mitochondrial"]
</pre><p>Alternatively, these tables are labeled with ID numbers 1 and 2, respectively:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Data import CodonTable
&gt;&gt;&gt; standard_table = CodonTable.unambiguous_dna_by_id[1]
&gt;&gt;&gt; mito_table = CodonTable.unambiguous_dna_by_id[2]
</pre><p>You can compare the actual tables visually by printing them:
</p><pre class="verbatim">&gt;&gt;&gt; print(standard_table)
Table 1 Standard, SGC0

  |  T      |  C      |  A      |  G      |
--+---------+---------+---------+---------+--
T | TTT F   | TCT S   | TAT Y   | TGT C   | T
T | TTC F   | TCC S   | TAC Y   | TGC C   | C
T | TTA L   | TCA S   | TAA Stop| TGA Stop| A
T | TTG L(s)| TCG S   | TAG Stop| TGG W   | G
--+---------+---------+---------+---------+--
C | CTT L   | CCT P   | CAT H   | CGT R   | T
C | CTC L   | CCC P   | CAC H   | CGC R   | C
C | CTA L   | CCA P   | CAA Q   | CGA R   | A
C | CTG L(s)| CCG P   | CAG Q   | CGG R   | G
--+---------+---------+---------+---------+--
A | ATT I   | ACT T   | AAT N   | AGT S   | T
A | ATC I   | ACC T   | AAC N   | AGC S   | C
A | ATA I   | ACA T   | AAA K   | AGA R   | A
A | ATG M(s)| ACG T   | AAG K   | AGG R   | G
--+---------+---------+---------+---------+--
G | GTT V   | GCT A   | GAT D   | GGT G   | T
G | GTC V   | GCC A   | GAC D   | GGC G   | C
G | GTA V   | GCA A   | GAA E   | GGA G   | A
G | GTG V   | GCG A   | GAG E   | GGG G   | G
--+---------+---------+---------+---------+--
</pre><p>and:
</p><pre class="verbatim">&gt;&gt;&gt; print(mito_table)
Table 2 Vertebrate Mitochondrial, SGC1

  |  T      |  C      |  A      |  G      |
--+---------+---------+---------+---------+--
T | TTT F   | TCT S   | TAT Y   | TGT C   | T
T | TTC F   | TCC S   | TAC Y   | TGC C   | C
T | TTA L   | TCA S   | TAA Stop| TGA W   | A
T | TTG L   | TCG S   | TAG Stop| TGG W   | G
--+---------+---------+---------+---------+--
C | CTT L   | CCT P   | CAT H   | CGT R   | T
C | CTC L   | CCC P   | CAC H   | CGC R   | C
C | CTA L   | CCA P   | CAA Q   | CGA R   | A
C | CTG L   | CCG P   | CAG Q   | CGG R   | G
--+---------+---------+---------+---------+--
A | ATT I(s)| ACT T   | AAT N   | AGT S   | T
A | ATC I(s)| ACC T   | AAC N   | AGC S   | C
A | ATA M(s)| ACA T   | AAA K   | AGA Stop| A
A | ATG M(s)| ACG T   | AAG K   | AGG Stop| G
--+---------+---------+---------+---------+--
G | GTT V   | GCT A   | GAT D   | GGT G   | T
G | GTC V   | GCC A   | GAC D   | GGC G   | C
G | GTA V   | GCA A   | GAA E   | GGA G   | A
G | GTG V(s)| GCG A   | GAG E   | GGG G   | G
--+---------+---------+---------+---------+--
</pre><p>You may find these following properties useful &#X2013; for example if you are trying
to do your own gene finding:
</p><pre class="verbatim">&gt;&gt;&gt; mito_table.stop_codons
['TAA', 'TAG', 'AGA', 'AGG']
&gt;&gt;&gt; mito_table.start_codons
['ATT', 'ATC', 'ATA', 'ATG', 'GTG']
&gt;&gt;&gt; mito_table.forward_table["ACG"]
'T'
</pre>
<!--TOC section id="sec28" Comparing Seq objects-->
<h2 id="sec28" class="section">3.11&#XA0;&#XA0;Comparing Seq objects</h2><!--SEC END --><p>
<a id="sec:seq-comparison"></a></p><p>Sequence comparison is actually a very complicated topic, and there is no easy
way to decide if two sequences are equal. The basic problem is the meaning of
the letters in a sequence are context dependent - the letter &#X201C;A&#X201D; could be part
of a DNA, RNA or protein sequence. Biopython uses alphabet objects as part of
each <code>Seq</code> object to try and capture this information - so comparing two
<code>Seq</code> objects means considering both the sequence strings <em>and</em> the
alphabets.</p><p>For example, you might argue that the two DNA <code>Seq</code> objects
<span style="font-family:monospace">Seq("ACGT", IUPAC.unambiguous_dna)</span> and
<span style="font-family:monospace">Seq("ACGT", IUPAC.ambiguous_dna)</span> should be equal, even though
they do have different alphabets. Depending on the context this could be
important.</p><p>This gets worse &#X2013; suppose you think <span style="font-family:monospace">Seq("ACGT",
IUPAC.unambiguous_dna)</span> and <span style="font-family:monospace">Seq("ACGT")</span> (i.e. the default generic
alphabet) should be equal. Then, logically, <span style="font-family:monospace">Seq("ACGT", IUPAC.protein)</span>
and <span style="font-family:monospace">Seq("ACGT")</span> should also be equal. Now, in logic if <span style="font-style:italic">A</span>=<span style="font-style:italic">B</span> and
<span style="font-style:italic">B</span>=<span style="font-style:italic">C</span>, by transitivity we expect <span style="font-style:italic">A</span>=<span style="font-style:italic">C</span>. So for logical consistency we&#X2019;d
require <span style="font-family:monospace">Seq("ACGT", IUPAC.unambiguous_dna)</span> and <span style="font-family:monospace">Seq("ACGT",
IUPAC.protein)</span> to be equal &#X2013; which most people would agree is just not right.
This transitivity problem would also have implications for using <code>Seq</code>
objects as Python dictionary keys.</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; seq1 = Seq("ACGT", IUPAC.unambiguous_dna)
&gt;&gt;&gt; seq2 = Seq("ACGT", IUPAC.unambiguous_dna)
</pre><p>So, what does Biopython do? Well, the equality test is the default for Python
objects &#X2013; it tests to see if they are the same object in memory. This is a
very strict test:
</p><pre class="verbatim">&gt;&gt;&gt; seq1 == seq2
False
&gt;&gt;&gt; seq1 == seq1
True
</pre><p>If you actually want to do this, you can be more explicit by using the Python
<code>id</code> function,
</p><pre class="verbatim">&gt;&gt;&gt; id(seq1) == id(seq2)
False
&gt;&gt;&gt; id(seq1) == id(seq1)
True
</pre><p>Now, in every day use, your sequences will probably all have the same
alphabet, or at least all be the same type of sequence (all DNA, all RNA, or
all protein). What you probably want is to just compare the sequences as
strings &#X2013; so do this explicitly:
</p><pre class="verbatim">&gt;&gt;&gt; str(seq1) == str(seq2)
True
&gt;&gt;&gt; str(seq1) == str(seq1)
True
</pre><p>As an extension to this, while you can use a Python dictionary with
<code>Seq</code> objects as keys, it is generally more useful to use the sequence a
string for the key. See also Section&#XA0;<a href="#sec%3Aseq-to-string">3.4</a>.</p>
<!--TOC section id="sec29" MutableSeq objects-->
<h2 id="sec29" class="section">3.12&#XA0;&#XA0;MutableSeq objects</h2><!--SEC END --><p>
<a id="sec:mutable-seq"></a></p><p>Just like the normal Python string, the <code>Seq</code> object is &#X201C;read only&#X201D;, or in Python terminology, immutable. Apart from wanting the <code>Seq</code> object to act like a string, this is also a useful default since in many biological applications you want to ensure you are not changing your sequence data:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; my_seq = Seq("GCCATTGTAATGGGCCGCTGAAAGGGTGCCCGA", IUPAC.unambiguous_dna)
</pre><p>Observe what happens if you try to edit the sequence:
</p><pre class="verbatim">&gt;&gt;&gt; my_seq[5] = "G"
Traceback (most recent call last):
...
TypeError: 'Seq' object does not support item assignment
</pre><p>However, you can convert it into a mutable sequence (a <code>MutableSeq</code> object) and do pretty much anything you want with it:</p><pre class="verbatim">&gt;&gt;&gt; mutable_seq = my_seq.tomutable()
&gt;&gt;&gt; mutable_seq
MutableSeq('GCCATTGTAATGGGCCGCTGAAAGGGTGCCCGA', IUPACUnambiguousDNA())
</pre><p>Alternatively, you can create a <code>MutableSeq</code> object directly from a string:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import MutableSeq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; mutable_seq = MutableSeq("GCCATTGTAATGGGCCGCTGAAAGGGTGCCCGA", IUPAC.unambiguous_dna)
</pre><p>Either way will give you a sequence object which can be changed:
</p><pre class="verbatim">&gt;&gt;&gt; mutable_seq
MutableSeq('GCCATTGTAATGGGCCGCTGAAAGGGTGCCCGA', IUPACUnambiguousDNA())
&gt;&gt;&gt; mutable_seq[5] = "C"
&gt;&gt;&gt; mutable_seq
MutableSeq('GCCATCGTAATGGGCCGCTGAAAGGGTGCCCGA', IUPACUnambiguousDNA())
&gt;&gt;&gt; mutable_seq.remove("T")
&gt;&gt;&gt; mutable_seq
MutableSeq('GCCACGTAATGGGCCGCTGAAAGGGTGCCCGA', IUPACUnambiguousDNA())
&gt;&gt;&gt; mutable_seq.reverse()
&gt;&gt;&gt; mutable_seq
MutableSeq('AGCCCGTGGGAAAGTCGCCGGGTAATGCACCG', IUPACUnambiguousDNA())
</pre><p>Do note that unlike the <code>Seq</code> object, the <code>MutableSeq</code> object&#X2019;s methods like <code>reverse_complement()</code> and <code>reverse()</code> act in-situ!</p><p>An important technical difference between mutable and immutable objects in Python means that you can&#X2019;t use a <code>MutableSeq</code> object as a dictionary key, but you can use a Python string or a <code>Seq</code> object in this way.</p><p>Once you have finished editing your a <code>MutableSeq</code> object, it&#X2019;s easy to get back to a read-only <code>Seq</code> object should you need to:</p><pre class="verbatim">&gt;&gt;&gt; new_seq = mutable_seq.toseq()
&gt;&gt;&gt; new_seq
Seq('AGCCCGTGGGAAAGTCGCCGGGTAATGCACCG', IUPACUnambiguousDNA())
</pre><p>You can also get a string from a <code>MutableSeq</code> object just like from a <code>Seq</code> object (Section&#XA0;<a href="#sec%3Aseq-to-string">3.4</a>).</p>
<!--TOC section id="sec30" UnknownSeq objects-->
<h2 id="sec30" class="section">3.13&#XA0;&#XA0;UnknownSeq objects</h2><!--SEC END --><p>
The <code>UnknownSeq</code> object is a subclass of the basic <code>Seq</code> object
and its purpose is to represent a
sequence where we know the length, but not the actual letters making it up.
You could of course use a normal <code>Seq</code> object in this situation, but it wastes
rather a lot of memory to hold a string of a million &#X201C;N&#X201D; characters when you could
just store a single letter &#X201C;N&#X201D; and the desired length as an integer.</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import UnknownSeq
&gt;&gt;&gt; unk = UnknownSeq(20)
&gt;&gt;&gt; unk
UnknownSeq(20, alphabet = Alphabet(), character = '?')
&gt;&gt;&gt; print(unk)
????????????????????
&gt;&gt;&gt; len(unk)
20
</pre><p>You can of course specify an alphabet, meaning for nucleotide sequences
the letter defaults to &#X201C;N&#X201D; and for proteins &#X201C;X&#X201D;, rather than just &#X201C;?&#X201D;.</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import UnknownSeq
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; unk_dna = UnknownSeq(20, alphabet=IUPAC.ambiguous_dna)
&gt;&gt;&gt; unk_dna
UnknownSeq(20, alphabet = IUPACAmbiguousDNA(), character = 'N')
&gt;&gt;&gt; print(unk_dna)
NNNNNNNNNNNNNNNNNNNN
</pre><p>You can use all the usual <code>Seq</code> object methods too, note these give back
memory saving <code>UnknownSeq</code> objects where appropriate as you might expect:</p><pre class="verbatim">&gt;&gt;&gt; unk_dna
UnknownSeq(20, alphabet = IUPACAmbiguousDNA(), character = 'N')
&gt;&gt;&gt; unk_dna.complement()
UnknownSeq(20, alphabet = IUPACAmbiguousDNA(), character = 'N')
&gt;&gt;&gt; unk_dna.reverse_complement()
UnknownSeq(20, alphabet = IUPACAmbiguousDNA(), character = 'N')
&gt;&gt;&gt; unk_dna.transcribe()
UnknownSeq(20, alphabet = IUPACAmbiguousRNA(), character = 'N')
&gt;&gt;&gt; unk_protein = unk_dna.translate()
&gt;&gt;&gt; unk_protein
UnknownSeq(6, alphabet = ProteinAlphabet(), character = 'X')
&gt;&gt;&gt; print(unk_protein)
XXXXXX
&gt;&gt;&gt; len(unk_protein)
6
</pre><p>You may be able to find a use for the <code>UnknownSeq</code> object in your own
code, but it is more likely that you will first come across them in a
<code>SeqRecord</code> object created by <code>Bio.SeqIO</code>
(see Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>).
Some sequence file formats don&#X2019;t always include the actual sequence, for
example GenBank and EMBL files may include a list of features but for the
sequence just present the contig information. Alternatively, the QUAL files
used in sequencing work hold quality scores but they <em>never</em> contain a
sequence &#X2013; instead there is a partner FASTA file which <em>does</em> have the
sequence.</p>
<!--TOC section id="sec31" Working with strings directly-->
<h2 id="sec31" class="section">3.14&#XA0;&#XA0;Working with strings directly</h2><!--SEC END --><p>
<a id="sec:seq-module-functions"></a>
To close this chapter, for those you who <em>really</em> don&#X2019;t want to use the sequence
objects (or who prefer a functional programming style to an object orientated one),
there are module level functions in <code>Bio.Seq</code> will accept plain Python strings,
<code>Seq</code> objects (including <code>UnknownSeq</code> objects) or <code>MutableSeq</code> objects:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import reverse_complement, transcribe, back_transcribe, translate
&gt;&gt;&gt; my_string = "GCTGTTATGGGTCGTTGGAAGGGTGGTCGTGCTGCTGGTTAG"
&gt;&gt;&gt; reverse_complement(my_string)
'CTAACCAGCAGCACGACCACCCTTCCAACGACCCATAACAGC'
&gt;&gt;&gt; transcribe(my_string)
'GCUGUUAUGGGUCGUUGGAAGGGUGGUCGUGCUGCUGGUUAG'
&gt;&gt;&gt; back_transcribe(my_string)
'GCTGTTATGGGTCGTTGGAAGGGTGGTCGTGCTGCTGGTTAG'
&gt;&gt;&gt; translate(my_string)
'AVMGRWKGGRAAG*'
</pre><p>You are, however, encouraged to work with <code>Seq</code> objects by default.</p>
<!--TOC chapter id="sec32" Sequence annotation objects-->
<h1 id="sec32" class="chapter">Chapter&#XA0;4&#XA0;&#XA0;Sequence annotation objects</h1><!--SEC END --><p>
<a id="chapter:SeqRecord"></a></p><p>Chapter&#XA0;<a href="#chapter%3ABio.Seq">3</a> introduced the sequence classes. Immediately &#X201C;above&#X201D; the <code>Seq</code> class is the Sequence Record or <code>SeqRecord</code> class, defined in the <code>Bio.SeqRecord</code> module. This class allows higher level features such as identifiers and features (as <code>SeqFeature</code> objects) to be associated with the sequence, and is used throughout the sequence input/output interface <code>Bio.SeqIO</code> described fully in Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>.</p><p>If you are only going to be working with simple data like FASTA files, you can probably skip this chapter
for now. If on the other hand you are going to be using richly annotated sequence data, say from GenBank
or EMBL files, this information is quite important.</p><p>While this chapter should cover most things to do with the <code>SeqRecord</code> and <code>SeqFeature</code> objects in this chapter, you may also want to read the <code>SeqRecord</code> wiki page (<a href="http://biopython.org/wiki/SeqRecord"><span style="font-family:monospace">http://biopython.org/wiki/SeqRecord</span></a>), and the built in documentation (also online &#X2013; <a href="http://biopython.org/DIST/docs/api/Bio.SeqRecord.SeqRecord-class.html">SeqRecord</a> and <a href="http://biopython.org/DIST/docs/api/Bio.SeqFeature.SeqFeature-class.html">SeqFeature</a>):</p><pre class="verbatim">&gt;&gt;&gt; from Bio.SeqRecord import SeqRecord
&gt;&gt;&gt; help(SeqRecord)
...
</pre>
<!--TOC section id="sec33" The SeqRecord object-->
<h2 id="sec33" class="section">4.1&#XA0;&#XA0;The SeqRecord object</h2><!--SEC END --><p>
<a id="sec:SeqRecord"></a></p><p>The <code>SeqRecord</code> (Sequence Record) class is defined in the <code>Bio.SeqRecord</code> module. This class allows higher level features such as identifiers and features to be associated with a sequence (see Chapter&#XA0;<a href="#chapter%3ABio.Seq">3</a>), and is the basic data type for the <code>Bio.SeqIO</code> sequence input/output interface (see Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>).</p><p>The <code>SeqRecord</code> class itself is quite simple, and offers the following information as attributes:</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold">.seq</span></dt><dd class="dd-description"> &#X2013; The sequence itself, typically a <code>Seq</code> object.</dd><dt class="dt-description"><span style="font-weight:bold">.id</span></dt><dd class="dd-description"> &#X2013; The primary ID used to identify the sequence &#X2013; a string. In most cases this is something like an accession number.</dd><dt class="dt-description"><span style="font-weight:bold">.name</span></dt><dd class="dd-description"> &#X2013; A &#X201C;common&#X201D; name/id for the sequence &#X2013; a string. In some cases this will be the same as the accession number, but it could also be a clone name. I think of this as being analogous to the LOCUS id in a GenBank record.</dd><dt class="dt-description"><span style="font-weight:bold">.description</span></dt><dd class="dd-description"> &#X2013; A human readable description or expressive name for the sequence &#X2013; a string.</dd><dt class="dt-description"><span style="font-weight:bold">.letter_annotations</span></dt><dd class="dd-description"> &#X2013; Holds per-letter-annotations using a (restricted) dictionary of additional information about the letters in the sequence. The keys are the name of the information, and the information is contained in the value as a Python sequence (i.e. a list, tuple or string) with the same length as the sequence itself. This is often used for quality scores (e.g. Section&#XA0;<a href="#sec%3AFASTQ-filtering-example">18.1.6</a>) or secondary structure information (e.g. from Stockholm/PFAM alignment files).</dd><dt class="dt-description"><span style="font-weight:bold">.annotations</span></dt><dd class="dd-description"> &#X2013; A dictionary of additional information about the sequence. The keys are the name of the information, and the information is contained in the value. This allows the addition of more &#X201C;unstructured&#X201D; information to the sequence.</dd><dt class="dt-description"><span style="font-weight:bold">.features</span></dt><dd class="dd-description"> &#X2013; A list of <code>SeqFeature</code> objects with more structured information about the features on a sequence (e.g. position of genes on a genome, or domains on a protein sequence). The structure of sequence features is described below in Section&#XA0;<a href="#sec%3Aseq_features">4.3</a>.</dd><dt class="dt-description"><span style="font-weight:bold">.dbxrefs</span></dt><dd class="dd-description"> - A list of database cross-references as strings.
</dd></dl>
<!--TOC section id="sec34" Creating a SeqRecord-->
<h2 id="sec34" class="section">4.2&#XA0;&#XA0;Creating a SeqRecord</h2><!--SEC END --><p>Using a <code>SeqRecord</code> object is not very complicated, since all of the
information is presented as attributes of the class. Usually you won&#X2019;t create
a <code>SeqRecord</code> &#X201C;by hand&#X201D;, but instead use <code>Bio.SeqIO</code> to read in a
sequence file for you (see Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a> and the examples
below). However, creating <code>SeqRecord</code> can be quite simple.</p>
<!--TOC subsection id="sec35" SeqRecord objects from scratch-->
<h3 id="sec35" class="subsection">4.2.1&#XA0;&#XA0;SeqRecord objects from scratch</h3><!--SEC END --><p>To create a <code>SeqRecord</code> at a minimum you just need a <code>Seq</code> object:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; simple_seq = Seq("GATC")
&gt;&gt;&gt; from Bio.SeqRecord import SeqRecord
&gt;&gt;&gt; simple_seq_r = SeqRecord(simple_seq)
</pre><p>Additionally, you can also pass the id, name and description to the initialization function, but if not they will be set as strings indicating they are unknown, and can be modified subsequently:</p><pre class="verbatim">&gt;&gt;&gt; simple_seq_r.id
'&lt;unknown id&gt;'
&gt;&gt;&gt; simple_seq_r.id = "AC12345"
&gt;&gt;&gt; simple_seq_r.description = "Made up sequence I wish I could write a paper about"
&gt;&gt;&gt; print(simple_seq_r.description)
Made up sequence I wish I could write a paper about
&gt;&gt;&gt; simple_seq_r.seq
Seq('GATC', Alphabet())
</pre><p>Including an identifier is very important if you want to output your <code>SeqRecord</code> to a file. You would normally include this when creating the object:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; simple_seq = Seq("GATC")
&gt;&gt;&gt; from Bio.SeqRecord import SeqRecord
&gt;&gt;&gt; simple_seq_r = SeqRecord(simple_seq, id="AC12345")
</pre><p>As mentioned above, the <code>SeqRecord</code> has an dictionary attribute <code>annotations</code>. This is used
for any miscellaneous annotations that doesn&#X2019;t fit under one of the other more specific attributes.
Adding annotations is easy, and just involves dealing directly with the annotation dictionary:</p><pre class="verbatim">&gt;&gt;&gt; simple_seq_r.annotations["evidence"] = "None. I just made it up."
&gt;&gt;&gt; print(simple_seq_r.annotations)
{'evidence': 'None. I just made it up.'}
&gt;&gt;&gt; print(simple_seq_r.annotations["evidence"])
None. I just made it up.
</pre><p>Working with per-letter-annotations is similar, <code>letter_annotations</code> is a
dictionary like attribute which will let you assign any Python sequence (i.e.
a string, list or tuple) which has the same length as the sequence:</p><pre class="verbatim">&gt;&gt;&gt; simple_seq_r.letter_annotations["phred_quality"] = [40, 40, 38, 30]
&gt;&gt;&gt; print(simple_seq_r.letter_annotations)
{'phred_quality': [40, 40, 38, 30]}
&gt;&gt;&gt; print(simple_seq_r.letter_annotations["phred_quality"])
[40, 40, 38, 30]
</pre><p>The <code>dbxrefs</code> and <code>features</code> attributes are just Python lists, and
should be used to store strings and <code>SeqFeature</code> objects (discussed later
in this chapter) respectively.</p>
<!--TOC subsection id="sec36" SeqRecord objects from FASTA files-->
<h3 id="sec36" class="subsection">4.2.2&#XA0;&#XA0;SeqRecord objects from FASTA files</h3><!--SEC END --><p>This example uses a fairly large FASTA file containing the whole sequence for <span style="font-style:italic">Yersinia pestis biovar Microtus</span> str. 91001 plasmid pPCP1, originally downloaded from the NCBI. This file is included with the Biopython unit tests under the GenBank folder, or online <a href="http://biopython.org/SRC/biopython/Tests/GenBank/NC_005816.fna"><span style="font-family:monospace">NC_005816.fna</span></a> from our website.</p><p>The file starts like this - and you can check there is only one record present (i.e. only one line starting with a greater than symbol):</p><pre class="verbatim">&gt;gi|45478711|ref|NC_005816.1| Yersinia pestis biovar Microtus ... pPCP1, complete sequence
TGTAACGAACGGTGCAATAGTGATCCACACCCAACGCCTGAAATCAGATCCAGGGGGTAATCTGCTCTCC
...
</pre><p>Back in Chapter&#XA0;<a href="#chapter%3Aquick-start">2</a> you will have seen the function <code>Bio.SeqIO.parse(...)</code>
used to loop over all the records in a file as <code>SeqRecord</code> objects. The <code>Bio.SeqIO</code> module
has a sister function for use on files which contain just one record which we&#X2019;ll use here (see Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a> for details):</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; record = SeqIO.read("NC_005816.fna", "fasta")
&gt;&gt;&gt; record
SeqRecord(seq=Seq('TGTAACGAACGGTGCAATAGTGATCCACACCCAACGCCTGAAATCAGATCCAGG...CTG',
SingleLetterAlphabet()), id='gi|45478711|ref|NC_005816.1|', name='gi|45478711|ref|NC_005816.1|',
description='gi|45478711|ref|NC_005816.1| Yersinia pestis biovar Microtus ... sequence',
dbxrefs=[])
</pre><p>Now, let&#X2019;s have a look at the key attributes of this <code>SeqRecord</code>
individually &#X2013; starting with the <code>seq</code> attribute which gives you a
<code>Seq</code> object:</p><pre class="verbatim">&gt;&gt;&gt; record.seq
Seq('TGTAACGAACGGTGCAATAGTGATCCACACCCAACGCCTGAAATCAGATCCAGG...CTG', SingleLetterAlphabet())
</pre><p>Here <code>Bio.SeqIO</code> has defaulted to a generic alphabet, rather
than guessing that this is DNA. If you know in advance what kind of sequence
your FASTA file contains, you can tell <code>Bio.SeqIO</code> which alphabet to use
(see Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>).</p><p>Next, the identifiers and description:</p><pre class="verbatim">&gt;&gt;&gt; record.id
'gi|45478711|ref|NC_005816.1|'
&gt;&gt;&gt; record.name
'gi|45478711|ref|NC_005816.1|'
&gt;&gt;&gt; record.description
'gi|45478711|ref|NC_005816.1| Yersinia pestis biovar Microtus ... pPCP1, complete sequence'
</pre><p>As you can see above, the first word of the FASTA record&#X2019;s title line (after
removing the greater than symbol) is used for both the <code>id</code> and
<code>name</code> attributes. The whole title line (after removing the greater than
symbol) is used for the record description. This is deliberate, partly for
backwards compatibility reasons, but it also makes sense if you have a FASTA
file like this:</p><pre class="verbatim">&gt;Yersinia pestis biovar Microtus str. 91001 plasmid pPCP1
TGTAACGAACGGTGCAATAGTGATCCACACCCAACGCCTGAAATCAGATCCAGGGGGTAATCTGCTCTCC
...
</pre><p>Note that none of the other annotation attributes get populated when reading a
FASTA file:</p><pre class="verbatim">&gt;&gt;&gt; record.dbxrefs
[]
&gt;&gt;&gt; record.annotations
{}
&gt;&gt;&gt; record.letter_annotations
{}
&gt;&gt;&gt; record.features
[]
</pre><p>In this case our example FASTA file was from the NCBI, and they have a fairly well defined set of conventions for formatting their FASTA lines. This means it would be possible to parse this information and extract the GI number and accession for example. However, FASTA files from other sources vary, so this isn&#X2019;t possible in general.</p>
<!--TOC subsection id="sec37" SeqRecord objects from GenBank files-->
<h3 id="sec37" class="subsection">4.2.3&#XA0;&#XA0;SeqRecord objects from GenBank files</h3><!--SEC END --><p>As in the previous example, we&#X2019;re going to look at the whole sequence for <span style="font-style:italic">Yersinia pestis biovar Microtus</span> str. 91001 plasmid pPCP1, originally downloaded from the NCBI, but this time as a GenBank file.
Again, this file is included with the Biopython unit tests under the GenBank folder, or online <a href="http://biopython.org/SRC/biopython/Tests/GenBank/NC_005816.gb"><span style="font-family:monospace">NC_005816.gb</span></a> from our website.</p><p>This file contains a single record (i.e. only one LOCUS line) and starts:
</p><pre class="verbatim">LOCUS       NC_005816               9609 bp    DNA     circular BCT 21-JUL-2008
DEFINITION  Yersinia pestis biovar Microtus str. 91001 plasmid pPCP1, complete
            sequence.
ACCESSION   NC_005816
VERSION     NC_005816.1  GI:45478711
PROJECT     GenomeProject:10638
...
</pre><p>Again, we&#X2019;ll use <code>Bio.SeqIO</code> to read this file in, and the code is almost identical to that for used above for the FASTA file (see Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a> for details):</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; record = SeqIO.read("NC_005816.gb", "genbank")
&gt;&gt;&gt; record
SeqRecord(seq=Seq('TGTAACGAACGGTGCAATAGTGATCCACACCCAACGCCTGAAATCAGATCCAGG...CTG',
IUPACAmbiguousDNA()), id='NC_005816.1', name='NC_005816',
description='Yersinia pestis biovar Microtus str. 91001 plasmid pPCP1, complete sequence.',
dbxrefs=['Project:10638'])
</pre><p>You should be able to spot some differences already! But taking the attributes individually,
the sequence string is the same as before, but this time <code>Bio.SeqIO</code> has been able to automatically assign a more specific alphabet (see Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a> for details):</p><pre class="verbatim">&gt;&gt;&gt; record.seq
Seq('TGTAACGAACGGTGCAATAGTGATCCACACCCAACGCCTGAAATCAGATCCAGG...CTG', IUPACAmbiguousDNA())
</pre><p>The <code>name</code> comes from the LOCUS line, while the <code>id</code> includes the version suffix.
The description comes from the DEFINITION line:</p><pre class="verbatim">&gt;&gt;&gt; record.id
'NC_005816.1'
&gt;&gt;&gt; record.name
'NC_005816'
&gt;&gt;&gt; record.description
'Yersinia pestis biovar Microtus str. 91001 plasmid pPCP1, complete sequence.'
</pre><p>GenBank files don&#X2019;t have any per-letter annotations:</p><pre class="verbatim">&gt;&gt;&gt; record.letter_annotations
{}
</pre><p>Most of the annotations information gets recorded in the <code>annotations</code> dictionary, for example:</p><pre class="verbatim">&gt;&gt;&gt; len(record.annotations)
11
&gt;&gt;&gt; record.annotations["source"]
'Yersinia pestis biovar Microtus str. 91001'
</pre><p>The <code>dbxrefs</code> list gets populated from any PROJECT or DBLINK lines:</p><pre class="verbatim">&gt;&gt;&gt; record.dbxrefs
['Project:10638']
</pre><p>Finally, and perhaps most interestingly, all the entries in the features table (e.g. the genes or CDS features) get recorded as <code>SeqFeature</code> objects in the <code>features</code> list.</p><pre class="verbatim">&gt;&gt;&gt; len(record.features)
29
</pre><p>We&#X2019;ll talk about <code>SeqFeature</code> objects next, in
Section&#XA0;<a href="#sec%3Aseq_features">4.3</a>.</p>
<!--TOC section id="sec38" Feature, location and position objects-->
<h2 id="sec38" class="section">4.3&#XA0;&#XA0;Feature, location and position objects</h2><!--SEC END --><p>
<a id="sec:seq_features"></a></p>
<!--TOC subsection id="sec39" SeqFeature objects-->
<h3 id="sec39" class="subsection">4.3.1&#XA0;&#XA0;SeqFeature objects</h3><!--SEC END --><p>Sequence features are an essential part of describing a sequence. Once you get beyond the sequence itself, you need some way to organize and easily get at the more &#X201C;abstract&#X201D; information that is known about the sequence. While it is probably impossible to develop a general sequence feature class that will cover everything, the Biopython <code>SeqFeature</code> class attempts to encapsulate as much of the information about the sequence as possible. The design is heavily based on the GenBank/EMBL feature tables, so if you understand how they look, you&#X2019;ll probably have an easier time grasping the structure of the Biopython classes.</p><p>The key idea about each <code>SeqFeature</code> object is to describe a region on a parent sequence, typically a <code>SeqRecord</code> object. That region is described with a location object, typically a range between two positions (see Section&#XA0;<a href="#sec%3Alocations">4.3.2</a> below).</p><p>The <code>SeqFeature</code> class has a number of attributes, so first we&#X2019;ll list them and their general features, and then later in the chapter work through examples to show how this applies to a real life example. The attributes of a SeqFeature are:</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold">.type</span></dt><dd class="dd-description"> &#X2013; This is a textual description of the type of feature (for instance, this will be something like &#X2018;CDS&#X2019; or &#X2018;gene&#X2019;).</dd><dt class="dt-description"><span style="font-weight:bold">.location</span></dt><dd class="dd-description"> &#X2013; The location of the <code>SeqFeature</code> on the sequence
that you are dealing with, see Section&#XA0;<a href="#sec%3Alocations">4.3.2</a> below. The
<code>SeqFeature</code> delegates much of its functionality to the location
object, and includes a number of shortcut attributes for properties
of the location:<dl class="description"><dt class="dt-description">
<span style="font-weight:bold">.ref</span></dt><dd class="dd-description"> &#X2013; shorthand for <code>.location.ref</code> &#X2013; any (different)
reference sequence the location is referring to. Usually just None.</dd><dt class="dt-description"><span style="font-weight:bold">.ref_db</span></dt><dd class="dd-description"> &#X2013; shorthand for <code>.location.ref_db</code> &#X2013; specifies
the database any identifier in <code>.ref</code> refers to. Usually just None.</dd><dt class="dt-description"><span style="font-weight:bold">.strand</span></dt><dd class="dd-description"> &#X2013; shorthand for <code>.location.strand</code> &#X2013; the strand on
the sequence that the feature is located on. For double stranded nucleotide
sequence this may either be 1 for the top strand, &#X2212;1 for the bottom
strand, 0 if the strand is important but is unknown, or <span style="font-family:monospace">None</span>
if it doesn&#X2019;t matter. This is None for proteins, or single stranded sequences.
</dd></dl></dd><dt class="dt-description"><span style="font-weight:bold">.qualifiers</span></dt><dd class="dd-description"> &#X2013; This is a Python dictionary of additional information about the feature. The key is some kind of terse one-word description of what the information contained in the value is about, and the value is the actual information. For example, a common key for a qualifier might be &#X201C;evidence&#X201D; and the value might be &#X201C;computational (non-experimental).&#X201D; This is just a way to let the person who is looking at the feature know that it has not be experimentally (i.&#XA0;e.&#XA0;in a wet lab) confirmed. Note that other the value will be a list of strings (even when there is only one string). This is a reflection of the feature tables in GenBank/EMBL files.</dd><dt class="dt-description"><span style="font-weight:bold">.sub_features</span></dt><dd class="dd-description"> &#X2013; This used to be used to represent features with complicated locations like &#X2018;joins&#X2019; in GenBank/EMBL files. This has been deprecated with the introduction of the <code>CompoundLocation</code> object, and should now be ignored.</dd></dl>
<!--TOC subsection id="sec40" Positions and locations-->
<h3 id="sec40" class="subsection">4.3.2&#XA0;&#XA0;Positions and locations</h3><!--SEC END --><p>
<a id="sec:locations"></a></p><p>The key idea about each <code>SeqFeature</code> object is to describe a
region on a parent sequence, for which we use a location object,
typically describing a range between two positions. Two try to
clarify the terminology we&#X2019;re using:</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold">position</span></dt><dd class="dd-description"> &#X2013; This refers to a single position on a sequence,
which may be fuzzy or not. For instance, 5, 20, <code>&lt;100</code> and
<code>&gt;200</code> are all positions.</dd><dt class="dt-description"><span style="font-weight:bold">location</span></dt><dd class="dd-description"> &#X2013; A location is region of sequence bounded by
some positions. For instance 5..20 (i.&#XA0;e.&#XA0;5 to 20) is a location.
</dd></dl><p>I just mention this because sometimes I get confused between the two.</p>
<!--TOC subsubsection id="sec41" FeatureLocation object-->
<h4 id="sec41" class="subsubsection">4.3.2.1&#XA0;&#XA0;FeatureLocation object</h4><!--SEC END --><p>Unless you work with eukaryotic genes, most <code>SeqFeature</code> locations are
extremely simple - you just need start and end coordinates and a strand.
That&#X2019;s essentially all the basic <code>FeatureLocation</code> object does.</p><p>In practise of course, things can be more complicated. First of all
we have to handle compound locations made up of several regions.
Secondly, the positions themselves may be fuzzy (inexact).</p>
<!--TOC subsubsection id="sec42" CompoundLocation object-->
<h4 id="sec42" class="subsubsection">4.3.2.2&#XA0;&#XA0;CompoundLocation object</h4><!--SEC END --><p>Biopython 1.62 introduced the <code>CompoundLocation</code> as part of
a restructuring of how complex locations made up of multiple regions
are represented.
The main usage is for handling &#X2018;join&#X2019; locations in EMBL/GenBank files.</p>
<!--TOC subsubsection id="sec43" Fuzzy Positions-->
<h4 id="sec43" class="subsubsection">4.3.2.3&#XA0;&#XA0;Fuzzy Positions</h4><!--SEC END --><p>So far we&#X2019;ve only used simple positions. One complication in dealing
with feature locations comes in the positions themselves.
In biology many times things aren&#X2019;t entirely certain
(as much as us wet lab biologists try to make them certain!). For
instance, you might do a dinucleotide priming experiment and discover
that the start of mRNA transcript starts at one of two sites. This
is very useful information, but the complication comes in how to
represent this as a position. To help us deal with this, we have
the concept of fuzzy positions. Basically there are several types
of fuzzy positions, so we have five classes do deal with them:</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold">ExactPosition</span></dt><dd class="dd-description"> &#X2013; As its name suggests, this class represents a position which is specified as exact along the sequence. This is represented as just a number, and you can get the position by looking at the <code>position</code> attribute of the object.</dd><dt class="dt-description"><span style="font-weight:bold">BeforePosition</span></dt><dd class="dd-description"> &#X2013; This class represents a fuzzy position
that occurs prior to some specified site. In GenBank/EMBL notation,
this is represented as something like <code>`&lt;13'</code>, signifying that
the real position is located somewhere less than 13. To get
the specified upper boundary, look at the <code>position</code>
attribute of the object.</dd><dt class="dt-description"><span style="font-weight:bold">AfterPosition</span></dt><dd class="dd-description"> &#X2013; Contrary to <code>BeforePosition</code>, this
class represents a position that occurs after some specified site.
This is represented in GenBank as <code>`&gt;13'</code>, and like
<code>BeforePosition</code>, you get the boundary number by looking
at the <code>position</code> attribute of the object.</dd><dt class="dt-description"><span style="font-weight:bold">WithinPosition</span></dt><dd class="dd-description"> &#X2013; Occasionally used for GenBank/EMBL locations,
this class models a position which occurs somewhere between two
specified nucleotides. In GenBank/EMBL notation, this would be
represented as &#X2018;(1.5)&#X2019;, to represent that the position is somewhere
within the range 1 to 5. To get the information in this class you
have to look at two attributes. The <code>position</code> attribute
specifies the lower boundary of the range we are looking at, so in
our example case this would be one. The <code>extension</code> attribute
specifies the range to the higher boundary, so in this case it
would be 4. So <code>object.position</code> is the lower boundary and
<code>object.position + object.extension</code> is the upper boundary.</dd><dt class="dt-description"><span style="font-weight:bold">OneOfPosition</span></dt><dd class="dd-description"> &#X2013; Occasionally used for GenBank/EMBL locations,
this class deals with a position where several possible values exist,
for instance you could use this if the start codon was unclear and
there where two candidates for the start of the gene. Alternatively,
that might be handled explicitly as two related gene features.</dd><dt class="dt-description"><span style="font-weight:bold">UnknownPosition</span></dt><dd class="dd-description"> &#X2013; This class deals with a position of unknown
location. This is not used in GenBank/EMBL, but corresponds to the &#X2018;?&#X2019;
feature coordinate used in UniProt.</dd></dl><p>Here&#X2019;s an example where we create a location with fuzzy end points:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqFeature
&gt;&gt;&gt; start_pos = SeqFeature.AfterPosition(5)
&gt;&gt;&gt; end_pos = SeqFeature.BetweenPosition(9, left=8, right=9)
&gt;&gt;&gt; my_location = SeqFeature.FeatureLocation(start_pos, end_pos)
</pre><p>Note that the details of some of the fuzzy-locations changed in Biopython 1.59,
in particular for BetweenPosition and WithinPosition you must now make it explicit
which integer position should be used for slicing etc. For a start position this
is generally the lower (left) value, while for an end position this would generally
be the higher (right) value.</p><p>If you print out a <code>FeatureLocation</code> object, you can get a nice representation of the information:</p><pre class="verbatim">&gt;&gt;&gt; print(my_location)
[&gt;5:(8^9)]
</pre><p>We can access the fuzzy start and end positions using the start and end attributes of the location:</p><pre class="verbatim">&gt;&gt;&gt; my_location.start
AfterPosition(5)
&gt;&gt;&gt; print(my_location.start)
&gt;5
&gt;&gt;&gt; my_location.end
BetweenPosition(9, left=8, right=9)
&gt;&gt;&gt; print(my_location.end)
(8^9)
</pre><p>If you don&#X2019;t want to deal with fuzzy positions and just want numbers,
they are actually subclasses of integers so should work like integers:</p><pre class="verbatim">&gt;&gt;&gt; int(my_location.start)
5
&gt;&gt;&gt; int(my_location.end)
9
</pre><p>For compatibility with older versions of Biopython you can ask for the
<code>nofuzzy_start</code> and <code>nofuzzy_end</code> attributes of the location
which are plain integers:</p><pre class="verbatim">&gt;&gt;&gt; my_location.nofuzzy_start
5
&gt;&gt;&gt; my_location.nofuzzy_end
9
</pre><p>Notice that this just gives you back the position attributes of the fuzzy locations.</p><p>Similarly, to make it easy to create a position without worrying about fuzzy positions, you can just pass in numbers to the <code>FeaturePosition</code> constructors, and you&#X2019;ll get back out <code>ExactPosition</code> objects:</p><pre class="verbatim">&gt;&gt;&gt; exact_location = SeqFeature.FeatureLocation(5, 9)
&gt;&gt;&gt; print(exact_location)
[5:9]
&gt;&gt;&gt; exact_location.start
ExactPosition(5)
&gt;&gt;&gt; int(exact_location.start)
5
&gt;&gt;&gt; exact_location.nofuzzy_start
5
</pre><p>That is most of the nitty gritty about dealing with fuzzy positions in Biopython.
It has been designed so that dealing with fuzziness is not that much more
complicated than dealing with exact positions, and hopefully you find that true!</p>
<!--TOC subsubsection id="sec44" Location testing-->
<h4 id="sec44" class="subsubsection">4.3.2.4&#XA0;&#XA0;Location testing</h4><!--SEC END --><p>You can use the Python keyword <code>in</code> with a <code>SeqFeature</code> or location
object to see if the base/residue for a parent coordinate is within the
feature/location or not.</p><p>For example, suppose you have a SNP of interest and you want to know which
features this SNP is within, and lets suppose this SNP is at index 4350
(Python counting!). Here is a simple brute force solution where we just
check all the features one by one in a loop:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; my_snp = 4350
&gt;&gt;&gt; record = SeqIO.read("NC_005816.gb", "genbank")
&gt;&gt;&gt; for feature in record.features:
...     if my_snp in feature:
...         print("%s %s" % (feature.type, feature.qualifiers.get('db_xref')))
... 
source ['taxon:229193']
gene ['GeneID:2767712']
CDS ['GI:45478716', 'GeneID:2767712']
</pre><p>Note that gene and CDS features from GenBank or EMBL files defined with joins
are the union of the exons &#X2013; they do not cover any introns.</p>
<!--TOC subsection id="sec45" Sequence described by a feature or location-->
<h3 id="sec45" class="subsection">4.3.3&#XA0;&#XA0;Sequence described by a feature or location</h3><!--SEC END --><p>A <code>SeqFeature</code> or location object doesn&#X2019;t directly contain a sequence, instead the location (see Section&#XA0;<a href="#sec%3Alocations">4.3.2</a>) describes how to get this from the parent sequence. For example consider a (short) gene sequence with location 5:18 on the reverse strand, which in GenBank/EMBL notation using 1-based counting would be <span style="font-family:monospace">complement(6..18)</span>, like this:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.SeqFeature import SeqFeature, FeatureLocation
&gt;&gt;&gt; example_parent = Seq("ACCGAGACGGCAAAGGCTAGCATAGGTATGAGACTTCCTTCCTGCCAGTGCTGAGGAACTGGGAGCCTAC")
&gt;&gt;&gt; example_feature = SeqFeature(FeatureLocation(5, 18), type="gene", strand=-1)
</pre><p>You could take the parent sequence, slice it to extract 5:18, and then take the reverse complement.
If you are using Biopython 1.59 or later, the feature location&#X2019;s start and end are integer like so this works:</p><pre class="verbatim">&gt;&gt;&gt; feature_seq = example_parent[example_feature.location.start:example_feature.location.end].reverse_complement()
&gt;&gt;&gt; print(feature_seq)
AGCCTTTGCCGTC
</pre><p>This is a simple example so this isn&#X2019;t too bad &#X2013; however once you have to deal with compound features (joins) this is rather messy. Instead, the <code>SeqFeature</code> object has an <code>extract</code> method to take care of all this:</p><pre class="verbatim">&gt;&gt;&gt; feature_seq = example_feature.extract(example_parent)
&gt;&gt;&gt; print(feature_seq)
AGCCTTTGCCGTC
</pre><p>The length of a <code>SeqFeature</code> or location matches
that of the region of sequence it describes.</p><pre class="verbatim">&gt;&gt;&gt; print(example_feature.extract(example_parent))
AGCCTTTGCCGTC
&gt;&gt;&gt; print(len(example_feature.extract(example_parent)))
13
&gt;&gt;&gt; print(len(example_feature))
13
&gt;&gt;&gt; print(len(example_feature.location))
13
</pre><p>For simple <code>FeatureLocation</code> objects the length is just
the difference between the start and end positions. However,
for a <code>CompoundLocation</code> the length is the sum of the
constituent regions.</p>
<!--TOC section id="sec46" References-->
<h2 id="sec46" class="section">4.4&#XA0;&#XA0;References</h2><!--SEC END --><p>Another common annotation related to a sequence is a reference to a journal or other published work dealing with the sequence. We have a fairly simple way of representing a Reference in Biopython &#X2013; we have a <code>Bio.SeqFeature.Reference</code> class that stores the relevant information about a reference as attributes of an object.</p><p>The attributes include things that you would expect to see in a reference like <code>journal</code>, <code>title</code> and <code>authors</code>. Additionally, it also can hold the <code>medline_id</code> and <code>pubmed_id</code> and a <code>comment</code> about the reference. These are all accessed simply as attributes of the object.</p><p>A reference also has a <code>location</code> object so that it can specify a particular location on the sequence that the reference refers to. For instance, you might have a journal that is dealing with a particular gene located on a BAC, and want to specify that it only refers to this position exactly. The <code>location</code> is a potentially fuzzy location, as described in section&#XA0;<a href="#sec%3Alocations">4.3.2</a>.</p><p>Any reference objects are stored as a list in the <code>SeqRecord</code> object&#X2019;s <code>annotations</code> dictionary under the key &#X201C;references&#X201D;.
That&#X2019;s all there is too it. References are meant to be easy to deal with, and hopefully general enough to cover lots of usage cases.</p>
<!--TOC section id="sec47" The format method-->
<h2 id="sec47" class="section">4.5&#XA0;&#XA0;The format method</h2><!--SEC END --><p>
<a id="sec:SeqRecord-format"></a></p><p>The <code>format()</code> method of the <code>SeqRecord</code> class gives a string
containing your record formatted using one of the output file formats
supported by <code>Bio.SeqIO</code>, such as FASTA:</p><pre class="verbatim">from Bio.Seq import Seq
from Bio.SeqRecord import SeqRecord
from Bio.Alphabet import generic_protein

record = SeqRecord(Seq("MMYQQGCFAGGTVLRLAKDLAENNRGARVLVVCSEITAVTFRGPSETHLDSMVGQALFGD" \
                      +"GAGAVIVGSDPDLSVERPLYELVWTGATLLPDSEGAIDGHLREVGLTFHLLKDVPGLISK" \
                      +"NIEKSLKEAFTPLGISDWNSTFWIAHPGGPAILDQVEAKLGLKEEKMRATREVLSEYGNM" \
                      +"SSAC", generic_protein),
                   id="gi|14150838|gb|AAK54648.1|AF376133_1",
                   description="chalcone synthase [Cucumis sativus]")
                   
print(record.format("fasta"))
</pre><p>which should give:
</p><pre class="verbatim">&gt;gi|14150838|gb|AAK54648.1|AF376133_1 chalcone synthase [Cucumis sativus]
MMYQQGCFAGGTVLRLAKDLAENNRGARVLVVCSEITAVTFRGPSETHLDSMVGQALFGD
GAGAVIVGSDPDLSVERPLYELVWTGATLLPDSEGAIDGHLREVGLTFHLLKDVPGLISK
NIEKSLKEAFTPLGISDWNSTFWIAHPGGPAILDQVEAKLGLKEEKMRATREVLSEYGNM
SSAC
</pre><p>This <code>format</code> method takes a single mandatory argument, a lower case string which is
supported by <code>Bio.SeqIO</code> as an output format (see Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>).
However, some of the file formats <code>Bio.SeqIO</code> can write to <em>require</em> more than
one record (typically the case for multiple sequence alignment formats), and thus won&#X2019;t
work via this <code>format()</code> method. See also Section&#XA0;<a href="#sec%3ABio.SeqIO-and-StringIO">5.5.4</a>.</p>
<!--TOC section id="sec48" Slicing a SeqRecord-->
<h2 id="sec48" class="section">4.6&#XA0;&#XA0;Slicing a SeqRecord</h2><!--SEC END --><p>
<a id="sec:SeqRecord-slicing"></a></p><p>You can slice a <code>SeqRecord</code>, to give you a new <code>SeqRecord</code> covering just
part of the sequence. What is important
here is that any per-letter annotations are also sliced, and any features which fall
completely within the new sequence are preserved (with their locations adjusted).</p><p>For example, taking the same GenBank file used earlier:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; record = SeqIO.read("NC_005816.gb", "genbank")
</pre><pre class="verbatim">&gt;&gt;&gt; record
SeqRecord(seq=Seq('TGTAACGAACGGTGCAATAGTGATCCACACCCAACGCCTGAAATCAGATCCAGG...CTG',
IUPACAmbiguousDNA()), id='NC_005816.1', name='NC_005816',
description='Yersinia pestis biovar Microtus str. 91001 plasmid pPCP1, complete sequence.',
dbxrefs=['Project:10638'])
</pre><pre class="verbatim">&gt;&gt;&gt; len(record)
9609
&gt;&gt;&gt; len(record.features)
41
</pre><p>For this example we&#X2019;re going to focus in on the <code>pim</code> gene, <code>YP_pPCP05</code>.
If you have a look at the GenBank file directly you&#X2019;ll find this gene/CDS has
location string <span style="font-family:monospace">4343..4780</span>, or in Python counting <span style="font-family:monospace">4342:4780</span>.
From looking at the file you can work out that these are the twelfth and
thirteenth entries in the file, so in Python zero-based counting they are
entries 11 and 12 in the <span style="font-family:monospace">features</span> list:</p><pre class="verbatim">&gt;&gt;&gt; print(record.features[20])
type: gene
location: [4342:4780](+)
qualifiers: 
    Key: db_xref, Value: ['GeneID:2767712']
    Key: gene, Value: ['pim']
    Key: locus_tag, Value: ['YP_pPCP05']
&lt;BLANKLINE&gt;
</pre><pre class="verbatim">&gt;&gt;&gt; print(record.features[21])
type: CDS
location: [4342:4780](+)
qualifiers: 
    Key: codon_start, Value: ['1']
    Key: db_xref, Value: ['GI:45478716', 'GeneID:2767712']
    Key: gene, Value: ['pim']
    Key: locus_tag, Value: ['YP_pPCP05']
    Key: note, Value: ['similar to many previously sequenced pesticin immunity ...']
    Key: product, Value: ['pesticin immunity protein']
    Key: protein_id, Value: ['NP_995571.1']
    Key: transl_table, Value: ['11']
    Key: translation, Value: ['MGGGMISKLFCLALIFLSSSGLAEKNTYTAKDILQNLELNTFGNSLSH...']
</pre><p>Let&#X2019;s slice this parent record from 4300 to 4800 (enough to include the <code>pim</code>
gene/CDS), and see how many features we get:</p><pre class="verbatim">&gt;&gt;&gt; sub_record = record[4300:4800]
</pre><pre class="verbatim">&gt;&gt;&gt; sub_record
SeqRecord(seq=Seq('ATAAATAGATTATTCCAAATAATTTATTTATGTAAGAACAGGATGGGAGGGGGA...TTA',
IUPACAmbiguousDNA()), id='NC_005816.1', name='NC_005816',
description='Yersinia pestis biovar Microtus str. 91001 plasmid pPCP1, complete sequence.',
dbxrefs=[])
</pre><pre class="verbatim">&gt;&gt;&gt; len(sub_record)
500
&gt;&gt;&gt; len(sub_record.features)
2
</pre><p>Our sub-record just has two features, the gene and CDS entries for <code>YP_pPCP05</code>:</p><pre class="verbatim">&gt;&gt;&gt; print(sub_record.features[0])
type: gene
location: [42:480](+)
qualifiers: 
    Key: db_xref, Value: ['GeneID:2767712']
    Key: gene, Value: ['pim']
    Key: locus_tag, Value: ['YP_pPCP05']
&lt;BLANKLINE&gt;
</pre><pre class="verbatim">&gt;&gt;&gt; print(sub_record.features[20])
type: CDS
location: [42:480](+)
qualifiers: 
    Key: codon_start, Value: ['1']
    Key: db_xref, Value: ['GI:45478716', 'GeneID:2767712']
    Key: gene, Value: ['pim']
    Key: locus_tag, Value: ['YP_pPCP05']
    Key: note, Value: ['similar to many previously sequenced pesticin immunity ...']
    Key: product, Value: ['pesticin immunity protein']
    Key: protein_id, Value: ['NP_995571.1']
    Key: transl_table, Value: ['11']
    Key: translation, Value: ['MGGGMISKLFCLALIFLSSSGLAEKNTYTAKDILQNLELNTFGNSLSH...']
</pre><p>Notice that their locations have been adjusted to reflect the new parent sequence!</p><p>While Biopython has done something sensible and hopefully intuitive with the features
(and any per-letter annotation), for the other annotation it is impossible to know if
this still applies to the sub-sequence or not. To avoid guessing, the <span style="font-family:monospace">annotations</span>
and <span style="font-family:monospace">dbxrefs</span> are omitted from the sub-record, and it is up to you to transfer
any relevant information as appropriate.</p><pre class="verbatim">&gt;&gt;&gt; sub_record.annotations
{}
&gt;&gt;&gt; sub_record.dbxrefs
[]
</pre><p>The same point could be made about the record <span style="font-family:monospace">id</span>, <span style="font-family:monospace">name</span>
and <span style="font-family:monospace">description</span>, but for practicality these are preserved:</p><pre class="verbatim">&gt;&gt;&gt; sub_record.id
'NC_005816.1'
&gt;&gt;&gt; sub_record.name
'NC_005816'
&gt;&gt;&gt; sub_record.description
'Yersinia pestis biovar Microtus str. 91001 plasmid pPCP1, complete sequence.'
</pre><p>This illustrates the problem nicely though, our new sub-record is
<em>not</em> the complete sequence of the plasmid, so the description is wrong!
Let&#X2019;s fix this and then view the sub-record as a reduced GenBank file using
the <span style="font-family:monospace">format</span> method described above in Section&#XA0;<a href="#sec%3ASeqRecord-format">4.5</a>:</p><pre class="verbatim">&gt;&gt;&gt; sub_record.description = "Yersinia pestis biovar Microtus str. 91001 plasmid pPCP1, partial."
&gt;&gt;&gt; print(sub_record.format("genbank"))
...
</pre><p>See Sections&#XA0;<a href="#sec%3AFASTQ-slicing-off-primer">18.1.7</a>
and&#XA0;<a href="#sec%3AFASTQ-slicing-off-adaptor">18.1.8</a> for some FASTQ examples where the
per-letter annotations (the read quality scores) are also sliced.</p>
<!--TOC section id="sec49" Adding SeqRecord objects-->
<h2 id="sec49" class="section">4.7&#XA0;&#XA0;Adding SeqRecord objects</h2><!--SEC END --><p>
<a id="sec:SeqRecord-addition"></a></p><p>You can add <code>SeqRecord</code> objects together, giving a new <code>SeqRecord</code>.
What is important here is that any common
per-letter annotations are also added, all the features are preserved (with their
locations adjusted), and any other common annotation is also kept (like the id, name
and description).</p><p>For an example with per-letter annotation, we&#X2019;ll use the first record in a
FASTQ file. Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a> will explain the <code>SeqIO</code> functions:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; record = next(SeqIO.parse("example.fastq", "fastq"))
&gt;&gt;&gt; len(record)
25
&gt;&gt;&gt; print(record.seq)
CCCTTCTTGTCTTCAGCGTTTCTCC
</pre><pre class="verbatim">&gt;&gt;&gt; print(record.letter_annotations["phred_quality"])
[26, 26, 18, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 22, 26, 26, 26, 26,
26, 26, 26, 23, 23]
</pre><p>Let&#X2019;s suppose this was Roche 454 data, and that from other information
you think the <span style="font-family:monospace">TTT</span> should be only <span style="font-family:monospace">TT</span>. We can make a new edited
record by first slicing the <code>SeqRecord</code> before and after the &#X201C;extra&#X201D;
third <span style="font-family:monospace">T</span>:</p><pre class="verbatim">&gt;&gt;&gt; left = record[:20]
&gt;&gt;&gt; print(left.seq)
CCCTTCTTGTCTTCAGCGTT
&gt;&gt;&gt; print(left.letter_annotations["phred_quality"])
[26, 26, 18, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 22, 26, 26, 26, 26]
&gt;&gt;&gt; right = record[21:]
&gt;&gt;&gt; print(right.seq)
CTCC
&gt;&gt;&gt; print(right.letter_annotations["phred_quality"])
[26, 26, 23, 23]
</pre><p>Now add the two parts together:</p><pre class="verbatim">&gt;&gt;&gt; edited = left + right
&gt;&gt;&gt; len(edited)
24
&gt;&gt;&gt; print(edited.seq)
CCCTTCTTGTCTTCAGCGTTCTCC
</pre><pre class="verbatim">&gt;&gt;&gt; print(edited.letter_annotations["phred_quality"])
[26, 26, 18, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 22, 26, 26, 26, 26,
26, 26, 23, 23]
</pre><p>Easy and intuitive? We hope so! You can make this shorter with just:</p><pre class="verbatim">&gt;&gt;&gt; edited = record[:20] + record[21:]
</pre><p>Now, for an example with features, we&#X2019;ll use a GenBank file.
Suppose you have a circular genome:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; record = SeqIO.read("NC_005816.gb", "genbank")
</pre><pre class="verbatim">&gt;&gt;&gt; record
SeqRecord(seq=Seq('TGTAACGAACGGTGCAATAGTGATCCACACCCAACGCCTGAAATCAGATCCAGG...CTG',
IUPACAmbiguousDNA()), id='NC_005816.1', name='NC_005816',
description='Yersinia pestis biovar Microtus str. 91001 plasmid pPCP1, complete sequence.',
dbxrefs=['Project:10638'])
</pre><pre class="verbatim">&gt;&gt;&gt; len(record)
9609
&gt;&gt;&gt; len(record.features)
41
&gt;&gt;&gt; record.dbxrefs
['Project:58037']
</pre><pre class="verbatim">&gt;&gt;&gt; record.annotations.keys()
['comment', 'sequence_version', 'source', 'taxonomy', 'keywords', 'references',
'accessions', 'data_file_division', 'date', 'organism', 'gi']
</pre><p>You can shift the origin like this:</p><pre class="verbatim">&gt;&gt;&gt; shifted = record[2000:] + record[:2000]
</pre><pre class="verbatim">&gt;&gt;&gt; shifted
SeqRecord(seq=Seq('GATACGCAGTCATATTTTTTACACAATTCTCTAATCCCGACAAGGTCGTAGGTC...GGA',
IUPACAmbiguousDNA()), id='NC_005816.1', name='NC_005816',
description='Yersinia pestis biovar Microtus str. 91001 plasmid pPCP1, complete sequence.',
dbxrefs=[])
</pre><pre class="verbatim">&gt;&gt;&gt; len(shifted)
9609
</pre><p>Note that this isn&#X2019;t perfect in that some annotation like the database cross references
and one of the features (the source feature) have been lost:</p><pre class="verbatim">&gt;&gt;&gt; len(shifted.features)
40
&gt;&gt;&gt; shifted.dbxrefs
[]
&gt;&gt;&gt; shifted.annotations.keys()
[]
</pre><p>This is because the <code>SeqRecord</code> slicing step is cautious in what annotation
it preserves (erroneously propagating annotation can cause major problems). If
you want to keep the database cross references or the annotations dictionary,
this must be done explicitly:</p><pre class="verbatim">&gt;&gt;&gt; shifted.dbxrefs = record.dbxrefs[:]
&gt;&gt;&gt; shifted.annotations = record.annotations.copy()
&gt;&gt;&gt; shifted.dbxrefs
['Project:10638']
&gt;&gt;&gt; shifted.annotations.keys()
['comment', 'sequence_version', 'source', 'taxonomy', 'keywords', 'references',
'accessions', 'data_file_division', 'date', 'organism', 'gi']
</pre><p>Also note that in an example like this, you should probably change the record
identifiers since the NCBI references refer to the <em>original</em> unmodified
sequence.</p>
<!--TOC section id="sec50" Reverse-complementing SeqRecord objects-->
<h2 id="sec50" class="section">4.8&#XA0;&#XA0;Reverse-complementing SeqRecord objects</h2><!--SEC END --><p>
<a id="sec:SeqRecord-reverse-complement"></a></p><p>One of the new features in Biopython 1.57 was the <code>SeqRecord</code> object&#X2019;s
<code>reverse_complement</code> method. This tries to balance easy of use with worries
about what to do with the annotation in the reverse complemented record.</p><p>For the sequence, this uses the Seq object&#X2019;s reverse complement method. Any
features are transferred with the location and strand recalculated. Likewise
any per-letter-annotation is also copied but reversed (which makes sense for
typical examples like quality scores). However, transfer of most annotation
is problematical.</p><p>For instance, if the record ID was an accession, that accession should not really
apply to the reverse complemented sequence, and transferring the identifier by
default could easily cause subtle data corruption in downstream analysis.
Therefore by default, the <code>SeqRecord</code>&#X2019;s id, name, description, annotations
and database cross references are all <em>not</em> transferred by default.</p><p>The <code>SeqRecord</code> object&#X2019;s <code>reverse_complement</code> method takes a number
of optional arguments corresponding to properties of the record. Setting these
arguments to <code>True</code> means copy the old values, while <code>False</code> means
drop the old values and use the default value. You can alternatively provide
the new desired value instead.</p><p>Consider this example record:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; record = SeqIO.read("NC_005816.gb", "genbank")
&gt;&gt;&gt; print("%s %i %i %i %i" % (record.id, len(record), len(record.features), len(record.dbxrefs), len(record.annotations)))
NC_005816.1 9609 41 1 11
</pre><p>Here we take the reverse complement and specify a new identifier &#X2013; but notice
how most of the annotation is dropped (but not the features):</p><pre class="verbatim">&gt;&gt;&gt; rc = record.reverse_complement(id="TESTING")
&gt;&gt;&gt; print("%s %i %i %i %i" % (rc.id, len(rc), len(rc.features), len(rc.dbxrefs), len(rc.annotations)))
TESTING 9609 41 0 0
</pre>
<!--TOC chapter id="sec51" Sequence Input/Output-->
<h1 id="sec51" class="chapter">Chapter&#XA0;5&#XA0;&#XA0;Sequence Input/Output</h1><!--SEC END --><p>
<a id="chapter:Bio.SeqIO"></a></p><p>In this chapter we&#X2019;ll discuss in more detail the <code>Bio.SeqIO</code> module, which was briefly introduced in Chapter&#XA0;<a href="#chapter%3Aquick-start">2</a> and also used in Chapter&#XA0;<a href="#chapter%3ASeqRecord">4</a>. This aims to provide a simple interface for working with assorted sequence file formats in a uniform way.
See also the <code>Bio.SeqIO</code> wiki page (<a href="http://biopython.org/wiki/SeqIO"><span style="font-family:monospace">http://biopython.org/wiki/SeqIO</span></a>), and the built in documentation (also <a href="http://biopython.org/DIST/docs/api/Bio.SeqIO-module.html">online</a>):</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; help(SeqIO)
...
</pre><p>The &#X201C;catch&#X201D; is that you have to work with <code>SeqRecord</code> objects (see Chapter&#XA0;<a href="#chapter%3ASeqRecord">4</a>), which contain a <code>Seq</code> object (see Chapter&#XA0;<a href="#chapter%3ABio.Seq">3</a>) plus annotation like an identifier and description.</p>
<!--TOC section id="sec52" Parsing or Reading Sequences-->
<h2 id="sec52" class="section">5.1&#XA0;&#XA0;Parsing or Reading Sequences</h2><!--SEC END --><p>
<a id="sec:Bio.SeqIO-input"></a></p><p>The workhorse function <code>Bio.SeqIO.parse()</code> is used to read in sequence data as SeqRecord objects. This function expects two arguments:</p><ol class="enumerate" type=1><li class="li-enumerate">
The first argument is a <span style="font-style:italic">handle</span> to read the data from, or a filename. A handle is typically a file opened for reading, but could be the output from a command line program, or data downloaded from the internet (see Section&#XA0;<a href="#sec%3ASeqIO_Online">5.3</a>). See Section&#XA0;<a href="#sec%3Aappendix-handles">22.1</a> for more about handles.
</li><li class="li-enumerate">The second argument is a lower case string specifying sequence format &#X2013; we don&#X2019;t try and guess the file format for you! See <a href="http://biopython.org/wiki/SeqIO"><span style="font-family:monospace">http://biopython.org/wiki/SeqIO</span></a> for a full listing of supported formats.
</li></ol><p>There is an optional argument <code>alphabet</code> to specify the alphabet to be used. This is useful for file formats like FASTA where otherwise <code>Bio.SeqIO</code> will default to a generic alphabet.</p><p>The <code>Bio.SeqIO.parse()</code> function returns an <span style="font-style:italic">iterator</span> which gives <code>SeqRecord</code> objects. Iterators are typically used in a for loop as shown below.</p><p>Sometimes you&#X2019;ll find yourself dealing with files which contain only a single record. For this situation use the function <code>Bio.SeqIO.read()</code> which takes the same arguments. Provided there is one and only one record in the file, this is returned as a <code>SeqRecord</code> object. Otherwise an exception is raised.</p>
<!--TOC subsection id="sec53" Reading Sequence Files-->
<h3 id="sec53" class="subsection">5.1.1&#XA0;&#XA0;Reading Sequence Files</h3><!--SEC END --><p>In general <code>Bio.SeqIO.parse()</code> is used to read in sequence files as <code>SeqRecord</code> objects, and is typically used with a for loop like this:</p><pre class="verbatim">from Bio import SeqIO
for seq_record in SeqIO.parse("ls_orchid.fasta", "fasta"):
    print(seq_record.id)
    print(repr(seq_record.seq))
    print(len(seq_record))
</pre><p>The above example is repeated from the introduction in Section&#XA0;<a href="#sec%3Asequence-parsing">2.4</a>, and will load the orchid DNA sequences in the FASTA format file <a href="http://biopython.org/DIST/docs/tutorial/examples/ls_orchid.fasta">ls_orchid.fasta</a>. If instead you wanted to load a GenBank format file like <a href="http://biopython.org/DIST/docs/tutorial/examples/ls_orchid.gbk">ls_orchid.gbk</a> then all you need to do is change the filename and the format string:</p><pre class="verbatim">from Bio import SeqIO
for seq_record in SeqIO.parse("ls_orchid.gbk", "genbank"):
    print(seq_record.id)
    print(seq_record.seq)
    print(len(seq_record))
</pre><p>Similarly, if you wanted to read in a file in another file format, then assuming <code>Bio.SeqIO.parse()</code> supports it you would just need to change the format string as appropriate, for example &#X201C;swiss&#X201D; for SwissProt files or &#X201C;embl&#X201D; for EMBL text files. There is a full listing on the wiki page (<a href="http://biopython.org/wiki/SeqIO"><span style="font-family:monospace">http://biopython.org/wiki/SeqIO</span></a>) and in the built in documentation (also <a href="http://biopython.org/DIST/docs/api/Bio.SeqIO-module.html">online</a>).</p><p>Another very common way to use a Python iterator is within a list comprehension (or
a generator expression). For example, if all you wanted to extract from the file was
a list of the record identifiers we can easily do this with the following list comprehension:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; identifiers = [seq_record.id for seq_record in SeqIO.parse("ls_orchid.gbk", "genbank")]
&gt;&gt;&gt; identifiers
['Z78533.1', 'Z78532.1', 'Z78531.1', 'Z78530.1', 'Z78529.1', 'Z78527.1', ..., 'Z78439.1']
</pre><p>There are more examples using <code>SeqIO.parse()</code> in a list
comprehension like this in Section&#XA0;<a href="#seq%3Asequence-parsing-plus-pylab">18.2</a>
(e.g. for plotting sequence lengths or GC%).</p>
<!--TOC subsection id="sec54" Iterating over the records in a sequence file-->
<h3 id="sec54" class="subsection">5.1.2&#XA0;&#XA0;Iterating over the records in a sequence file</h3><!--SEC END --><p>In the above examples, we have usually used a for loop to iterate over all the records one by one. You can use the for loop with all sorts of Python objects (including lists, tuples and strings) which support the iteration interface.</p><p>The object returned by <code>Bio.SeqIO</code> is actually an iterator which returns <code>SeqRecord</code> objects. You get to see each record in turn, but once and only once. The plus point is that an iterator can save you memory when dealing with large files.</p><p>Instead of using a for loop, can also use the <code>next()</code> function on an iterator to step through the entries, like this:</p><pre class="verbatim">from Bio import SeqIO
record_iterator = SeqIO.parse("ls_orchid.fasta", "fasta")

first_record = next(record_iterator)
print(first_record.id)
print(first_record.description)

second_record = next(record_iterator)
print(second_record.id)
print(second_record.description)
</pre><p>Note that if you try to use <code>next()</code> and there are no more results, you&#X2019;ll get the special <code>StopIteration</code> exception.</p><p>One special case to consider is when your sequence files have multiple records, but you only want the first one. In this situation the following code is very concise:</p><pre class="verbatim">from Bio import SeqIO
first_record = next(SeqIO.parse("ls_orchid.gbk", "genbank"))
</pre><p>A word of warning here &#X2013; using the <code>next()</code> function like this will silently ignore any additional records in the file.
If your files have <span style="font-style:italic">one and only one</span> record, like some of the online examples later in this chapter, or a GenBank file for a single chromosome, then use the new <code>Bio.SeqIO.read()</code> function instead.
This will check there are no extra unexpected records present.</p>
<!--TOC subsection id="sec55" Getting a list of the records in a sequence file-->
<h3 id="sec55" class="subsection">5.1.3&#XA0;&#XA0;Getting a list of the records in a sequence file</h3><!--SEC END --><p>In the previous section we talked about the fact that <code>Bio.SeqIO.parse()</code> gives you a <code>SeqRecord</code> iterator, and that you get the records one by one. Very often you need to be able to access the records in any order. The Python <code>list</code> data type is perfect for this, and we can turn the record iterator into a list of <code>SeqRecord</code> objects using the built-in Python function <code>list()</code> like so:</p><pre class="verbatim">from Bio import SeqIO
records = list(SeqIO.parse("ls_orchid.gbk", "genbank"))

print("Found %i records" % len(records))

print("The last record")
last_record = records[-1] #using Python's list tricks
print(last_record.id)
print(repr(last_record.seq))
print(len(last_record))

print("The first record")
first_record = records[0] #remember, Python counts from zero
print(first_record.id)
print(repr(first_record.seq))
print(len(first_record))
</pre><p>Giving:</p><pre class="verbatim">Found 94 records
The last record
Z78439.1
Seq('CATTGTTGAGATCACATAATAATTGATCGAGTTAATCTGGAGGATCTGTTTACT...GCC', IUPACAmbiguousDNA())
592
The first record
Z78533.1
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGATGAGACCGTGG...CGC', IUPACAmbiguousDNA())
740
</pre><p>You can of course still use a for loop with a list of <code>SeqRecord</code> objects. Using a list is much more flexible than an iterator (for example, you can determine the number of records from the length of the list), but does need more memory because it will hold all the records in memory at once.</p>
<!--TOC subsection id="sec56" Extracting data-->
<h3 id="sec56" class="subsection">5.1.4&#XA0;&#XA0;Extracting data</h3><!--SEC END --><p>The <code>SeqRecord</code> object and its annotation structures are described more fully in
Chapter&#XA0;<a href="#chapter%3ASeqRecord">4</a>. As an example of how annotations are stored, we&#X2019;ll look at the output from parsing the first record in the GenBank file <a href="http://biopython.org/DIST/docs/tutorial/examples/ls_orchid.gbk">ls_orchid.gbk</a>.</p><pre class="verbatim">from Bio import SeqIO
record_iterator = SeqIO.parse("ls_orchid.gbk", "genbank")
first_record = next(record_iterator)
print(first_record)
</pre><p>That should give something like this:</p><pre class="verbatim">ID: Z78533.1
Name: Z78533
Description: C.irapeanum 5.8S rRNA gene and ITS1 and ITS2 DNA.
Number of features: 5
/sequence_version=1
/source=Cypripedium irapeanum
/taxonomy=['Eukaryota', 'Viridiplantae', 'Streptophyta', ..., 'Cypripedium']
/keywords=['5.8S ribosomal RNA', '5.8S rRNA gene', ..., 'ITS1', 'ITS2']
/references=[...]
/accessions=['Z78533']
/data_file_division=PLN
/date=30-NOV-2006
/organism=Cypripedium irapeanum
/gi=2765658
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGATGAGACCGTGG...CGC', IUPACAmbiguousDNA())
</pre><p>This gives a human readable summary of most of the annotation data for the <code>SeqRecord</code>.
For this example we&#X2019;re going to use the <code>.annotations</code> attribute which is just a Python dictionary.
The contents of this annotations dictionary were shown when we printed the record above.
You can also print them out directly:
</p><pre class="verbatim">print(first_record.annotations)
</pre><p>Like any Python dictionary, you can easily get a list of the keys:
</p><pre class="verbatim">print(first_record.annotations.keys())
</pre><p>or values:
</p><pre class="verbatim">print(first_record.annotations.values())
</pre><p>In general, the annotation values are strings, or lists of strings. One special case is any references in the file get stored as reference objects. </p><p>Suppose you wanted to extract a list of the species from the <a href="http://biopython.org/DIST/docs/tutorial/examples/ls_orchid.gbk">ls_orchid.gbk</a> GenBank file. The information we want, <em>Cypripedium irapeanum</em>, is held in the annotations dictionary under &#X2018;source&#X2019; and &#X2018;organism&#X2019;, which we can access like this:</p><pre class="verbatim">&gt;&gt;&gt; print(first_record.annotations["source"])
Cypripedium irapeanum
</pre><p>or:</p><pre class="verbatim">&gt;&gt;&gt; print(first_record.annotations["organism"])
Cypripedium irapeanum
</pre><p>In general, &#X2018;organism&#X2019; is used for the scientific name (in Latin, e.g. <span style="font-style:italic">Arabidopsis thaliana</span>),
while &#X2018;source&#X2019; will often be the common name (e.g. thale cress). In this example, as is often the case,
the two fields are identical. </p><p>Now let&#X2019;s go through all the records, building up a list of the species each orchid sequence is from:</p><pre class="verbatim">from Bio import SeqIO
all_species = []
for seq_record in SeqIO.parse("ls_orchid.gbk", "genbank"):
    all_species.append(seq_record.annotations["organism"])
print(all_species)
</pre><p>Another way of writing this code is to use a list comprehension:</p><pre class="verbatim">from Bio import SeqIO
all_species = [seq_record.annotations["organism"] for seq_record in \
               SeqIO.parse("ls_orchid.gbk", "genbank")]
print(all_species)
</pre><p>In either case, the result is:</p><pre class="verbatim">['Cypripedium irapeanum', 'Cypripedium californicum', ..., 'Paphiopedilum barbatum']
</pre><p>Great. That was pretty easy because GenBank files are annotated in a standardised way.</p><p>Now, let&#X2019;s suppose you wanted to extract a list of the species from a FASTA file, rather than the GenBank file. The bad news is you will have to write some code to extract the data you want from the record&#X2019;s description line - if the information is in the file in the first place! Our example FASTA format file <a href="http://biopython.org/DIST/docs/tutorial/examples/ls_orchid.fasta">ls_orchid.fasta</a> starts like this:</p><pre class="verbatim">&gt;gi|2765658|emb|Z78533.1|CIZ78533 C.irapeanum 5.8S rRNA gene and ITS1 and ITS2 DNA
CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGATGAGACCGTGGAATAAACGATCGAGTG
AATCCGGAGGACCGGTGTACTCAGCTCACCGGGGGCATTGCTCCCGTGGTGACCCTGATTTGTTGTTGGG
...
</pre><p>You can check by hand, but for every record the species name is in the description line as the second word. This means if we break up each record&#X2019;s <code>.description</code> at the spaces, then the species is there as field number one (field zero is the record identifier). That means we can do this:</p><pre class="verbatim">from Bio import SeqIO
all_species = []
for seq_record in SeqIO.parse("ls_orchid.fasta", "fasta"):
    all_species.append(seq_record.description.split()[1])
print(all_species)
</pre><p>This gives:</p><pre class="verbatim">['C.irapeanum', 'C.californicum', 'C.fasciculatum', 'C.margaritaceum', ..., 'P.barbatum']
</pre><p>The concise alternative using list comprehensions would be:</p><pre class="verbatim">from Bio import SeqIO
all_species == [seq_record.description.split()[1] for seq_record in \
                SeqIO.parse("ls_orchid.fasta", "fasta")]
print(all_species)
</pre><p>In general, extracting information from the FASTA description line is not very nice.
If you can get your sequences in a well annotated file format like GenBank or EMBL,
then this sort of annotation information is much easier to deal with.</p>
<!--TOC section id="sec57" Parsing sequences from compressed files-->
<h2 id="sec57" class="section">5.2&#XA0;&#XA0;Parsing sequences from compressed files</h2><!--SEC END --><p>
<a id="sec:SeqIO_compressed"></a>
In the previous section, we looked at parsing sequence data from a file.
Instead of using a filename, you can give <code>Bio.SeqIO</code> a handle
(see Section&#XA0;<a href="#sec%3Aappendix-handles">22.1</a>), and in this section
we&#X2019;ll use handles to parse sequence from compressed files.</p><p>As you&#X2019;ll have seen above, we can use <code>Bio.SeqIO.read()</code> or
<code>Bio.SeqIO.parse()</code> with a filename - for instance this quick
example calculates the total length of the sequences in a multiple
record GenBank file using a generator expression:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; print(sum(len(r) for r in SeqIO.parse("ls_orchid.gbk", "gb")))
67518
</pre><p>Here we use a file handle instead, using the <code>with</code> statement
to close the handle automatically:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; with open("ls_orchid.gbk") as handle:
...     print(sum(len(r) for r in SeqIO.parse(handle, "gb")))
67518
</pre><p>Or, the old fashioned way where you manually close the handle:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; handle = open("ls_orchid.gbk")
&gt;&gt;&gt; print(sum(len(r) for r in SeqIO.parse(handle, "gb")))
67518
&gt;&gt;&gt; handle.close()
</pre><p>Now, suppose we have a gzip compressed file instead? These are very
commonly used on Linux. We can use Python&#X2019;s <code>gzip</code> module to open
the compressed file for reading - which gives us a handle object:</p><pre class="verbatim">&gt;&gt;&gt; import gzip
&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; handle = gzip.open("ls_orchid.gbk.gz", "r")
&gt;&gt;&gt; print(sum(len(r) for r in SeqIO.parse(handle, "gb")))
67518
&gt;&gt;&gt; handle.close()
</pre><p>Similarly if we had a bzip2 compressed file (sadly the function name isn&#X2019;t
quite as consistent):</p><pre class="verbatim">&gt;&gt;&gt; import bz2
&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; handle = bz2.BZ2File("ls_orchid.gbk.bz2", "r")
&gt;&gt;&gt; print(sum(len(r) for r in SeqIO.parse(handle, "gb")))
67518
&gt;&gt;&gt; handle.close()
</pre><p>If you are using Python 2.7 or later, the <code>with</code>-version works for
gzip and bz2 as well. Unfortunately this is broken on older versions of
Python (<a href="http://bugs.python.org/issue3860">Issue 3860</a>) and you&#X2019;d
get an <code>AttributeError</code> about <code>__exit__</code> being missing.</p><p>There is a gzip (GNU Zip) variant called BGZF (Blocked GNU Zip Format),
which can be treated like an ordinary gzip file for reading, but has
advantages for random access later which we&#X2019;ll talk about later in
Section&#XA0;<a href="#sec%3ASeqIO-index-bgzf">5.4.4</a>.</p>
<!--TOC section id="sec58" Parsing sequences from the net-->
<h2 id="sec58" class="section">5.3&#XA0;&#XA0;Parsing sequences from the net</h2><!--SEC END --><p>
<a id="sec:SeqIO_Online"></a>
In the previous sections, we looked at parsing sequence data from a file
(using a filename or handle), and from compressed files (using a handle).
Here we&#X2019;ll use <code>Bio.SeqIO</code> with another type of handle, a network
connection, to download and parse sequences from the internet.</p><p>Note that just because you <em>can</em> download sequence data and parse it into
a <code>SeqRecord</code> object in one go doesn&#X2019;t mean this is a good idea.
In general, you should probably download sequences <em>once</em> and save them to
a file for reuse.</p>
<!--TOC subsection id="sec59" Parsing GenBank records from the net-->
<h3 id="sec59" class="subsection">5.3.1&#XA0;&#XA0;Parsing GenBank records from the net</h3><!--SEC END --><p>
<a id="sec:SeqIO_GenBank_Online"></a>
Section&#XA0;<a href="#sec%3Aefetch">9.6</a> talks about the Entrez EFetch interface in more detail,
but for now let&#X2019;s just connect to the NCBI and get a few <span style="font-style:italic">Opuntia</span> (prickly-pear)
sequences from GenBank using their GI numbers.</p><p>First of all, let&#X2019;s fetch just one record. If you don&#X2019;t care about the
annotations and features downloading a FASTA file is a good choice as these
are compact. Now remember, when you expect the handle to contain one and
only one record, use the <code>Bio.SeqIO.read()</code> function:</p><pre class="verbatim">from Bio import Entrez
from Bio import SeqIO
Entrez.email = "A.N.Other@example.com"
handle = Entrez.efetch(db="nucleotide", rettype="fasta", retmode="text", id="6273291")
seq_record = SeqIO.read(handle, "fasta")
handle.close()
print("%s with %i features" % (seq_record.id, len(seq_record.features)))
</pre><p>Expected output:</p><pre class="verbatim">gi|6273291|gb|AF191665.1|AF191665 with 0 features
</pre><p>The NCBI will also let you ask for the file in other formats, in particular as
a GenBank file. Until Easter 2009, the Entrez EFetch API let you use &#X201C;genbank&#X201D;
as the return type, however the NCBI now insist on using the official
return types of &#X201C;gb&#X201D; (or &#X201C;gp&#X201D; for proteins) as described on
<a href="http://www.ncbi.nlm.nih.gov/entrez/query/static/efetchseq_help.html">EFetch for Sequence and other Molecular Biology Databases</a>.
As a result, in Biopython 1.50 onwards, we support &#X201C;gb&#X201D; as an
alias for &#X201C;genbank&#X201D; in <code>Bio.SeqIO</code>.</p><pre class="verbatim">from Bio import Entrez
from Bio import SeqIO
Entrez.email = "A.N.Other@example.com"
handle = Entrez.efetch(db="nucleotide", rettype="gb", retmode="text", id="6273291")
seq_record = SeqIO.read(handle, "gb") #using "gb" as an alias for "genbank"
handle.close()
print("%s with %i features" % (seq_record.id, len(seq_record.features)))
</pre><p>The expected output of this example is:</p><pre class="verbatim">AF191665.1 with 3 features
</pre><p>Notice this time we have three features.</p><p>Now let&#X2019;s fetch several records. This time the handle contains multiple records,
so we must use the <code>Bio.SeqIO.parse()</code> function:</p><pre class="verbatim">from Bio import Entrez
from Bio import SeqIO
Entrez.email = "A.N.Other@example.com"
handle = Entrez.efetch(db="nucleotide", rettype="gb", retmode="text", \
                       id="6273291,6273290,6273289")
for seq_record in SeqIO.parse(handle, "gb"):
    print seq_record.id, seq_record.description[:50] + "..."
    print "Sequence length %i," % len(seq_record),
    print "%i features," % len(seq_record.features),
    print "from: %s" % seq_record.annotations["source"]
handle.close()
</pre><p>That should give the following output:</p><pre class="verbatim">AF191665.1 Opuntia marenae rpl16 gene; chloroplast gene for c...
Sequence length 902, 3 features, from: chloroplast Opuntia marenae
AF191664.1 Opuntia clavata rpl16 gene; chloroplast gene for c...
Sequence length 899, 3 features, from: chloroplast Grusonia clavata
AF191663.1 Opuntia bradtiana rpl16 gene; chloroplast gene for...
Sequence length 899, 3 features, from: chloroplast Opuntia bradtianaa
</pre><p>See Chapter&#XA0;<a href="#chapter%3Aentrez">9</a> for more about the <code>Bio.Entrez</code> module, and make sure to read about the NCBI guidelines for using Entrez (Section&#XA0;<a href="#sec%3Aentrez-guidelines">9.1</a>).</p>
<!--TOC subsection id="sec60" Parsing SwissProt sequences from the net-->
<h3 id="sec60" class="subsection">5.3.2&#XA0;&#XA0;Parsing SwissProt sequences from the net</h3><!--SEC END --><p>
<a id="sec:SeqIO_ExPASy_and_SwissProt"></a>
Now let&#X2019;s use a handle to download a SwissProt file from ExPASy,
something covered in more depth in Chapter&#XA0;<a href="#chapter%3Aswiss_prot">10</a>.
As mentioned above, when you expect the handle to contain one and only one record,
use the <code>Bio.SeqIO.read()</code> function:</p><pre class="verbatim">from Bio import ExPASy
from Bio import SeqIO
handle = ExPASy.get_sprot_raw("O23729")
seq_record = SeqIO.read(handle, "swiss")
handle.close()
print(seq_record.id)
print(seq_record.name)
print(seq_record.description)
print(repr(seq_record.seq))
print("Length %i" % len(seq_record))
print(seq_record.annotations["keywords"])
</pre><p>Assuming your network connection is OK, you should get back:</p><pre class="verbatim">O23729
CHS3_BROFI
RecName: Full=Chalcone synthase 3; EC=2.3.1.74; AltName: Full=Naringenin-chalcone synthase 3;
Seq('MAPAMEEIRQAQRAEGPAAVLAIGTSTPPNALYQADYPDYYFRITKSEHLTELK...GAE', ProteinAlphabet())
Length 394
['Acyltransferase', 'Flavonoid biosynthesis', 'Transferase']
</pre>
<!--TOC section id="sec61" Sequence files as Dictionaries-->
<h2 id="sec61" class="section">5.4&#XA0;&#XA0;Sequence files as Dictionaries</h2><!--SEC END --><p>We&#X2019;re now going to introduce three related functions in the <code>Bio.SeqIO</code>
module which allow dictionary like random access to a multi-sequence file.
There is a trade off here between flexibility and memory usage. In summary:
</p><ul class="itemize"><li class="li-itemize">
<code>Bio.SeqIO.to_dict()</code> is the most flexible but also the most
memory demanding option (see Section&#XA0;<a href="#SeqIO%3Ato_dict">5.4.1</a>). This is basically
a helper function to build a normal Python <code>dictionary</code> with each entry
held as a <code>SeqRecord</code> object in memory, allowing you to modify the
records.
</li><li class="li-itemize"><code>Bio.SeqIO.index()</code> is a useful middle ground, acting like a
read only dictionary and parsing sequences into <code>SeqRecord</code> objects
on demand (see Section&#XA0;<a href="#sec%3ASeqIO-index">5.4.2</a>).
</li><li class="li-itemize"><code>Bio.SeqIO.index_db()</code> also acts like a read only dictionary
but stores the identifiers and file offsets in a file on disk (as an
SQLite3 database), meaning it has very low memory requirements (see
Section&#XA0;<a href="#sec%3ASeqIO-index-db">5.4.3</a>), but will be a little bit slower.
</li></ul><p>
See the discussion for an broad overview
(Section&#XA0;<a href="#sec%3ASeqIO-indexing-discussion">5.4.5</a>).</p>
<!--TOC subsection id="sec62" Sequence files as Dictionaries &#X2013; In memory-->
<h3 id="sec62" class="subsection">5.4.1&#XA0;&#XA0;Sequence files as Dictionaries &#X2013; In memory</h3><!--SEC END --><p>
<a id="SeqIO:to_dict"></a></p><p>The next thing that we&#X2019;ll do with our ubiquitous orchid files is to show how
to index them and access them like a database using the Python <code>dictionary</code>
data type (like a hash in Perl). This is very useful for moderately large files
where you only need to access certain elements of the file, and makes for a nice
quick &#X2019;n dirty database. For dealing with larger files where memory becomes a
problem, see Section&#XA0;<a href="#sec%3ASeqIO-index">5.4.2</a> below.</p><p>You can use the function <code>Bio.SeqIO.to_dict()</code> to make a SeqRecord dictionary
(in memory). By default this will use each record&#X2019;s identifier (i.e. the <code>.id</code>
attribute) as the key. Let&#X2019;s try this using our GenBank file:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; orchid_dict = SeqIO.to_dict(SeqIO.parse("ls_orchid.gbk", "genbank"))
</pre><p>There is just one required argument for <code>Bio.SeqIO.to_dict()</code>, a list or
generator giving <code>SeqRecord</code> objects. Here we have just used the output
from the <code>SeqIO.parse</code> function. As the name suggests, this returns a
Python dictionary.</p><p>Since this variable <code>orchid_dict</code> is an ordinary Python dictionary, we can look at all of the keys we have available:</p><pre class="verbatim">&gt;&gt;&gt; len(orchid_dict)
94
</pre><pre class="verbatim">&gt;&gt;&gt; orchid_dict.keys()
['Z78484.1', 'Z78464.1', 'Z78455.1', 'Z78442.1', 'Z78532.1', 'Z78453.1', ..., 'Z78471.1']
</pre><p>If you really want to, you can even look at all the records at once:
</p><pre class="verbatim">&gt;&gt;&gt; orchid_dict.values() #lots of output!
...
</pre><p>We can access a single <code>SeqRecord</code> object via the keys and manipulate the object as normal:</p><pre class="verbatim">&gt;&gt;&gt; seq_record = orchid_dict["Z78475.1"]
&gt;&gt;&gt; print(seq_record.description)
P.supardii 5.8S rRNA gene and ITS1 and ITS2 DNA.
&gt;&gt;&gt; print(repr(seq_record.seq))
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...GGT', IUPACAmbiguousDNA())
</pre><p>So, it is very easy to create an in memory &#X201C;database&#X201D; of our GenBank records. Next we&#X2019;ll try this for the FASTA file instead.</p><p>Note that those of you with prior Python experience should all be able to construct a dictionary like this &#X201C;by hand&#X201D;. However, typical dictionary construction methods will not deal with the case of repeated keys very nicely. Using the <code>Bio.SeqIO.to_dict()</code> will explicitly check for duplicate keys, and raise an exception if any are found.</p>
<!--TOC subsubsection id="sec63" Specifying the dictionary keys-->
<h4 id="sec63" class="subsubsection">5.4.1.1&#XA0;&#XA0;Specifying the dictionary keys</h4><!--SEC END --><p>
<a id="seq:seqio-todict-functionkey"></a></p><p>Using the same code as above, but for the FASTA file instead:</p><pre class="verbatim">from Bio import SeqIO
orchid_dict = SeqIO.to_dict(SeqIO.parse("ls_orchid.fasta", "fasta"))
print(orchid_dict.keys())
</pre><p>This time the keys are:</p><pre class="verbatim">['gi|2765596|emb|Z78471.1|PDZ78471', 'gi|2765646|emb|Z78521.1|CCZ78521', ...
 ..., 'gi|2765613|emb|Z78488.1|PTZ78488', 'gi|2765583|emb|Z78458.1|PHZ78458']
</pre><p>You should recognise these strings from when we parsed the FASTA file earlier in Section&#XA0;<a href="#sec%3Afasta-parsing">2.4.1</a>. Suppose you would rather have something else as the keys - like the accession numbers. This brings us nicely to <code>SeqIO.to_dict()</code>&#X2019;s optional argument <code>key_function</code>, which lets you define what to use as the dictionary key for your records.</p><p>First you must write your own function to return the key you want (as a string) when given a <code>SeqRecord</code> object. In general, the details of function will depend on the sort of input records you are dealing with. But for our orchids, we can just split up the record&#X2019;s identifier using the &#X201C;pipe&#X201D; character (the vertical line) and return the fourth entry (field three):</p><pre class="verbatim">def get_accession(record):
    """"Given a SeqRecord, return the accession number as a string.
  
    e.g. "gi|2765613|emb|Z78488.1|PTZ78488" -&gt; "Z78488.1"
    """
    parts = record.id.split("|")
    assert len(parts) == 5 and parts[0] == "gi" and parts[2] == "emb"
    return parts[3]
</pre><p>Then we can give this function to the <code>SeqIO.to_dict()</code> function to use in building the dictionary:</p><pre class="verbatim">from Bio import SeqIO
orchid_dict = SeqIO.to_dict(SeqIO.parse("ls_orchid.fasta", "fasta"), key_function=get_accession)
print(orchid_dict.keys())
</pre><p>Finally, as desired, the new dictionary keys:</p><pre class="verbatim">&gt;&gt;&gt; print(orchid_dict.keys())
['Z78484.1', 'Z78464.1', 'Z78455.1', 'Z78442.1', 'Z78532.1', 'Z78453.1', ..., 'Z78471.1']
</pre><p>Not too complicated, I hope!</p>
<!--TOC subsubsection id="sec64" Indexing a dictionary using the SEGUID checksum-->
<h4 id="sec64" class="subsubsection">5.4.1.2&#XA0;&#XA0;Indexing a dictionary using the SEGUID checksum</h4><!--SEC END --><p>To give another example of working with dictionaries of <code>SeqRecord</code> objects, we&#X2019;ll use the SEGUID checksum function. This is a relatively recent checksum, and collisions should be very rare (i.e. two different sequences with the same checksum), an improvement on the CRC64 checksum.</p><p>Once again, working with the orchids GenBank file:</p><pre class="verbatim">from Bio import SeqIO
from Bio.SeqUtils.CheckSum import seguid
for record in SeqIO.parse("ls_orchid.gbk", "genbank"):
    print(record.id, seguid(record.seq))
</pre><p>This should give:</p><pre class="verbatim">Z78533.1 JUEoWn6DPhgZ9nAyowsgtoD9TTo
Z78532.1 MN/s0q9zDoCVEEc+k/IFwCNF2pY
...
Z78439.1 H+JfaShya/4yyAj7IbMqgNkxdxQ
</pre><p>Now, recall the <code>Bio.SeqIO.to_dict()</code> function&#X2019;s <code>key_function</code> argument expects a function which turns a <code>SeqRecord</code> into a string. We can&#X2019;t use the <code>seguid()</code> function directly because it expects to be given a <code>Seq</code> object (or a string). However, we can use Python&#X2019;s <code>lambda</code> feature to create a &#X201C;one off&#X201D; function to give to <code>Bio.SeqIO.to_dict()</code> instead:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; from Bio.SeqUtils.CheckSum import seguid
&gt;&gt;&gt; seguid_dict = SeqIO.to_dict(SeqIO.parse("ls_orchid.gbk", "genbank"),
...                             lambda rec : seguid(rec.seq))
&gt;&gt;&gt; record = seguid_dict["MN/s0q9zDoCVEEc+k/IFwCNF2pY"]
&gt;&gt;&gt; print(record.id)
Z78532.1
&gt;&gt;&gt; print(record.description)
C.californicum 5.8S rRNA gene and ITS1 and ITS2 DNA.
</pre><p>That should have retrieved the record <span style="font-family:monospace">Z78532.1</span>, the second entry in the file.</p>
<!--TOC subsection id="sec65" Sequence files as Dictionaries &#X2013; Indexed files-->
<h3 id="sec65" class="subsection">5.4.2&#XA0;&#XA0;Sequence files as Dictionaries &#X2013; Indexed files</h3><!--SEC END --><p>
<a id="sec:SeqIO-index"></a></p><p>As the previous couple of examples tried to illustrate, using
<code>Bio.SeqIO.to_dict()</code> is very flexible. However, because it holds
everything in memory, the size of file you can work with is limited by your
computer&#X2019;s RAM. In general, this will only work on small to medium files.</p><p>For larger files you should consider
<code>Bio.SeqIO.index()</code>, which works a little differently. Although
it still returns a dictionary like object, this does <em>not</em> keep
<em>everything</em> in memory. Instead, it just records where each record
is within the file &#X2013; when you ask for a particular record, it then parses
it on demand.</p><p>As an example, let&#X2019;s use the same GenBank file as before:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; orchid_dict = SeqIO.index("ls_orchid.gbk", "genbank")
&gt;&gt;&gt; len(orchid_dict)
94
</pre><pre class="verbatim">&gt;&gt;&gt; orchid_dict.keys()
['Z78484.1', 'Z78464.1', 'Z78455.1', 'Z78442.1', 'Z78532.1', 'Z78453.1', ..., 'Z78471.1']
</pre><pre class="verbatim">&gt;&gt;&gt; seq_record = orchid_dict["Z78475.1"]
&gt;&gt;&gt; print(seq_record.description)
P.supardii 5.8S rRNA gene and ITS1 and ITS2 DNA.
&gt;&gt;&gt; seq_record.seq
Seq('CGTAACAAGGTTTCCGTAGGTGAACCTGCGGAAGGATCATTGTTGAGATCACAT...GGT', IUPACAmbiguousDNA())
&gt;&gt;&gt; orchid_dict.close()
</pre><p>Note that <code>Bio.SeqIO.index()</code> won&#X2019;t take a handle,
but only a filename. There are good reasons for this, but it is a little
technical. The second argument is the file format (a lower case string as
used in the other <code>Bio.SeqIO</code> functions). You can use many other
simple file formats, including FASTA and FASTQ files (see the example in
Section&#XA0;<a href="#sec%3Afastq-indexing">18.1.11</a>). However, alignment
formats like PHYLIP or Clustal are not supported. Finally as an optional
argument you can supply an alphabet, or a key function.</p><p>Here is the same example using the FASTA file - all we change is the
filename and the format name:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; orchid_dict = SeqIO.index("ls_orchid.fasta", "fasta")
&gt;&gt;&gt; len(orchid_dict)
94
&gt;&gt;&gt; orchid_dict.keys()
['gi|2765596|emb|Z78471.1|PDZ78471', 'gi|2765646|emb|Z78521.1|CCZ78521', ...
 ..., 'gi|2765613|emb|Z78488.1|PTZ78488', 'gi|2765583|emb|Z78458.1|PHZ78458']
</pre>
<!--TOC subsubsection id="sec66" Specifying the dictionary keys-->
<h4 id="sec66" class="subsubsection">5.4.2.1&#XA0;&#XA0;Specifying the dictionary keys</h4><!--SEC END --><p>
<a id="seq:seqio-index-functionkey"></a></p><p>Suppose you want to use the same keys as before? Much like with the
<code>Bio.SeqIO.to_dict()</code> example in Section&#XA0;<a href="#seq%3Aseqio-todict-functionkey">5.4.1.1</a>,
you&#X2019;ll need to write a tiny function to map from the FASTA identifier
(as a string) to the key you want:</p><pre class="verbatim">def get_acc(identifier):
    """"Given a SeqRecord identifier string, return the accession number as a string.
  
    e.g. "gi|2765613|emb|Z78488.1|PTZ78488" -&gt; "Z78488.1"
    """
    parts = identifier.split("|")
    assert len(parts) == 5 and parts[0] == "gi" and parts[2] == "emb"
    return parts[3]
</pre><p>Then we can give this function to the <code>Bio.SeqIO.index()</code>
function to use in building the dictionary:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; orchid_dict = SeqIO.index("ls_orchid.fasta", "fasta", key_function=get_acc)
&gt;&gt;&gt; print(orchid_dict.keys())
['Z78484.1', 'Z78464.1', 'Z78455.1', 'Z78442.1', 'Z78532.1', 'Z78453.1', ..., 'Z78471.1']
</pre><p>Easy when you know how?</p>
<!--TOC subsubsection id="sec67" Getting the raw data for a record-->
<h4 id="sec67" class="subsubsection">5.4.2.2&#XA0;&#XA0;Getting the raw data for a record</h4><!--SEC END --><p>
<a id="sec:seqio-index-getraw"></a></p><p>The dictionary-like object from <code>Bio.SeqIO.index()</code> gives you each
entry as a <code>SeqRecord</code> object. However, it is sometimes useful to
be able to get the original raw data straight from the file. For this
use the <code>get_raw()</code> method which takes a
single argument (the record identifier) and returns a string (extracted
from the file without modification).</p><p>A motivating example is extracting a subset of a records from a large
file where either <code>Bio.SeqIO.write()</code> does not (yet) support the
output file format (e.g. the plain text SwissProt file format) or
where you need to preserve the text exactly (e.g. GenBank or EMBL
output from Biopython does not yet preserve every last bit of
annotation).</p><p>Let&#X2019;s suppose you have download the whole of UniProt in the plain
text SwissPort file format from their FTP site
(<a href="ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.dat.gz"><span style="font-family:monospace">ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.dat.gz</span></a>)
and uncompressed it as the file <code>uniprot_sprot.dat</code>, and you
want to extract just a few records from it:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; uniprot = SeqIO.index("uniprot_sprot.dat", "swiss")
&gt;&gt;&gt; handle = open("selected.dat", "w")
&gt;&gt;&gt; for acc in ["P33487", "P19801", "P13689", "Q8JZQ5", "Q9TRC7"]:
...     handle.write(uniprot.get_raw(acc))
&gt;&gt;&gt; handle.close()
</pre><p>There is a longer example in Section&#XA0;<a href="#sec%3ASeqIO-sort">18.1.5</a> using the
<code>SeqIO.index()</code> function to sort a large sequence file (without
loading everything into memory at once).</p>
<!--TOC subsection id="sec68" Sequence files as Dictionaries &#X2013; Database indexed files-->
<h3 id="sec68" class="subsection">5.4.3&#XA0;&#XA0;Sequence files as Dictionaries &#X2013; Database indexed files</h3><!--SEC END --><p>
<a id="sec:SeqIO-index-db"></a></p><p>Biopython 1.57 introduced an alternative, <code>Bio.SeqIO.index_db()</code>, which
can work on even extremely large files since it stores the record information
as a file on disk (using an SQLite3 database) rather than in memory. Also,
you can index multiple files together (providing all the record identifiers
are unique).</p><p>The <code>Bio.SeqIO.index()</code> function takes three required arguments:
</p><ul class="itemize"><li class="li-itemize">
Index filename, we suggest using something ending <span style="font-family:monospace">.idx</span>.
This index file is actually an SQLite3 database.
</li><li class="li-itemize">List of sequence filenames to index (or a single filename)
</li><li class="li-itemize">File format (lower case string as used in the rest of the
<code>SeqIO</code> module).
</li></ul><p>As an example, consider the GenBank flat file releases from the NCBI FTP site,
<a href="ftp://ftp.ncbi.nih.gov/genbank/"><span style="font-family:monospace">ftp://ftp.ncbi.nih.gov/genbank/</span></a>, which are gzip compressed GenBank files.
As of GenBank release 182, there are 16 files making up the viral sequences,
<span style="font-family:monospace">gbvrl1.seq</span>, &#X2026;, <span style="font-family:monospace">gbvrl16.seq</span>, containing in total almost
one million records. You can index them like this:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; files = ["gbvrl%i.seq" % (i+1) for i in range(16)]
&gt;&gt;&gt; gb_vrl = SeqIO.index_db("gbvrl.idx", files, "genbank")
&gt;&gt;&gt; print("%i sequences indexed" % len(gb_vrl))
958086 sequences indexed
</pre><p>That takes about two minutes to run on my machine. If you rerun it then the
index file (here <span style="font-family:monospace">gbvrl.idx</span>) is reloaded in under a second. You can
use the index as a read only Python dictionary - without having to worry
about which file the sequence comes from, e.g.</p><pre class="verbatim">&gt;&gt;&gt; print(gb_vrl["GQ333173.1"].description)
HIV-1 isolate F12279A1 from Uganda gag protein (gag) gene, partial cds.
</pre>
<!--TOC subsubsection id="sec69" Getting the raw data for a record-->
<h4 id="sec69" class="subsubsection">5.4.3.1&#XA0;&#XA0;Getting the raw data for a record</h4><!--SEC END --><p>Just as with the <code>Bio.SeqIO.index()</code> function discussed above in
Section&#XA0;<a href="#sec%3Aseqio-index-getraw">5.4.2.2</a>, the dictionary like object also lets you
get at the raw text of each record:</p><pre class="verbatim">&gt;&gt;&gt; print(gb_vrl.get_raw("GQ333173.1"))
LOCUS       GQ333173                 459 bp    DNA     linear   VRL 21-OCT-2009
DEFINITION  HIV-1 isolate F12279A1 from Uganda gag protein (gag) gene, partial
            cds.
ACCESSION   GQ333173
...
//
</pre>
<!--TOC subsection id="sec70" Indexing compressed files-->
<h3 id="sec70" class="subsection">5.4.4&#XA0;&#XA0;Indexing compressed files</h3><!--SEC END --><p>
<a id="sec:SeqIO-index-bgzf"></a></p><p>Very often when you are indexing a sequence file it can be quite large &#X2013; so
you may want to compress it on disk. Unfortunately efficient random access
is difficult with the more common file formats like gzip and bzip2. In this
setting, BGZF (Blocked GNU Zip Format) can be very helpful. This is a variant
of gzip (and can be decompressed using standard gzip tools) popularised by
the BAM file format, <a href="http://samtools.sourceforge.net/">samtools</a>, and
<a href="http://samtools.sourceforge.net/tabix.shtml">tabix</a>.</p><p>To create a BGZF compressed file you can use the command line tool <code>bgzip</code>
which comes with samtools. In our examples we use a filename extension
<code>*.bgz</code>, so they can be distinguished from normal gzipped files (named
<code>*.gz</code>). You can also use the <code>Bio.bgzf</code> module to read and write
BGZF files from within Python.</p><p>The <code>Bio.SeqIO.index()</code> and <code>Bio.SeqIO.index_db()</code> can both be
used with BGZF compressed files. For example, if you started with an
uncompressed GenBank file:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; orchid_dict = SeqIO.index("ls_orchid.gbk", "genbank")
&gt;&gt;&gt; len(orchid_dict)
94
&gt;&gt;&gt; orchid_dict.close()
</pre><p>You could compress this (while keeping the original file) at the command
line using the following command &#X2013; but don&#X2019;t worry, the compressed file
is already included with the other example files:</p><pre class="verbatim">$ bgzip -c ls_orchid.gbk &gt; ls_orchid.gbk.bgz
</pre><p>You can use the compressed file in exactly the same way:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; orchid_dict = SeqIO.index("ls_orchid.gbk.bgz", "genbank")
&gt;&gt;&gt; len(orchid_dict)
94
&gt;&gt;&gt; orchid_dict.close()
</pre><p>or:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; orchid_dict = SeqIO.index_db("ls_orchid.gbk.bgz.idx", "ls_orchid.gbk.bgz", "genbank")
&gt;&gt;&gt; len(orchid_dict)
94
&gt;&gt;&gt; orchid_dict.close()
</pre><p>The <code>SeqIO</code> indexing automatically detects the BGZF compression. Note
that you can&#X2019;t use the same index file for the uncompressed and compressed files.</p>
<!--TOC subsection id="sec71" Discussion-->
<h3 id="sec71" class="subsection">5.4.5&#XA0;&#XA0;Discussion</h3><!--SEC END --><p>
<a id="sec:SeqIO-indexing-discussion"></a></p><p>So, which of these methods should you use and why? It depends on what you are
trying to do (and how much data you are dealing with). However, in general
picking <code>Bio.SeqIO.index()</code> is a good starting point. If you are dealing
with millions of records, multiple files, or repeated analyses, then look at
<code>Bio.SeqIO.index_db()</code>.</p><p>Reasons to choose <code>Bio.SeqIO.to_dict()</code> over either
<code>Bio.SeqIO.index()</code> or <code>Bio.SeqIO.index_db()</code> boil down to a need
for flexibility despite its high memory needs. The advantage of storing the
<code>SeqRecord</code> objects in memory is they can be changed, added to, or
removed at will. In addition to the downside of high memory consumption,
indexing can also take longer because all the records must be fully parsed.</p><p>Both <code>Bio.SeqIO.index()</code> and <code>Bio.SeqIO.index_db()</code> only parse
records on demand. When indexing, they scan the file once looking for the
start of each record and do as little work as possible to extract the
identifier.</p><p>Reasons to choose <code>Bio.SeqIO.index()</code> over <code>Bio.SeqIO.index_db()</code>
include:
</p><ul class="itemize"><li class="li-itemize">
Faster to build the index (more noticeable in simple file formats)
</li><li class="li-itemize">Slightly faster access as SeqRecord objects (but the difference is only
really noticeable for simple to parse file formats).
</li><li class="li-itemize">Can use any immutable Python object as the dictionary keys (e.g. a
tuple of strings, or a frozen set) not just strings.
</li><li class="li-itemize">Don&#X2019;t need to worry about the index database being out of date if the
sequence file being indexed has changed.
</li></ul><p>Reasons to choose <code>Bio.SeqIO.index_db()</code> over <code>Bio.SeqIO.index()</code>
include:
</p><ul class="itemize"><li class="li-itemize">
Not memory limited &#X2013; this is already important with files from second
generation sequencing where 10s of millions of sequences are common, and
using <code>Bio.SeqIO.index()</code> can require more than 4GB of RAM and therefore
a 64bit version of Python.
</li><li class="li-itemize">Because the index is kept on disk, it can be reused. Although building
the index database file takes longer, if you have a script which will be
rerun on the same datafiles in future, this could save time in the long run.
</li><li class="li-itemize">Indexing multiple files together
</li><li class="li-itemize">The <code>get_raw()</code> method can be much faster, since for most file
formats the length of each record is stored as well as its offset.
</li></ul>
<!--TOC section id="sec72" Writing Sequence Files-->
<h2 id="sec72" class="section">5.5&#XA0;&#XA0;Writing Sequence Files</h2><!--SEC END --><p>We&#X2019;ve talked about using <code>Bio.SeqIO.parse()</code> for sequence input (reading files), and now we&#X2019;ll look at <code>Bio.SeqIO.write()</code> which is for sequence output (writing files). This is a function taking three arguments: some <code>SeqRecord</code> objects, a handle or filename to write to, and a sequence format.</p><p>Here is an example, where we start by creating a few <code>SeqRecord</code> objects the hard way (by hand, rather than by loading them from a file):</p><pre class="verbatim">from Bio.Seq import Seq
from Bio.SeqRecord import SeqRecord
from Bio.Alphabet import generic_protein

rec1 = SeqRecord(Seq("MMYQQGCFAGGTVLRLAKDLAENNRGARVLVVCSEITAVTFRGPSETHLDSMVGQALFGD" \
                    +"GAGAVIVGSDPDLSVERPLYELVWTGATLLPDSEGAIDGHLREVGLTFHLLKDVPGLISK" \
                    +"NIEKSLKEAFTPLGISDWNSTFWIAHPGGPAILDQVEAKLGLKEEKMRATREVLSEYGNM" \
                    +"SSAC", generic_protein),
                 id="gi|14150838|gb|AAK54648.1|AF376133_1",
                 description="chalcone synthase [Cucumis sativus]")

rec2 = SeqRecord(Seq("YPDYYFRITNREHKAELKEKFQRMCDKSMIKKRYMYLTEEILKENPSMCEYMAPSLDARQ" \
                    +"DMVVVEIPKLGKEAAVKAIKEWGQ", generic_protein),
                 id="gi|13919613|gb|AAK33142.1|",
                 description="chalcone synthase [Fragaria vesca subsp. bracteata]")

rec3 = SeqRecord(Seq("MVTVEEFRRAQCAEGPATVMAIGTATPSNCVDQSTYPDYYFRITNSEHKVELKEKFKRMC" \
                    +"EKSMIKKRYMHLTEEILKENPNICAYMAPSLDARQDIVVVEVPKLGKEAAQKAIKEWGQP" \
                    +"KSKITHLVFCTTSGVDMPGCDYQLTKLLGLRPSVKRFMMYQQGCFAGGTVLRMAKDLAEN" \
                    +"NKGARVLVVCSEITAVTFRGPNDTHLDSLVGQALFGDGAAAVIIGSDPIPEVERPLFELV" \
                    +"SAAQTLLPDSEGAIDGHLREVGLTFHLLKDVPGLISKNIEKSLVEAFQPLGISDWNSLFW" \
                    +"IAHPGGPAILDQVELKLGLKQEKLKATRKVLSNYGNMSSACVLFILDEMRKASAKEGLGT" \
                    +"TGEGLEWGVLFGFGPGLTVETVVLHSVAT", generic_protein),
                 id="gi|13925890|gb|AAK49457.1|",
                 description="chalcone synthase [Nicotiana tabacum]")
               
my_records = [rec1, rec2, rec3]
</pre><p>Now we have a list of <code>SeqRecord</code> objects, we&#X2019;ll write them to a FASTA format file:</p><pre class="verbatim">from Bio import SeqIO
SeqIO.write(my_records, "my_example.faa", "fasta")
</pre><p>And if you open this file in your favourite text editor it should look like this:</p><pre class="verbatim">&gt;gi|14150838|gb|AAK54648.1|AF376133_1 chalcone synthase [Cucumis sativus]
MMYQQGCFAGGTVLRLAKDLAENNRGARVLVVCSEITAVTFRGPSETHLDSMVGQALFGD
GAGAVIVGSDPDLSVERPLYELVWTGATLLPDSEGAIDGHLREVGLTFHLLKDVPGLISK
NIEKSLKEAFTPLGISDWNSTFWIAHPGGPAILDQVEAKLGLKEEKMRATREVLSEYGNM
SSAC
&gt;gi|13919613|gb|AAK33142.1| chalcone synthase [Fragaria vesca subsp. bracteata]
YPDYYFRITNREHKAELKEKFQRMCDKSMIKKRYMYLTEEILKENPSMCEYMAPSLDARQ
DMVVVEIPKLGKEAAVKAIKEWGQ
&gt;gi|13925890|gb|AAK49457.1| chalcone synthase [Nicotiana tabacum]
MVTVEEFRRAQCAEGPATVMAIGTATPSNCVDQSTYPDYYFRITNSEHKVELKEKFKRMC
EKSMIKKRYMHLTEEILKENPNICAYMAPSLDARQDIVVVEVPKLGKEAAQKAIKEWGQP
KSKITHLVFCTTSGVDMPGCDYQLTKLLGLRPSVKRFMMYQQGCFAGGTVLRMAKDLAEN
NKGARVLVVCSEITAVTFRGPNDTHLDSLVGQALFGDGAAAVIIGSDPIPEVERPLFELV
SAAQTLLPDSEGAIDGHLREVGLTFHLLKDVPGLISKNIEKSLVEAFQPLGISDWNSLFW
IAHPGGPAILDQVELKLGLKQEKLKATRKVLSNYGNMSSACVLFILDEMRKASAKEGLGT
TGEGLEWGVLFGFGPGLTVETVVLHSVAT
</pre><p>Suppose you wanted to know how many records the <code>Bio.SeqIO.write()</code> function wrote to the handle?
If your records were in a list you could just use <code>len(my_records)</code>, however you can&#X2019;t do that when your records come from a generator/iterator. The <code>Bio.SeqIO.write()</code> function returns the number of <code>SeqRecord</code> objects written to the file. </p><p><em>Note</em> - If you tell the <code>Bio.SeqIO.write()</code> function to write to a file that already exists, the old file will be overwritten without any warning.</p>
<!--TOC subsection id="sec73" Round trips-->
<h3 id="sec73" class="subsection">5.5.1&#XA0;&#XA0;Round trips</h3><!--SEC END --><p>Some people like their parsers to be &#X201C;round-tripable&#X201D;, meaning if you read in
a file and write it back out again it is unchanged. This requires that the parser
must extract enough information to reproduce the original file <em>exactly</em>.
<code>Bio.SeqIO</code> does <em>not</em> aim to do this.</p><p>As a trivial example, any line wrapping of the sequence data in FASTA files is
allowed. An identical <code>SeqRecord</code> would be given from parsing the following
two examples which differ only in their line breaks:</p><pre class="verbatim">&gt;YAL068C-7235.2170 Putative promoter sequence
TACGAGAATAATTTCTCATCATCCAGCTTTAACACAAAATTCGCACAGTTTTCGTTAAGA
GAACTTAACATTTTCTTATGACGTAAATGAAGTTTATATATAAATTTCCTTTTTATTGGA

&gt;YAL068C-7235.2170 Putative promoter sequence
TACGAGAATAATTTCTCATCATCCAGCTTTAACACAAAATTCGCA
CAGTTTTCGTTAAGAGAACTTAACATTTTCTTATGACGTAAATGA
AGTTTATATATAAATTTCCTTTTTATTGGA
</pre><p>To make a round-tripable FASTA parser you would need to keep track of where the
sequence line breaks occurred, and this extra information is usually pointless.
Instead Biopython uses a default line wrapping of 60 characters on output.
The same problem with white space applies in many other file formats too.
Another issue in some cases is that Biopython does not (yet) preserve every
last bit of annotation (e.g. GenBank and EMBL).</p><p>Occasionally preserving the original layout (with any quirks it may have) is
important. See Section&#XA0;<a href="#sec%3Aseqio-index-getraw">5.4.2.2</a> about the <code>get_raw()</code>
method of the <code>Bio.SeqIO.index()</code> dictionary-like object for one potential
solution.</p>
<!--TOC subsection id="sec74" Converting between sequence file formats-->
<h3 id="sec74" class="subsection">5.5.2&#XA0;&#XA0;Converting between sequence file formats</h3><!--SEC END --><p>
<a id="sec:SeqIO-conversion"></a></p><p>In previous example we used a list of <code>SeqRecord</code> objects as input to the <code>Bio.SeqIO.write()</code> function, but it will also accept a <code>SeqRecord</code> iterator like we get from <code>Bio.SeqIO.parse()</code> &#X2013; this lets us do file conversion by combining these two functions.</p><p>For this example we&#X2019;ll read in the GenBank format file <a href="http://biopython.org/DIST/docs/tutorial/examples/ls_orchid.gbk">ls_orchid.gbk</a> and write it out in FASTA format:</p><pre class="verbatim">from Bio import SeqIO
records = SeqIO.parse("ls_orchid.gbk", "genbank")
count = SeqIO.write(records, "my_example.fasta", "fasta")
print("Converted %i records" % count)
</pre><p>Still, that is a little bit complicated. So, because file conversion is such a
common task, there is a helper function letting you replace that with just:</p><pre class="verbatim">from Bio import SeqIO
count = SeqIO.convert("ls_orchid.gbk", "genbank", "my_example.fasta", "fasta")
print("Converted %i records" % count)
</pre><p>The <code>Bio.SeqIO.convert()</code> function will take handles <em>or</em> filenames.
Watch out though &#X2013; if the output file already exists, it will overwrite it!
To find out more, see the built in help:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; help(SeqIO.convert)
...
</pre><p>In principle, just by changing the filenames and the format names, this code
could be used to convert between any file formats available in Biopython.
However, writing some formats requires information (e.g. quality scores) which
other files formats don&#X2019;t contain. For example, while you can turn a FASTQ
file into a FASTA file, you can&#X2019;t do the reverse. See also
Sections&#XA0;<a href="#sec%3ASeqIO-fastq-conversion">18.1.9</a> and&#XA0;<a href="#sec%3ASeqIO-fasta-qual-conversion">18.1.10</a>
in the cookbook chapter which looks at inter-converting between different FASTQ formats.</p><p>Finally, as an added incentive for using the <code>Bio.SeqIO.convert()</code> function
(on top of the fact your code will be shorter), doing it this way may also be
faster! The reason for this is the convert function can take advantage of
several file format specific optimisations and tricks.</p>
<!--TOC subsection id="sec75" Converting a file of sequences to their reverse complements-->
<h3 id="sec75" class="subsection">5.5.3&#XA0;&#XA0;Converting a file of sequences to their reverse complements</h3><!--SEC END --><p>
<a id="sec:SeqIO-reverse-complement"></a></p><p>Suppose you had a file of nucleotide sequences, and you wanted to turn it into a file containing their reverse complement sequences. This time a little bit of work is required to transform the <code>SeqRecord</code> objects we get from our input file into something suitable for saving to our output file.</p><p>To start with, we&#X2019;ll use <code>Bio.SeqIO.parse()</code> to load some nucleotide
sequences from a file, then print out their reverse complements using
the <code>Seq</code> object&#X2019;s built in <code>.reverse_complement()</code> method (see Section&#XA0;<a href="#sec%3Aseq-reverse-complement">3.7</a>):</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; for record in SeqIO.parse("ls_orchid.gbk", "genbank"):
...     print(record.id)
...     print(record.seq.reverse_complement())
</pre><p>Now, if we want to save these reverse complements to a file, we&#X2019;ll need to make <code>SeqRecord</code> objects.
We can use the <code>SeqRecord</code> object&#X2019;s built in <code>.reverse_complement()</code> method (see Section&#XA0;<a href="#sec%3ASeqRecord-reverse-complement">4.8</a>) but we must decide how to name our new records.</p><p>This is an excellent place to demonstrate the power of list comprehensions which make a list in memory:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; records = [rec.reverse_complement(id="rc_"+rec.id, description = "reverse complement") \
...            for rec in SeqIO.parse("ls_orchid.fasta", "fasta")]
&gt;&gt;&gt; len(records)
94
</pre><p>Now list comprehensions have a nice trick up their sleeves, you can add a conditional statement:</p><pre class="verbatim">&gt;&gt;&gt; records = [rec.reverse_complement(id="rc_"+rec.id, description = "reverse complement") \
...            for rec in SeqIO.parse("ls_orchid.fasta", "fasta") if len(rec)&lt;700]
&gt;&gt;&gt; len(records)
18
</pre><p>That would create an in memory list of reverse complement records where the sequence length was under 700 base pairs. However, we can do exactly the same with a generator expression - but with the advantage that this does not create a list of all the records in memory at once:</p><pre class="verbatim">&gt;&gt;&gt; records = (rec.reverse_complement(id="rc_"+rec.id, description = "reverse complement") \
...           for rec in SeqIO.parse("ls_orchid.fasta", "fasta") if len(rec)&lt;700)
</pre><p>As a complete example:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; records = (rec.reverse_complement(id="rc_"+rec.id, description = "reverse complement") \
...            for rec in SeqIO.parse("ls_orchid.fasta", "fasta") if len(rec)&lt;700)
&gt;&gt;&gt; SeqIO.write(records, "rev_comp.fasta", "fasta")
18
</pre><p>There is a related example in Section&#XA0;<a href="#sec%3ASeqIO-translate">18.1.3</a>, translating each
record in a FASTA file from nucleotides to amino acids.</p>
<!--TOC subsection id="sec76" Getting your SeqRecord objects as formatted strings-->
<h3 id="sec76" class="subsection">5.5.4&#XA0;&#XA0;Getting your SeqRecord objects as formatted strings</h3><!--SEC END --><p>
<a id="sec:Bio.SeqIO-and-StringIO"></a>
Suppose that you don&#X2019;t really want to write your records to a file or handle &#X2013; instead you want a string containing the records in a particular file format. The <code>Bio.SeqIO</code> interface is based on handles, but Python has a useful built in module which provides a string based handle.</p><p>For an example of how you might use this, let&#X2019;s load in a bunch of <code>SeqRecord</code> objects from our orchids GenBank file, and create a string containing the records in FASTA format:</p><pre class="verbatim">from Bio import SeqIO
from StringIO import StringIO
records = SeqIO.parse("ls_orchid.gbk", "genbank")
out_handle = StringIO()
SeqIO.write(records, out_handle, "fasta")
fasta_data = out_handle.getvalue()
print(fasta_data)
</pre><p>This isn&#X2019;t entirely straightforward the first time you see it! On the bright side, for the special case where you would like a string containing a <em>single</em> record in a particular file format, use the the <code>SeqRecord</code> class&#X2019; <code>format()</code> method (see Section&#XA0;<a href="#sec%3ASeqRecord-format">4.5</a>).</p><p>Note that although we don&#X2019;t encourage it, you <em>can</em> use the <code>format()</code> method to write to a file, for example something like this:
</p><pre class="verbatim">from Bio import SeqIO
out_handle = open("ls_orchid_long.tab", "w")
for record in SeqIO.parse("ls_orchid.gbk", "genbank"):
    if len(record) &gt; 100:
        out_handle.write(record.format("tab"))
out_handle.close()
</pre><p>While this style of code will work for a simple sequential file format like FASTA or the simple tab separated format used here, it will <em>not</em> work for more complex or interlaced file formats. This is why we still recommend using <code>Bio.SeqIO.write()</code>, as in the following example:
</p><pre class="verbatim">from Bio import SeqIO
records = (rec for rec in SeqIO.parse("ls_orchid.gbk", "genbank") if len(rec) &gt; 100)
SeqIO.write(records, "ls_orchid.tab", "tab")
</pre><p>Making a single call to <code>SeqIO.write(...)</code> is also much quicker than
multiple calls to the <code>SeqRecord.format(...)</code> method.</p>
<!--TOC chapter id="sec77" Multiple Sequence Alignment objects-->
<h1 id="sec77" class="chapter">Chapter&#XA0;6&#XA0;&#XA0;Multiple Sequence Alignment objects</h1><!--SEC END --><p>
<a id="chapter:Bio.AlignIO"></a></p><p>This chapter is about Multiple Sequence Alignments, by which we mean a collection of
multiple sequences which have been aligned together &#X2013; usually with the insertion of gap
characters, and addition of leading or trailing gaps &#X2013; such that all the sequence
strings are the same length. Such an alignment can be regarded as a matrix of letters,
where each row is held as a <code>SeqRecord</code> object internally.</p><p>We will introduce the <code>MultipleSeqAlignment</code> object which holds this kind of data,
and the <code>Bio.AlignIO</code> module for reading and writing them as various file formats
(following the design of the <code>Bio.SeqIO</code> module from the previous chapter).
Note that both <code>Bio.SeqIO</code> and <code>Bio.AlignIO</code> can read and write sequence
alignment files. The appropriate choice will depend largely on what you want to do
with the data.</p><p>The final part of this chapter is about our command line wrappers for common multiple
sequence alignment tools like ClustalW and MUSCLE.</p>
<!--TOC section id="sec78" Parsing or Reading Sequence Alignments-->
<h2 id="sec78" class="section">6.1&#XA0;&#XA0;Parsing or Reading Sequence Alignments</h2><!--SEC END --><p>We have two functions for reading in sequence alignments, <code>Bio.AlignIO.read()</code> and <code>Bio.AlignIO.parse()</code> which following the convention introduced in <code>Bio.SeqIO</code> are for files containing one or multiple alignments respectively.</p><p>Using <code>Bio.AlignIO.parse()</code> will return an <span style="font-style:italic">iterator</span> which gives <code>MultipleSeqAlignment</code> objects. Iterators are typically used in a for loop. Examples of situations where you will have multiple different alignments include resampled alignments from the PHYLIP tool <code>seqboot</code>, or multiple pairwise alignments from the EMBOSS tools <code>water</code> or <code>needle</code>, or Bill Pearson&#X2019;s FASTA tools.</p><p>However, in many situations you will be dealing with files which contain only a single alignment. In this case, you should use the <code>Bio.AlignIO.read()</code> function which returns a single <code>MultipleSeqAlignment</code> object.</p><p>Both functions expect two mandatory arguments:</p><ol class="enumerate" type=1><li class="li-enumerate">
The first argument is a <span style="font-style:italic">handle</span> to read the data from, typically an open file (see Section&#XA0;<a href="#sec%3Aappendix-handles">22.1</a>), or a filename.
</li><li class="li-enumerate">The second argument is a lower case string specifying the alignment format. As in <code>Bio.SeqIO</code> we don&#X2019;t try and guess the file format for you! See <a href="http://biopython.org/wiki/AlignIO"><span style="font-family:monospace">http://biopython.org/wiki/AlignIO</span></a> for a full listing of supported formats.
</li></ol><p>There is also an optional <code>seq_count</code> argument which is discussed in Section&#XA0;<a href="#sec%3AAlignIO-count-argument">6.1.3</a> below for dealing with ambiguous file formats which may contain more than one alignment.</p><p>A further optional <code>alphabet</code> argument allowing you to specify the expected alphabet. This can be useful as many alignment file formats do not explicitly label the sequences as RNA, DNA or protein &#X2013; which means <code>Bio.AlignIO</code> will default to using a generic alphabet.</p>
<!--TOC subsection id="sec79" Single Alignments-->
<h3 id="sec79" class="subsection">6.1.1&#XA0;&#XA0;Single Alignments</h3><!--SEC END --><p>
As an example, consider the following annotation rich protein alignment in the PFAM or Stockholm file format:</p><pre class="verbatim"># STOCKHOLM 1.0
#=GS COATB_BPIKE/30-81  AC P03620.1
#=GS COATB_BPIKE/30-81  DR PDB; 1ifl ; 1-52;
#=GS Q9T0Q8_BPIKE/1-52  AC Q9T0Q8.1
#=GS COATB_BPI22/32-83  AC P15416.1
#=GS COATB_BPM13/24-72  AC P69541.1
#=GS COATB_BPM13/24-72  DR PDB; 2cpb ; 1-49;
#=GS COATB_BPM13/24-72  DR PDB; 2cps ; 1-49;
#=GS COATB_BPZJ2/1-49   AC P03618.1
#=GS Q9T0Q9_BPFD/1-49   AC Q9T0Q9.1
#=GS Q9T0Q9_BPFD/1-49   DR PDB; 1nh4 A; 1-49;
#=GS COATB_BPIF1/22-73  AC P03619.2
#=GS COATB_BPIF1/22-73  DR PDB; 1ifk ; 1-50;
COATB_BPIKE/30-81             AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIRLFKKFSSKA
#=GR COATB_BPIKE/30-81  SS    -HHHHHHHHHHHHHH--HHHHHHHH--HHHHHHHHHHHHHHHHHHHHH----
Q9T0Q8_BPIKE/1-52             AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIKLFKKFVSRA
COATB_BPI22/32-83             DGTSTATSYATEAMNSLKTQATDLIDQTWPVVTSVAVAGLAIRLFKKFSSKA
COATB_BPM13/24-72             AEGDDP...AKAAFNSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA
#=GR COATB_BPM13/24-72  SS    ---S-T...CHCHHHHCCCCTCCCTTCHHHHHHHHHHHHHHHHHHHHCTT--
COATB_BPZJ2/1-49              AEGDDP...AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFASKA
Q9T0Q9_BPFD/1-49              AEGDDP...AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA
#=GR Q9T0Q9_BPFD/1-49   SS    ------...-HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH--
COATB_BPIF1/22-73             FAADDATSQAKAAFDSLTAQATEMSGYAWALVVLVVGATVGIKLFKKFVSRA
#=GR COATB_BPIF1/22-73  SS    XX-HHHH--HHHHHH--HHHHHHH--HHHHHHHHHHHHHHHHHHHHHHH---
#=GC SS_cons                  XHHHHHHHHHHHHHHHCHHHHHHHHCHHHHHHHHHHHHHHHHHHHHHHHC--
#=GC seq_cons                 AEssss...AptAhDSLpspAT-hIu.sWshVsslVsAsluIKLFKKFsSKA
//
</pre><p>This is the seed alignment for the Phage_Coat_Gp8 (PF05371) PFAM entry, downloaded from a now out of date release of PFAM from <a href="http://pfam.sanger.ac.uk/"><span style="font-family:monospace">http://pfam.sanger.ac.uk/</span></a>. We can load this file as follows (assuming it has been saved to disk as &#X201C;PF05371_seed.sth&#X201D; in the current working directory):</p><pre class="verbatim">&gt;&gt;&gt; from Bio import AlignIO
&gt;&gt;&gt; alignment = AlignIO.read("PF05371_seed.sth", "stockholm")
</pre><p>This code will print out a summary of the alignment:</p><pre class="verbatim">&gt;&gt;&gt; print(alignment)
SingleLetterAlphabet() alignment with 7 rows and 52 columns
AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIRL...SKA COATB_BPIKE/30-81
AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIKL...SRA Q9T0Q8_BPIKE/1-52
DGTSTATSYATEAMNSLKTQATDLIDQTWPVVTSVAVAGLAIRL...SKA COATB_BPI22/32-83
AEGDDP---AKAAFNSLQASATEYIGYAWAMVVVIVGATIGIKL...SKA COATB_BPM13/24-72
AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKL...SKA COATB_BPZJ2/1-49
AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKL...SKA Q9T0Q9_BPFD/1-49
FAADDATSQAKAAFDSLTAQATEMSGYAWALVVLVVGATVGIKL...SRA COATB_BPIF1/22-73
</pre><p>You&#X2019;ll notice in the above output the sequences have been truncated. We could instead write our own code to format this as we please by iterating over the rows as <code>SeqRecord</code> objects:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import AlignIO
&gt;&gt;&gt; alignment = AlignIO.read("PF05371_seed.sth", "stockholm")
&gt;&gt;&gt; print("Alignment length %i" % alignment.get_alignment_length())
Alignment length 52
&gt;&gt;&gt; for record in alignment:
...     print("%s - %s" % (record.seq, record.id))
AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIRLFKKFSSKA - COATB_BPIKE/30-81
AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIKLFKKFVSRA - Q9T0Q8_BPIKE/1-52
DGTSTATSYATEAMNSLKTQATDLIDQTWPVVTSVAVAGLAIRLFKKFSSKA - COATB_BPI22/32-83
AEGDDP---AKAAFNSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA - COATB_BPM13/24-72
AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFASKA - COATB_BPZJ2/1-49
AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA - Q9T0Q9_BPFD/1-49
FAADDATSQAKAAFDSLTAQATEMSGYAWALVVLVVGATVGIKLFKKFVSRA - COATB_BPIF1/22-73
</pre><p>You could also use the alignment object&#X2019;s <code>format</code> method to show it in a particular file format &#X2013; see Section&#XA0;<a href="#sec%3Aalignment-format-method">6.2.2</a> for details.</p><p>Did you notice in the raw file above that several of the sequences include database cross-references to the PDB and the associated known secondary structure? Try this:</p><pre class="verbatim">&gt;&gt;&gt; for record in alignment:
...     if record.dbxrefs:
...         print("%s %s" % (record.id, record.dbxrefs))
COATB_BPIKE/30-81 ['PDB; 1ifl ; 1-52;']
COATB_BPM13/24-72 ['PDB; 2cpb ; 1-49;', 'PDB; 2cps ; 1-49;']
Q9T0Q9_BPFD/1-49 ['PDB; 1nh4 A; 1-49;']
COATB_BPIF1/22-73 ['PDB; 1ifk ; 1-50;']
</pre><p>To have a look at all the sequence annotation, try this:</p><pre class="verbatim">&gt;&gt;&gt; for record in alignment:
...     print(record)
</pre><p>Sanger provide a nice web interface at <a href="http://pfam.sanger.ac.uk/family?acc=PF05371"><span style="font-family:monospace">http://pfam.sanger.ac.uk/family?acc=PF05371</span></a> which will actually let you download this alignment in several other formats. This is what the file looks like in the FASTA file format:</p><pre class="verbatim">&gt;COATB_BPIKE/30-81
AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIRLFKKFSSKA
&gt;Q9T0Q8_BPIKE/1-52
AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIKLFKKFVSRA
&gt;COATB_BPI22/32-83
DGTSTATSYATEAMNSLKTQATDLIDQTWPVVTSVAVAGLAIRLFKKFSSKA
&gt;COATB_BPM13/24-72
AEGDDP---AKAAFNSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA
&gt;COATB_BPZJ2/1-49
AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFASKA
&gt;Q9T0Q9_BPFD/1-49
AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA
&gt;COATB_BPIF1/22-73
FAADDATSQAKAAFDSLTAQATEMSGYAWALVVLVVGATVGIKLFKKFVSRA
</pre><p>Note the website should have an option about showing gaps as periods (dots) or dashes, we&#X2019;ve shown dashes above. Assuming you download and save this as file &#X201C;PF05371_seed.faa&#X201D; then you can load it with almost exactly the same code:</p><pre class="verbatim">from Bio import AlignIO
alignment = AlignIO.read("PF05371_seed.faa", "fasta")
print(alignment)
</pre><p>All that has changed in this code is the filename and the format string. You&#X2019;ll get the same output as before, the sequences and record identifiers are the same.
However, as you should expect, if you check each <code>SeqRecord</code> there is no annotation nor database cross-references because these are not included in the FASTA file format.</p><p>Note that rather than using the Sanger website, you could have used <code>Bio.AlignIO</code> to convert the original Stockholm format file into a FASTA file yourself (see below).</p><p>With any supported file format, you can load an alignment in exactly the same way just by changing the format string. For example, use &#X201C;phylip&#X201D; for PHYLIP files, &#X201C;nexus&#X201D; for NEXUS files or &#X201C;emboss&#X201D; for the alignments output by the EMBOSS tools. There is a full listing on the wiki page (<a href="http://biopython.org/wiki/AlignIO"><span style="font-family:monospace">http://biopython.org/wiki/AlignIO</span></a>) and in the built in documentation (also <a href="http://biopython.org/DIST/docs/api/Bio.AlignIO-module.html">online</a>):</p><pre class="verbatim">&gt;&gt;&gt; from Bio import AlignIO
&gt;&gt;&gt; help(AlignIO)
...
</pre>
<!--TOC subsection id="sec80" Multiple Alignments-->
<h3 id="sec80" class="subsection">6.1.2&#XA0;&#XA0;Multiple Alignments</h3><!--SEC END --><p>The previous section focused on reading files containing a single alignment. In general however, files can contain more than one alignment, and to read these files we must use the <code>Bio.AlignIO.parse()</code> function.</p><p>Suppose you have a small alignment in PHYLIP format:</p><pre class="verbatim">    5    6
Alpha     AACAAC
Beta      AACCCC
Gamma     ACCAAC
Delta     CCACCA
Epsilon   CCAAAC
</pre><p>If you wanted to bootstrap a phylogenetic tree using the PHYLIP tools, one of the steps would be to create a set of many resampled alignments using the tool <code>bootseq</code>. This would give output something like this, which has been abbreviated for conciseness:</p><pre class="verbatim">    5     6
Alpha     AAACCA
Beta      AAACCC
Gamma     ACCCCA
Delta     CCCAAC
Epsilon   CCCAAA
    5     6
Alpha     AAACAA
Beta      AAACCC
Gamma     ACCCAA
Delta     CCCACC
Epsilon   CCCAAA
    5     6
Alpha     AAAAAC
Beta      AAACCC
Gamma     AACAAC
Delta     CCCCCA
Epsilon   CCCAAC
...
    5     6
Alpha     AAAACC
Beta      ACCCCC
Gamma     AAAACC
Delta     CCCCAA
Epsilon   CAAACC
</pre><p>If you wanted to read this in using <code>Bio.AlignIO</code> you could use:</p><pre class="verbatim">from Bio import AlignIO
alignments = AlignIO.parse("resampled.phy", "phylip")
for alignment in alignments:
    print(alignment)
    print("")
</pre><p>This would give the following output, again abbreviated for display:</p><pre class="verbatim">SingleLetterAlphabet() alignment with 5 rows and 6 columns
AAACCA Alpha
AAACCC Beta
ACCCCA Gamma
CCCAAC Delta
CCCAAA Epsilon

SingleLetterAlphabet() alignment with 5 rows and 6 columns
AAACAA Alpha
AAACCC Beta
ACCCAA Gamma
CCCACC Delta
CCCAAA Epsilon

SingleLetterAlphabet() alignment with 5 rows and 6 columns
AAAAAC Alpha
AAACCC Beta
AACAAC Gamma
CCCCCA Delta
CCCAAC Epsilon

...

SingleLetterAlphabet() alignment with 5 rows and 6 columns
AAAACC Alpha
ACCCCC Beta
AAAACC Gamma
CCCCAA Delta
CAAACC Epsilon
</pre><p>As with the function <code>Bio.SeqIO.parse()</code>, using <code>Bio.AlignIO.parse()</code> returns an iterator.
If you want to keep all the alignments in memory at once, which will allow you to access them in any order, then turn the iterator into a list:</p><pre class="verbatim">from Bio import AlignIO
alignments = list(AlignIO.parse("resampled.phy", "phylip"))
last_align = alignments[-1]
first_align = alignments[0]
</pre>
<!--TOC subsection id="sec81" Ambiguous Alignments-->
<h3 id="sec81" class="subsection">6.1.3&#XA0;&#XA0;Ambiguous Alignments</h3><!--SEC END --><p>
<a id="sec:AlignIO-count-argument"></a>
Many alignment file formats can explicitly store more than one alignment, and the division between each alignment is clear. However, when a general sequence file format has been used there is no such block structure. The most common such situation is when alignments have been saved in the FASTA file format. For example consider the following:</p><pre class="verbatim">&gt;Alpha
ACTACGACTAGCTCAG--G
&gt;Beta
ACTACCGCTAGCTCAGAAG
&gt;Gamma
ACTACGGCTAGCACAGAAG
&gt;Alpha
ACTACGACTAGCTCAGG--
&gt;Beta
ACTACCGCTAGCTCAGAAG
&gt;Gamma
ACTACGGCTAGCACAGAAG
</pre><p>This could be a single alignment containing six sequences (with repeated identifiers). Or, judging from the identifiers, this is probably two different alignments each with three sequences, which happen to all have the same length.</p><p>What about this next example?</p><pre class="verbatim">&gt;Alpha
ACTACGACTAGCTCAG--G
&gt;Beta
ACTACCGCTAGCTCAGAAG
&gt;Alpha
ACTACGACTAGCTCAGG--
&gt;Gamma
ACTACGGCTAGCACAGAAG
&gt;Alpha
ACTACGACTAGCTCAGG--
&gt;Delta
ACTACGGCTAGCACAGAAG
</pre><p>Again, this could be a single alignment with six sequences. However this time based on the identifiers we might guess this is three pairwise alignments which by chance have all got the same lengths.</p><p>This final example is similar:</p><pre class="verbatim">&gt;Alpha
ACTACGACTAGCTCAG--G
&gt;XXX
ACTACCGCTAGCTCAGAAG
&gt;Alpha
ACTACGACTAGCTCAGG
&gt;YYY
ACTACGGCAAGCACAGG
&gt;Alpha
--ACTACGAC--TAGCTCAGG
&gt;ZZZ
GGACTACGACAATAGCTCAGG
</pre><p>In this third example, because of the differing lengths, this cannot be treated as a single alignment containing all six records. However, it could be three pairwise alignments.</p><p>Clearly trying to store more than one alignment in a FASTA file is not ideal. However, if you are forced to deal with these as input files <code>Bio.AlignIO</code> can cope with the most common situation where all the alignments have the same number of records.
One example of this is a collection of pairwise alignments, which can be produced by the EMBOSS tools <code>needle</code> and <code>water</code> &#X2013; although in this situation, <code>Bio.AlignIO</code> should be able to understand their native output using &#X201C;emboss&#X201D; as the format string.</p><p>To interpret these FASTA examples as several separate alignments, we can use <code>Bio.AlignIO.parse()</code> with the optional <code>seq_count</code> argument which specifies how many sequences are expected in each alignment (in these examples, 3, 2 and 2 respectively).
For example, using the third example as the input data:</p><pre class="verbatim">for alignment in AlignIO.parse(handle, "fasta", seq_count=2):
    print("Alignment length %i" % alignment.get_alignment_length())
    for record in alignment:
        print("%s - %s" % (record.seq, record.id))
    print("")
</pre><p>giving:</p><pre class="verbatim">Alignment length 19
ACTACGACTAGCTCAG--G - Alpha
ACTACCGCTAGCTCAGAAG - XXX

Alignment length 17
ACTACGACTAGCTCAGG - Alpha
ACTACGGCAAGCACAGG - YYY

Alignment length 21
--ACTACGAC--TAGCTCAGG - Alpha
GGACTACGACAATAGCTCAGG - ZZZ
</pre><p>Using <code>Bio.AlignIO.read()</code> or <code>Bio.AlignIO.parse()</code> without the <code>seq_count</code> argument would give a single alignment containing all six records for the first two examples. For the third example, an exception would be raised because the lengths differ preventing them being turned into a single alignment.</p><p>If the file format itself has a block structure allowing <code>Bio.AlignIO</code> to determine the number of sequences in each alignment directly, then the <code>seq_count</code> argument is not needed. If it is supplied, and doesn&#X2019;t agree with the file contents, an error is raised.</p><p>Note that this optional <code>seq_count</code> argument assumes each alignment in the file has the same number of sequences. Hypothetically you may come across stranger situations, for example a FASTA file containing several alignments each with a different number of sequences &#X2013; although I would love to hear of a real world example of this. Assuming you cannot get the data in a nicer file format, there is no straight forward way to deal with this using <code>Bio.AlignIO</code>. In this case, you could consider reading in the sequences themselves using <code>Bio.SeqIO</code> and batching them together to create the alignments as appropriate.</p>
<!--TOC section id="sec82" Writing Alignments-->
<h2 id="sec82" class="section">6.2&#XA0;&#XA0;Writing Alignments</h2><!--SEC END --><p>We&#X2019;ve talked about using <code>Bio.AlignIO.read()</code> and <code>Bio.AlignIO.parse()</code> for alignment input (reading files), and now we&#X2019;ll look at <code>Bio.AlignIO.write()</code> which is for alignment output (writing files). This is a function taking three arguments: some <code>MultipleSeqAlignment</code> objects (or for backwards compatibility the obsolete <code>Alignment</code> objects), a handle or filename to write to, and a sequence format.</p><p>Here is an example, where we start by creating a few <code>MultipleSeqAlignment</code> objects the hard way (by hand, rather than by loading them from a file).
Note we create some <code>SeqRecord</code> objects to construct the alignment from.</p><pre class="verbatim">from Bio.Alphabet import generic_dna
from Bio.Seq import Seq
from Bio.SeqRecord import SeqRecord
from Bio.Align import MultipleSeqAlignment

align1 = MultipleSeqAlignment([
             SeqRecord(Seq("ACTGCTAGCTAG", generic_dna), id="Alpha"),
             SeqRecord(Seq("ACT-CTAGCTAG", generic_dna), id="Beta"),
             SeqRecord(Seq("ACTGCTAGDTAG", generic_dna), id="Gamma"),
         ])

align2 = MultipleSeqAlignment([
             SeqRecord(Seq("GTCAGC-AG", generic_dna), id="Delta"),
             SeqRecord(Seq("GACAGCTAG", generic_dna), id="Epsilon"),
             SeqRecord(Seq("GTCAGCTAG", generic_dna), id="Zeta"),
         ])

align3 = MultipleSeqAlignment([
             SeqRecord(Seq("ACTAGTACAGCTG", generic_dna), id="Eta"),
             SeqRecord(Seq("ACTAGTACAGCT-", generic_dna), id="Theta"),
             SeqRecord(Seq("-CTACTACAGGTG", generic_dna), id="Iota"),
         ])

my_alignments = [align1, align2, align3]
</pre><p>Now we have a list of <code>Alignment</code> objects, we&#X2019;ll write them to a PHYLIP format file:</p><pre class="verbatim">from Bio import AlignIO
AlignIO.write(my_alignments, "my_example.phy", "phylip")
</pre><p>And if you open this file in your favourite text editor it should look like this:</p><pre class="verbatim"> 3 12
Alpha      ACTGCTAGCT AG
Beta       ACT-CTAGCT AG
Gamma      ACTGCTAGDT AG
 3 9
Delta      GTCAGC-AG
Epislon    GACAGCTAG
Zeta       GTCAGCTAG
 3 13
Eta        ACTAGTACAG CTG
Theta      ACTAGTACAG CT-
Iota       -CTACTACAG GTG
</pre><p>Its more common to want to load an existing alignment, and save that, perhaps after some simple manipulation like removing certain rows or columns.</p><p>Suppose you wanted to know how many alignments the <code>Bio.AlignIO.write()</code> function wrote to the handle? If your alignments were in a list like the example above, you could just use <code>len(my_alignments)</code>, however you can&#X2019;t do that when your records come from a generator/iterator. Therefore the <code>Bio.AlignIO.write()</code> function returns the number of alignments written to the file. </p><p><em>Note</em> - If you tell the <code>Bio.AlignIO.write()</code> function to write to a file that already exists, the old file will be overwritten without any warning.</p>
<!--TOC subsection id="sec83" Converting between sequence alignment file formats-->
<h3 id="sec83" class="subsection">6.2.1&#XA0;&#XA0;Converting between sequence alignment file formats</h3><!--SEC END --><p>
<a id="sec:converting-alignments"></a></p><p>Converting between sequence alignment file formats with <code>Bio.AlignIO</code> works
in the same way as converting between sequence file formats with <code>Bio.SeqIO</code>
(Section&#XA0;<a href="#sec%3ASeqIO-conversion">5.5.2</a>). We load generally the alignment(s) using
<code>Bio.AlignIO.parse()</code> and then save them using the <code>Bio.AlignIO.write()</code>
&#X2013; or just use the <code>Bio.AlignIO.convert()</code> helper function.</p><p>For this example, we&#X2019;ll load the PFAM/Stockholm format file used earlier and save it as a Clustal W format file:</p><pre class="verbatim">from Bio import AlignIO
count = AlignIO.convert("PF05371_seed.sth", "stockholm", "PF05371_seed.aln", "clustal")
print("Converted %i alignments" % count)
</pre><p>Or, using <code>Bio.AlignIO.parse()</code> and <code>Bio.AlignIO.write()</code>:</p><pre class="verbatim">from Bio import AlignIO
alignments = AlignIO.parse("PF05371_seed.sth", "stockholm")
count = AlignIO.write(alignments, "PF05371_seed.aln", "clustal")
print("Converted %i alignments" % count)
</pre><p>The <code>Bio.AlignIO.write()</code> function expects to be given multiple alignment objects. In the example above we gave it the alignment iterator returned by <code>Bio.AlignIO.parse()</code>.</p><p>In this case, we know there is only one alignment in the file so we could have used <code>Bio.AlignIO.read()</code> instead, but notice we have to pass this alignment to <code>Bio.AlignIO.write()</code> as a single element list:</p><pre class="verbatim">from Bio import AlignIO
alignment = AlignIO.read("PF05371_seed.sth", "stockholm")
AlignIO.write([alignment], "PF05371_seed.aln", "clustal")
</pre><p>Either way, you should end up with the same new Clustal W format file &#X201C;PF05371_seed.aln&#X201D; with the following content:</p><pre class="verbatim">CLUSTAL X (1.81) multiple sequence alignment


COATB_BPIKE/30-81                   AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIRLFKKFSS
Q9T0Q8_BPIKE/1-52                   AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIKLFKKFVS
COATB_BPI22/32-83                   DGTSTATSYATEAMNSLKTQATDLIDQTWPVVTSVAVAGLAIRLFKKFSS
COATB_BPM13/24-72                   AEGDDP---AKAAFNSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTS
COATB_BPZJ2/1-49                    AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFAS
Q9T0Q9_BPFD/1-49                    AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTS
COATB_BPIF1/22-73                   FAADDATSQAKAAFDSLTAQATEMSGYAWALVVLVVGATVGIKLFKKFVS

COATB_BPIKE/30-81                   KA
Q9T0Q8_BPIKE/1-52                   RA
COATB_BPI22/32-83                   KA
COATB_BPM13/24-72                   KA
COATB_BPZJ2/1-49                    KA
Q9T0Q9_BPFD/1-49                    KA
COATB_BPIF1/22-73                   RA
</pre><p>Alternatively, you could make a PHYLIP format file which we&#X2019;ll name &#X201C;PF05371_seed.phy&#X201D;:</p><pre class="verbatim">from Bio import AlignIO
AlignIO.convert("PF05371_seed.sth", "stockholm", "PF05371_seed.phy", "phylip")
</pre><p>This time the output looks like this:</p><pre class="verbatim"> 7 52
COATB_BPIK AEPNAATNYA TEAMDSLKTQ AIDLISQTWP VVTTVVVAGL VIRLFKKFSS
Q9T0Q8_BPI AEPNAATNYA TEAMDSLKTQ AIDLISQTWP VVTTVVVAGL VIKLFKKFVS
COATB_BPI2 DGTSTATSYA TEAMNSLKTQ ATDLIDQTWP VVTSVAVAGL AIRLFKKFSS
COATB_BPM1 AEGDDP---A KAAFNSLQAS ATEYIGYAWA MVVVIVGATI GIKLFKKFTS
COATB_BPZJ AEGDDP---A KAAFDSLQAS ATEYIGYAWA MVVVIVGATI GIKLFKKFAS
Q9T0Q9_BPF AEGDDP---A KAAFDSLQAS ATEYIGYAWA MVVVIVGATI GIKLFKKFTS
COATB_BPIF FAADDATSQA KAAFDSLTAQ ATEMSGYAWA LVVLVVGATV GIKLFKKFVS

           KA
           RA
           KA
           KA
           KA
           KA
           RA
</pre><p>One of the big handicaps of the original PHYLIP alignment file format is
that the sequence identifiers are strictly truncated at ten characters.
In this example, as you can see the resulting names are still unique -
but they are not very readable. As a result, a more relaxed variant of
the original PHYLIP format is now quite widely used:</p><pre class="verbatim">from Bio import AlignIO
AlignIO.convert("PF05371_seed.sth", "stockholm", "PF05371_seed.phy", "phylip-relaxed")
</pre><p>This time the output looks like this, using a longer indentation to
allow all the identifers to be given in full::</p><pre class="verbatim"> 7 52
COATB_BPIKE/30-81  AEPNAATNYA TEAMDSLKTQ AIDLISQTWP VVTTVVVAGL VIRLFKKFSS
Q9T0Q8_BPIKE/1-52  AEPNAATNYA TEAMDSLKTQ AIDLISQTWP VVTTVVVAGL VIKLFKKFVS
COATB_BPI22/32-83  DGTSTATSYA TEAMNSLKTQ ATDLIDQTWP VVTSVAVAGL AIRLFKKFSS
COATB_BPM13/24-72  AEGDDP---A KAAFNSLQAS ATEYIGYAWA MVVVIVGATI GIKLFKKFTS
COATB_BPZJ2/1-49   AEGDDP---A KAAFDSLQAS ATEYIGYAWA MVVVIVGATI GIKLFKKFAS
Q9T0Q9_BPFD/1-49   AEGDDP---A KAAFDSLQAS ATEYIGYAWA MVVVIVGATI GIKLFKKFTS
COATB_BPIF1/22-73  FAADDATSQA KAAFDSLTAQ ATEMSGYAWA LVVLVVGATV GIKLFKKFVS

                   KA
                   RA
                   KA
                   KA
                   KA
                   KA
                   RA
</pre><p>If you have to work with the original strict PHYLIP format, then you may need to
compress the identifers somehow &#X2013; or assign your own names or numbering system.
This following bit of code manipulates the record identifiers before saving the output:</p><pre class="verbatim">from Bio import AlignIO
alignment = AlignIO.read("PF05371_seed.sth", "stockholm")
name_mapping = {}
for i, record in enumerate(alignment):
    name_mapping[i] = record.id
    record.id = "seq%i" % i
print(name_mapping)

AlignIO.write([alignment], "PF05371_seed.phy", "phylip")
</pre><p>This code used a Python dictionary to record a simple mapping from the new sequence system to the original identifier:
</p><pre class="verbatim">{0: 'COATB_BPIKE/30-81', 1: 'Q9T0Q8_BPIKE/1-52', 2: 'COATB_BPI22/32-83', ...}
</pre><p>Here is the new (strict) PHYLIP format output:
</p><pre class="verbatim"> 7 52
seq0       AEPNAATNYA TEAMDSLKTQ AIDLISQTWP VVTTVVVAGL VIRLFKKFSS
seq1       AEPNAATNYA TEAMDSLKTQ AIDLISQTWP VVTTVVVAGL VIKLFKKFVS
seq2       DGTSTATSYA TEAMNSLKTQ ATDLIDQTWP VVTSVAVAGL AIRLFKKFSS
seq3       AEGDDP---A KAAFNSLQAS ATEYIGYAWA MVVVIVGATI GIKLFKKFTS
seq4       AEGDDP---A KAAFDSLQAS ATEYIGYAWA MVVVIVGATI GIKLFKKFAS
seq5       AEGDDP---A KAAFDSLQAS ATEYIGYAWA MVVVIVGATI GIKLFKKFTS
seq6       FAADDATSQA KAAFDSLTAQ ATEMSGYAWA LVVLVVGATV GIKLFKKFVS

           KA
           RA
           KA
           KA
           KA
           KA
           RA
</pre><p>In general, because of the identifier limitation, working with
<span style="font-style:italic">strict</span> PHYLIP file formats shouldn&#X2019;t be your first choice. 
Using the PFAM/Stockholm format on the other hand allows you to record a lot of additional annotation too.</p>
<!--TOC subsection id="sec84" Getting your alignment objects as formatted strings-->
<h3 id="sec84" class="subsection">6.2.2&#XA0;&#XA0;Getting your alignment objects as formatted strings</h3><!--SEC END --><p>
<a id="sec:alignment-format-method"></a>
The <code>Bio.AlignIO</code> interface is based on handles, which means if you want to get your alignment(s) into a string in a particular file format you need to do a little bit more work (see below). 
However, you will probably prefer to take advantage of the alignment object&#X2019;s <code>format()</code> method.
This takes a single mandatory argument, a lower case string which is supported by <code>Bio.AlignIO</code> as an output format. For example:</p><pre class="verbatim">from Bio import AlignIO
alignment = AlignIO.read("PF05371_seed.sth", "stockholm")
print(alignment.format("clustal"))
</pre><p>As described in Section&#XA0;<a href="#sec%3ASeqRecord-format">4.5</a>, the <code>SeqRecord</code> object has a similar method using output formats supported by <code>Bio.SeqIO</code>.</p><p>Internally the <code>format()</code> method is using the <code>StringIO</code> string based handle and calling
<code>Bio.AlignIO.write()</code>. You can do this in your own code if for example you are using an
older version of Biopython:</p><pre class="verbatim">from Bio import AlignIO
from StringIO import StringIO

alignments = AlignIO.parse("PF05371_seed.sth", "stockholm")

out_handle = StringIO()
AlignIO.write(alignments, out_handle, "clustal")
clustal_data = out_handle.getvalue()

print(clustal_data)
</pre>
<!--TOC section id="sec85" Manipulating Alignments-->
<h2 id="sec85" class="section">6.3&#XA0;&#XA0;Manipulating Alignments</h2><!--SEC END --><p>
<a id="sec:manipulating-alignments"></a></p><p>Now that we&#X2019;ve covered loading and saving alignments, we&#X2019;ll look at what else you can do
with them.</p>
<!--TOC subsection id="sec86" Slicing alignments-->
<h3 id="sec86" class="subsection">6.3.1&#XA0;&#XA0;Slicing alignments</h3><!--SEC END --><p>
First of all, in some senses the alignment objects act like a Python <code>list</code> of
<code>SeqRecord</code> objects (the rows). With this model in mind hopefully the actions
of <code>len()</code> (the number of rows) and iteration (each row as a <code>SeqRecord</code>)
make sense:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import AlignIO
&gt;&gt;&gt; alignment = AlignIO.read("PF05371_seed.sth", "stockholm")
&gt;&gt;&gt; print("Number of rows: %i" % len(alignment))
Number of rows: 7
&gt;&gt;&gt; for record in alignment:
...     print("%s - %s" % (record.seq, record.id))
AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIRLFKKFSSKA - COATB_BPIKE/30-81
AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIKLFKKFVSRA - Q9T0Q8_BPIKE/1-52
DGTSTATSYATEAMNSLKTQATDLIDQTWPVVTSVAVAGLAIRLFKKFSSKA - COATB_BPI22/32-83
AEGDDP---AKAAFNSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA - COATB_BPM13/24-72
AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFASKA - COATB_BPZJ2/1-49
AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA - Q9T0Q9_BPFD/1-49
FAADDATSQAKAAFDSLTAQATEMSGYAWALVVLVVGATVGIKLFKKFVSRA - COATB_BPIF1/22-73
</pre><p>You can also use the list-like <code>append</code> and <code>extend</code> methods to add
more rows to the alignment (as <code>SeqRecord</code> objects). Keeping the list
metaphor in mind, simple slicing of the alignment should also make sense -
it selects some of the rows giving back another alignment object:</p><pre class="verbatim">&gt;&gt;&gt; print(alignment)
SingleLetterAlphabet() alignment with 7 rows and 52 columns
AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIRL...SKA COATB_BPIKE/30-81
AEPNAATNYATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIKL...SRA Q9T0Q8_BPIKE/1-52
DGTSTATSYATEAMNSLKTQATDLIDQTWPVVTSVAVAGLAIRL...SKA COATB_BPI22/32-83
AEGDDP---AKAAFNSLQASATEYIGYAWAMVVVIVGATIGIKL...SKA COATB_BPM13/24-72
AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKL...SKA COATB_BPZJ2/1-49
AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKL...SKA Q9T0Q9_BPFD/1-49
FAADDATSQAKAAFDSLTAQATEMSGYAWALVVLVVGATVGIKL...SRA COATB_BPIF1/22-73
&gt;&gt;&gt; print(alignment[3:7])
SingleLetterAlphabet() alignment with 4 rows and 52 columns
AEGDDP---AKAAFNSLQASATEYIGYAWAMVVVIVGATIGIKL...SKA COATB_BPM13/24-72
AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKL...SKA COATB_BPZJ2/1-49
AEGDDP---AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKL...SKA Q9T0Q9_BPFD/1-49
FAADDATSQAKAAFDSLTAQATEMSGYAWALVVLVVGATVGIKL...SRA COATB_BPIF1/22-73
</pre><p>What if you wanted to select by column? Those of you who have used the NumPy
matrix or array objects won&#X2019;t be surprised at this - you use a double index.</p><pre class="verbatim">&gt;&gt;&gt; print(alignment[2, 6])
T
</pre><p>Using two integer indices pulls out a single letter, short hand for this:</p><pre class="verbatim">&gt;&gt;&gt; print(alignment[2].seq[6])
T
</pre><p>You can pull out a single column as a string like this:</p><pre class="verbatim">&gt;&gt;&gt; print(alignment[:, 6])
TTT---T
</pre><p>You can also select a range of columns. For example, to pick out those same
three rows we extracted earlier, but take just their first six columns:</p><pre class="verbatim">&gt;&gt;&gt; print(alignment[3:6, :6])
SingleLetterAlphabet() alignment with 3 rows and 6 columns
AEGDDP COATB_BPM13/24-72
AEGDDP COATB_BPZJ2/1-49
AEGDDP Q9T0Q9_BPFD/1-49
</pre><p>Leaving the first index as <code>:</code> means take all the rows:</p><pre class="verbatim">&gt;&gt;&gt; print(alignment[:, :6])
SingleLetterAlphabet() alignment with 7 rows and 6 columns
AEPNAA COATB_BPIKE/30-81
AEPNAA Q9T0Q8_BPIKE/1-52
DGTSTA COATB_BPI22/32-83
AEGDDP COATB_BPM13/24-72
AEGDDP COATB_BPZJ2/1-49
AEGDDP Q9T0Q9_BPFD/1-49
FAADDA COATB_BPIF1/22-73
</pre><p>This brings us to a neat way to remove a section. Notice columns
7, 8 and 9 which are gaps in three of the seven sequences:</p><pre class="verbatim">&gt;&gt;&gt; print(alignment[:, 6:9])
SingleLetterAlphabet() alignment with 7 rows and 3 columns
TNY COATB_BPIKE/30-81
TNY Q9T0Q8_BPIKE/1-52
TSY COATB_BPI22/32-83
--- COATB_BPM13/24-72
--- COATB_BPZJ2/1-49
--- Q9T0Q9_BPFD/1-49
TSQ COATB_BPIF1/22-73
</pre><p>Again, you can slice to get everything after the ninth column:</p><pre class="verbatim">&gt;&gt;&gt; print(alignment[:, 9:])
SingleLetterAlphabet() alignment with 7 rows and 43 columns
ATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIRLFKKFSSKA COATB_BPIKE/30-81
ATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIKLFKKFVSRA Q9T0Q8_BPIKE/1-52
ATEAMNSLKTQATDLIDQTWPVVTSVAVAGLAIRLFKKFSSKA COATB_BPI22/32-83
AKAAFNSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA COATB_BPM13/24-72
AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFASKA COATB_BPZJ2/1-49
AKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA Q9T0Q9_BPFD/1-49
AKAAFDSLTAQATEMSGYAWALVVLVVGATVGIKLFKKFVSRA COATB_BPIF1/22-73
</pre><p>Now, the interesting thing is that addition of alignment objects works
by column. This lets you do this as a way to remove a block of columns:</p><pre class="verbatim">&gt;&gt;&gt; edited = alignment[:, :6] + alignment[:, 9:]
&gt;&gt;&gt; print(edited)
SingleLetterAlphabet() alignment with 7 rows and 49 columns
AEPNAAATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIRLFKKFSSKA COATB_BPIKE/30-81
AEPNAAATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIKLFKKFVSRA Q9T0Q8_BPIKE/1-52
DGTSTAATEAMNSLKTQATDLIDQTWPVVTSVAVAGLAIRLFKKFSSKA COATB_BPI22/32-83
AEGDDPAKAAFNSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA COATB_BPM13/24-72
AEGDDPAKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFASKA COATB_BPZJ2/1-49
AEGDDPAKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA Q9T0Q9_BPFD/1-49
FAADDAAKAAFDSLTAQATEMSGYAWALVVLVVGATVGIKLFKKFVSRA COATB_BPIF1/22-73
</pre><p>Another common use of alignment addition would be to combine alignments for
several different genes into a meta-alignment. Watch out though - the identifiers
need to match up (see Section&#XA0;<a href="#sec%3ASeqRecord-addition">4.7</a> for how adding
<code>SeqRecord</code> objects works). You may find it helpful to first sort the
alignment rows alphabetically by id:</p><pre class="verbatim">&gt;&gt;&gt; edited.sort()
&gt;&gt;&gt; print(edited)
SingleLetterAlphabet() alignment with 7 rows and 49 columns
DGTSTAATEAMNSLKTQATDLIDQTWPVVTSVAVAGLAIRLFKKFSSKA COATB_BPI22/32-83
FAADDAAKAAFDSLTAQATEMSGYAWALVVLVVGATVGIKLFKKFVSRA COATB_BPIF1/22-73
AEPNAAATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIRLFKKFSSKA COATB_BPIKE/30-81
AEGDDPAKAAFNSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA COATB_BPM13/24-72
AEGDDPAKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFASKA COATB_BPZJ2/1-49
AEPNAAATEAMDSLKTQAIDLISQTWPVVTTVVVAGLVIKLFKKFVSRA Q9T0Q8_BPIKE/1-52
AEGDDPAKAAFDSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFTSKA Q9T0Q9_BPFD/1-49
</pre><p>Note that you can only add two alignments together if they
have the same number of rows.</p>
<!--TOC subsection id="sec87" Alignments as arrays-->
<h3 id="sec87" class="subsection">6.3.2&#XA0;&#XA0;Alignments as arrays</h3><!--SEC END --><p>
Depending on what you are doing, it can be more useful to turn the alignment
object into an array of letters &#X2013; and you can do this with NumPy:</p><pre class="verbatim">&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; from Bio import AlignIO
&gt;&gt;&gt; alignment = AlignIO.read("PF05371_seed.sth", "stockholm")
&gt;&gt;&gt; align_array = np.array([list(rec) for rec in alignment], np.character)
&gt;&gt;&gt; print("Array shape %i by %i" % align_array.shape)
Array shape 7 by 52
</pre><p>If you will be working heavily with the columns, you can tell NumPy to store
the array by column (as in Fortran) rather then its default of by row (as in C):</p><pre class="verbatim">&gt;&gt;&gt; align_array = np.array([list(rec) for rec in alignment], np.character, order="F")
</pre><p>Note that this leaves the original Biopython alignment object and the NumPy array
in memory as separate objects - editing one will not update the other!</p>
<!--TOC section id="sec88" Alignment Tools-->
<h2 id="sec88" class="section">6.4&#XA0;&#XA0;Alignment Tools</h2><!--SEC END --><p>
<a id="sec:alignment-tools"></a></p><p>There are <em>lots</em> of algorithms out there for aligning sequences, both pairwise alignments
and multiple sequence alignments. These calculations are relatively slow, and you generally
wouldn&#X2019;t want to write such an algorithm in Python. Instead, you can use Biopython to invoke
a command line tool on your behalf. Normally you would:
</p><ol class="enumerate" type=1><li class="li-enumerate">
Prepare an input file of your unaligned sequences, typically this will be a FASTA file
which you might create using <code>Bio.SeqIO</code> (see Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>).
</li><li class="li-enumerate">Call the command line tool to process this input file, typically via one of Biopython&#X2019;s
command line wrappers (which we&#X2019;ll discuss here).
</li><li class="li-enumerate">Read the output from the tool, i.e. your aligned sequences, typically using
<code>Bio.AlignIO</code> (see earlier in this chapter). 
</li></ol><p>All the command line wrappers we&#X2019;re going to talk about in this chapter follow the same style.
You create a command line object specifying the options (e.g. the input filename and the
output filename), then invoke this command line via a Python operating system call (e.g.
using the <span style="font-family:monospace">subprocess</span> module).</p><p>Most of these wrappers are defined in the <code>Bio.Align.Applications</code> module:</p><pre class="verbatim">&gt;&gt;&gt; import Bio.Align.Applications
&gt;&gt;&gt; dir(Bio.Align.Applications)
...
['ClustalwCommandline', 'DialignCommandline', 'MafftCommandline', 'MuscleCommandline',
'PrankCommandline', 'ProbconsCommandline', 'TCoffeeCommandline' ...]
</pre><p>(Ignore the entries starting with an underscore &#X2013; these have
special meaning in Python.)
The module <code>Bio.Emboss.Applications</code> has wrappers for some of the
<a href="http://emboss.sourceforge.net/">EMBOSS suite</a>, including
<span style="font-family:monospace">needle</span> and <span style="font-family:monospace">water</span>, which are described below in
Section&#XA0;<a href="#seq%3Aemboss-needle-water">6.4.5</a>, and wrappers for the EMBOSS
packaged versions of the PHYLIP tools (which EMBOSS refer to as one
of their EMBASSY packages - third party tools with an EMBOSS style
interface).
We won&#X2019;t explore all these alignment tools here in the section, just a
sample, but the same principles apply.</p>
<!--TOC subsection id="sec89" ClustalW-->
<h3 id="sec89" class="subsection">6.4.1&#XA0;&#XA0;ClustalW</h3><!--SEC END --><p>
<a id="sec:align_clustal"></a>
ClustalW is a popular command line tool for multiple sequence alignment
(there is also a graphical interface called ClustalX). Biopython&#X2019;s
<code>Bio.Align.Applications</code> module has a wrapper for this alignment tool
(and several others).</p><p>Before trying to use ClustalW from within Python, you should first try running
the ClustalW tool yourself by hand at the command line, to familiarise
yourself the other options. You&#X2019;ll find the Biopython wrapper is very
faithful to the actual command line API:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Align.Applications import ClustalwCommandline
&gt;&gt;&gt; help(ClustalwCommandline)
...
</pre><p>For the most basic usage, all you need is to have a FASTA input file, such as
<a href="http://biopython.org/DIST/docs/tutorial/examples/opuntia.fasta">opuntia.fasta</a>
(available online or in the Doc/examples subdirectory of the Biopython source
code). This is a small FASTA file containing seven prickly-pear DNA sequences
(from the cactus family <span style="font-style:italic">Opuntia</span>).</p><p>By default ClustalW will generate an alignment and guide tree file with names
based on the input FASTA file, in this case <span style="font-family:monospace">opuntia.aln</span> and
<span style="font-family:monospace">opuntia.dnd</span>, but you can override this or make it explicit:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Align.Applications import ClustalwCommandline
&gt;&gt;&gt; cline = ClustalwCommandline("clustalw2", infile="opuntia.fasta")
&gt;&gt;&gt; print(cline)
clustalw2 -infile=opuntia.fasta
</pre><p>Notice here we have given the executable name as <span style="font-family:monospace">clustalw2</span>,
indicating we have version two installed, which has a different filename to
version one (<span style="font-family:monospace">clustalw</span>, the default). Fortunately both versions
support the same set of arguments at the command line (and indeed, should be
functionally identical).</p><p>You may find that even though you have ClustalW installed, the above command
doesn&#X2019;t work &#X2013; you may get a message about &#X201C;command not found&#X201D; (especially
on Windows). This indicated that the ClustalW executable is not on your PATH
(an environment variable, a list of directories to be searched). You can
either update your PATH setting to include the location of your copy of
ClustalW tools (how you do this will depend on your OS), or simply type in
the full path of the tool. For example:</p><pre class="verbatim">&gt;&gt;&gt; import os
&gt;&gt;&gt; from Bio.Align.Applications import ClustalwCommandline
&gt;&gt;&gt; clustalw_exe = r"C:\Program Files\new clustal\clustalw2.exe"
&gt;&gt;&gt; clustalw_cline = ClustalwCommandline(clustalw_exe, infile="opuntia.fasta")
</pre><pre class="verbatim">&gt;&gt;&gt; assert os.path.isfile(clustalw_exe), "Clustal W executable missing"
&gt;&gt;&gt; stdout, stderr = clustalw_cline()
</pre><p>Remember, in Python strings <code>\n</code> and <code>\t</code> are by default
interpreted as a new line and a tab &#X2013; which is why we&#X2019;re put a letter
&#X201C;r&#X201D; at the start for a raw string that isn&#X2019;t translated in this way.
This is generally good practice when specifying a Windows style file name.</p><p>Internally this uses the
<code>subprocess</code> module which is now the recommended way to run another
program in Python. This replaces older options like the <code>os.system()</code>
and the <code>os.popen*</code> functions.</p><p>Now, at this point it helps to know about how command line tools &#X201C;work&#X201D;.
When you run a tool at the command line, it will often print text output
directly to screen. This text can be captured or redirected, via
two &#X201C;pipes&#X201D;, called standard output (the normal results) and standard
error (for error messages and debug messages). There is also standard
input, which is any text fed into the tool. These names get shortened
to stdin, stdout and stderr. When the tool finishes, it has a return
code (an integer), which by convention is zero for success.</p><p>When you run the command line tool like this via the Biopython wrapper,
it will wait for it to finish, and check the return code. If this is
non zero (indicating an error), an exception is raised. The wrapper
then returns two strings, stdout and stderr.</p><p>In the case of ClustalW, when run at the command line all the important
output is written directly to the output files. Everything normally printed to
screen while you wait (via stdout or stderr) is boring and can be
ignored (assuming it worked).</p><p>What we care about are the two output files, the alignment and the guide
tree. We didn&#X2019;t tell ClustalW what filenames to use, but it defaults to
picking names based on the input file. In this case the output should be
in the file <code>opuntia.aln</code>.
You should be able to work out how to read in the alignment using
<code>Bio.AlignIO</code> by now:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import AlignIO
&gt;&gt;&gt; align = AlignIO.read("opuntia.aln", "clustal")
&gt;&gt;&gt; print(align)
SingleLetterAlphabet() alignment with 7 rows and 906 columns
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273285|gb|AF191659.1|AF191
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273284|gb|AF191658.1|AF191
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273287|gb|AF191661.1|AF191
TATACATAAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273286|gb|AF191660.1|AF191
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273290|gb|AF191664.1|AF191
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273289|gb|AF191663.1|AF191
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273291|gb|AF191665.1|AF191
</pre><p>In case you are interested (and this is an aside from the main thrust of this
chapter), the <span style="font-family:monospace">opuntia.dnd</span> file ClustalW creates is just a standard
Newick tree file, and <code>Bio.Phylo</code> can parse these:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Phylo
&gt;&gt;&gt; tree = Phylo.read("opuntia.dnd", "newick")
&gt;&gt;&gt; Phylo.draw_ascii(tree)
                             _______________ gi|6273291|gb|AF191665.1|AF191665
  __________________________|
 |                          |   ______ gi|6273290|gb|AF191664.1|AF191664
 |                          |__|
 |                             |_____ gi|6273289|gb|AF191663.1|AF191663
 |
_|_________________ gi|6273287|gb|AF191661.1|AF191661
 |
 |__________ gi|6273286|gb|AF191660.1|AF191660
 |
 |    __ gi|6273285|gb|AF191659.1|AF191659
 |___|
     | gi|6273284|gb|AF191658.1|AF191658

</pre><p>Chapter <a href="#sec%3APhylo">13</a> covers Biopython&#X2019;s support for phylogenetic trees in more
depth.</p>
<!--TOC subsection id="sec90" MUSCLE-->
<h3 id="sec90" class="subsection">6.4.2&#XA0;&#XA0;MUSCLE</h3><!--SEC END --><p>
MUSCLE is a more recent multiple sequence alignment tool than ClustalW, and
Biopython also has a wrapper for it under the <code>Bio.Align.Applications</code>
module. As before, we recommend you try using MUSCLE from the command line before
trying it from within Python, as the Biopython wrapper is very faithful to the
actual command line API:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Align.Applications import MuscleCommandline
&gt;&gt;&gt; help(MuscleCommandline)
...
</pre><p>For the most basic usage, all you need is to have a FASTA input file, such as
<a href="http://biopython.org/DIST/docs/tutorial/examples/opuntia.fasta">opuntia.fasta</a>
(available online or in the Doc/examples subdirectory of the Biopython source
code). You can then tell MUSCLE to read in this FASTA file, and write the
alignment to an output file:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Align.Applications import MuscleCommandline
&gt;&gt;&gt; cline = MuscleCommandline(input="opuntia.fasta", out="opuntia.txt")
&gt;&gt;&gt; print(cline)
muscle -in opuntia.fasta -out opuntia.txt
</pre><p>Note that MUSCLE uses &#X201C;-in&#X201D; and &#X201C;-out&#X201D; but in Biopython we have to use
&#X201C;input&#X201D; and &#X201C;out&#X201D; as the keyword arguments or property names. This is
because &#X201C;in&#X201D; is a reserved word in Python.</p><p>By default MUSCLE will output the alignment as a FASTA file (using gapped
sequences). The <code>Bio.AlignIO</code> module should be able to read this
alignment using <span style="font-family:monospace">format="fasta"</span>.
You can also ask for ClustalW-like output:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Align.Applications import MuscleCommandline
&gt;&gt;&gt; cline = MuscleCommandline(input="opuntia.fasta", out="opuntia.aln", clw=True)
&gt;&gt;&gt; print(cline)
muscle -in opuntia.fasta -out opuntia.aln -clw
</pre><p>Or, strict ClustalW output where the original ClustalW header line is
used for maximum compatibility:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Align.Applications import MuscleCommandline
&gt;&gt;&gt; cline = MuscleCommandline(input="opuntia.fasta", out="opuntia.aln", clwstrict=True)
&gt;&gt;&gt; print(cline)
muscle -in opuntia.fasta -out opuntia.aln -clwstrict
</pre><p>The <code>Bio.AlignIO</code> module should be able to read these alignments
using <span style="font-family:monospace">format="clustal"</span>.</p><p>MUSCLE can also output in GCG MSF format (using the <span style="font-family:monospace">msf</span> argument), but
Biopython can&#X2019;t currently parse that, or using HTML which would give a human
readable web page (not suitable for parsing).</p><p>You can also set the other optional parameters, for example the maximum number
of iterations. See the built in help for details.</p><p>You would then run MUSCLE command line string as described above for
ClustalW, and parse the output using <code>Bio.AlignIO</code> to get an
alignment object.</p>
<!--TOC subsection id="sec91" MUSCLE using stdout-->
<h3 id="sec91" class="subsection">6.4.3&#XA0;&#XA0;MUSCLE using stdout</h3><!--SEC END --><p>Using a MUSCLE command line as in the examples above will write the alignment
to a file. This means there will be no important information written to the
standard out (stdout) or standard error (stderr) handles. However, by default
MUSCLE will write the alignment to standard output (stdout). We can take
advantage of this to avoid having a temporary output file! For example:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Align.Applications import MuscleCommandline
&gt;&gt;&gt; muscle_cline = MuscleCommandline(input="opuntia.fasta")
&gt;&gt;&gt; print(muscle_cline)
muscle -in opuntia.fasta
</pre><p>If we run this via the wrapper, we get back the output as a string. In order
to parse this we can use <code>StringIO</code> to turn it into a handle.
Remember that MUSCLE defaults to using FASTA as the output format:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Align.Applications import MuscleCommandline
&gt;&gt;&gt; muscle_cline = MuscleCommandline(input="opuntia.fasta")
&gt;&gt;&gt; stdout, stderr = muscle_cline()
&gt;&gt;&gt; from StringIO import StringIO
&gt;&gt;&gt; from Bio import AlignIO
&gt;&gt;&gt; align = AlignIO.read(StringIO(stdout), "fasta")
&gt;&gt;&gt; print(align)
SingleLetterAlphabet() alignment with 7 rows and 906 columns
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273289|gb|AF191663.1|AF191663
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273291|gb|AF191665.1|AF191665
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273290|gb|AF191664.1|AF191664
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273287|gb|AF191661.1|AF191661
TATACATAAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273286|gb|AF191660.1|AF191660
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273285|gb|AF191659.1|AF191659
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273284|gb|AF191658.1|AF191658
</pre><p>The above approach is fairly simple, but if you are dealing with very large output
text the fact that all of stdout and stderr is loaded into memory as a string can
be a potential drawback. Using the <code>subprocess</code> module we can work directly
with handles instead:</p><pre class="verbatim">&gt;&gt;&gt; import subprocess
&gt;&gt;&gt; from Bio.Align.Applications import MuscleCommandline
&gt;&gt;&gt; muscle_cline = MuscleCommandline(input="opuntia.fasta")
&gt;&gt;&gt; child = subprocess.Popen(str(muscle_cline),
...                          stdout=subprocess.PIPE,
...                          stderr=subprocess.PIPE,
...                          shell=(sys.platform!="win32"))
&gt;&gt;&gt; from Bio import AlignIO
&gt;&gt;&gt; align = AlignIO.read(child.stdout, "fasta")
&gt;&gt;&gt; print(align)
SingleLetterAlphabet() alignment with 7 rows and 906 columns
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273289|gb|AF191663.1|AF191663
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273291|gb|AF191665.1|AF191665
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273290|gb|AF191664.1|AF191664
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273287|gb|AF191661.1|AF191661
TATACATAAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273286|gb|AF191660.1|AF191660
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273285|gb|AF191659.1|AF191659
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273284|gb|AF191658.1|AF191658
</pre>
<!--TOC subsection id="sec92" MUSCLE using stdin and stdout-->
<h3 id="sec92" class="subsection">6.4.4&#XA0;&#XA0;MUSCLE using stdin and stdout</h3><!--SEC END --><p>We don&#X2019;t actually <em>need</em> to have our FASTA input sequences prepared in a file,
because by default MUSCLE will read in the input sequence from standard input!
Note this is a bit more advanced and fiddly, so don&#X2019;t bother with this technique
unless you need to.</p><p>First, we&#X2019;ll need some unaligned sequences in memory as <code>SeqRecord</code> objects.
For this demonstration I&#X2019;m going to use a filtered version of the original FASTA
file (using a generator expression), taking just six of the seven sequences:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; records = (r for r in SeqIO.parse("opuntia.fasta", "fasta") if len(r) &lt; 900)
</pre><p>Then we create the MUSCLE command line, leaving the input and output to their
defaults (stdin and stdout). I&#X2019;m also going to ask for strict ClustalW format
as for the output.</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Align.Applications import MuscleCommandline
&gt;&gt;&gt; muscle_cline = MuscleCommandline(clwstrict=True)
&gt;&gt;&gt; print(muscle_cline)
muscle -clwstrict
</pre><p>Now for the fiddly bits using the <code>subprocess</code> module, stdin and stdout:</p><pre class="verbatim">&gt;&gt;&gt; import subprocess
&gt;&gt;&gt; import sys
&gt;&gt;&gt; child = subprocess.Popen(str(cline),
...                          stdin=subprocess.PIPE,
...                          stdout=subprocess.PIPE,
...                          stderr=subprocess.PIPE,
...                          universal_newlines=True,
...                          shell=(sys.platform!="win32"))                     
</pre><p>That should start MUSCLE, but it will be sitting waiting for its FASTA input
sequences, which we must supply via its stdin handle:</p><pre class="verbatim">&gt;&gt;&gt; SeqIO.write(records, child.stdin, "fasta")
6
&gt;&gt;&gt; child.stdin.close()
</pre><p>After writing the six sequences to the handle, MUSCLE will still be waiting
to see if that is all the FASTA sequences or not &#X2013; so we must signal that
this is all the input data by closing the handle. At that point MUSCLE should
start to run, and we can ask for the output:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import AlignIO
&gt;&gt;&gt; align = AlignIO.read(child.stdout, "clustal")
&gt;&gt;&gt; print(align)
SingleLetterAlphabet() alignment with 6 rows and 900 columns
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273290|gb|AF191664.1|AF19166
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273289|gb|AF191663.1|AF19166
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273287|gb|AF191661.1|AF19166
TATACATAAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273286|gb|AF191660.1|AF19166
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273285|gb|AF191659.1|AF19165
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273284|gb|AF191658.1|AF19165
</pre><p>Wow! There we are with a new alignment of just the six records, without having created
a temporary FASTA input file, or a temporary alignment output file. However, a word of
caution: Dealing with errors with this style of calling external programs is much more
complicated.
It also becomes far harder to diagnose problems, because you can&#X2019;t try running MUSCLE
manually outside of Biopython (because you don&#X2019;t have the input file to supply).
There can also be subtle cross platform issues (e.g. Windows versus Linux,
Python 2 versus Python 3), and how
you run your script can have an impact (e.g. at the command line, from IDLE or an
IDE, or as a GUI script). These are all generic Python issues though, and not
specific to Biopython.</p><p>If you find working directly with <span style="font-family:monospace">subprocess</span> like this scary, there is an
alternative. If you execute the tool with <span style="font-family:monospace">muscle_cline()</span> you can supply
any standard input as a big string, <span style="font-family:monospace">muscle_cline(stdin=...)</span>. So,
provided your data isn&#X2019;t very big, you can prepare the FASTA input in memory as
a string using <span style="font-family:monospace">StringIO</span> (see Section&#XA0;<a href="#sec%3Aappendix-handles">22.1</a>):</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; records = (r for r in SeqIO.parse("opuntia.fasta", "fasta") if len(r) &lt; 900)
&gt;&gt;&gt; from StringIO import StringIO
&gt;&gt;&gt; handle = StringIO()
&gt;&gt;&gt; SeqIO.write(records, handle, "fasta")
6
&gt;&gt;&gt; data = handle.getvalue()
</pre><p>You can then run the tool and parse the alignment as follows:</p><pre class="verbatim">&gt;&gt;&gt; stdout, stderr = muscle_cline(stdin=data)
&gt;&gt;&gt; from Bio import AlignIO
&gt;&gt;&gt; align = AlignIO.read(StringIO(stdout), "clustal")
&gt;&gt;&gt; print(align)
SingleLetterAlphabet() alignment with 6 rows and 900 columns
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273290|gb|AF191664.1|AF19166
TATACATTAAAGGAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273289|gb|AF191663.1|AF19166
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273287|gb|AF191661.1|AF19166
TATACATAAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273286|gb|AF191660.1|AF19166
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273285|gb|AF191659.1|AF19165
TATACATTAAAGAAGGGGGATGCGGATAAATGGAAAGGCGAAAG...AGA gi|6273284|gb|AF191658.1|AF19165
</pre><p>You might find this easier, but it does require more memory (RAM) for the strings
used for the input FASTA and output Clustal formatted data.</p>
<!--TOC subsection id="sec93" EMBOSS needle and water-->
<h3 id="sec93" class="subsection">6.4.5&#XA0;&#XA0;EMBOSS needle and water</h3><!--SEC END --><p>
<a id="seq:emboss-needle-water"></a>
The <a href="http://emboss.sourceforge.net/">EMBOSS</a> suite includes the <span style="font-family:monospace">water</span> and
<span style="font-family:monospace">needle</span> tools for Smith-Waterman algorithm local alignment, and Needleman-Wunsch
global alignment. The tools share the same style interface, so switching between the two
is trivial &#X2013; we&#X2019;ll just use <span style="font-family:monospace">needle</span> here.</p><p>Suppose you want to do a global pairwise alignment between two sequences, prepared in
FASTA format as follows:</p><pre class="verbatim">&gt;HBA_HUMAN
MVLSPADKTNVKAAWGKVGAHAGEYGAEALERMFLSFPTTKTYFPHFDLSHGSAQVKGHG
KKVADALTNAVAHVDDMPNALSALSDLHAHKLRVDPVNFKLLSHCLLVTLAAHLPAEFTP
AVHASLDKFLASVSTVLTSKYR
</pre><p>in a file <span style="font-family:monospace">alpha.fasta</span>, and secondly in a file <span style="font-family:monospace">beta.fasta</span>:</p><pre class="verbatim">&gt;HBB_HUMAN
MVHLTPEEKSAVTALWGKVNVDEVGGEALGRLLVVYPWTQRFFESFGDLSTPDAVMGNPK
VKAHGKKVLGAFSDGLAHLDNLKGTFATLSELHCDKLHVDPENFRLLGNVLVCVLAHHFG
KEFTPPVQAAYQKVVAGVANALAHKYH
</pre><p>Let&#X2019;s start by creating a complete <span style="font-family:monospace">needle</span> command line object in one go:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Emboss.Applications import NeedleCommandline
&gt;&gt;&gt; needle_cline = NeedleCommandline(asequence="alpha.faa", bsequence="beta.faa",
...                                  gapopen=10, gapextend=0.5, outfile="needle.txt")
&gt;&gt;&gt; print(needle_cline)
needle -outfile=needle.txt -asequence=alpha.faa -bsequence=beta.faa -gapopen=10 -gapextend=0.5
</pre><p>Why not try running this by hand at the command prompt? You should see it does a
pairwise comparison and records the output in the file <span style="font-family:monospace">needle.txt</span> (in the
default EMBOSS alignment file format).</p><p>Even if you have EMBOSS installed, running this command may not work &#X2013; you
might get a message about &#X201C;command not found&#X201D; (especially on Windows). This
probably means that the EMBOSS tools are not on your PATH environment
variable. You can either update your PATH setting, or simply tell Biopython
the full path to the tool, for example:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Emboss.Applications import NeedleCommandline
&gt;&gt;&gt; needle_cline = NeedleCommandline(r"C:\EMBOSS\needle.exe",
...                                  asequence="alpha.faa", bsequence="beta.faa",
...                                  gapopen=10, gapextend=0.5, outfile="needle.txt")
</pre><p>Remember in Python that for a default string <code>\n</code> or <code>\t</code> means a
new line or a tab &#X2013; which is why we&#X2019;re put a letter &#X201C;r&#X201D; at the start for a raw string.</p><p>At this point it might help to try running the EMBOSS tools yourself by hand at the
command line, to familiarise yourself the other options and compare them to the
Biopython help text:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Emboss.Applications import NeedleCommandline
&gt;&gt;&gt; help(NeedleCommandline)
...
</pre><p>Note that you can also specify (or change or look at) the settings like this:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Emboss.Applications import NeedleCommandline
&gt;&gt;&gt; needle_cline = NeedleCommandline()
&gt;&gt;&gt; needle_cline.asequence="alpha.faa"
&gt;&gt;&gt; needle_cline.bsequence="beta.faa"
&gt;&gt;&gt; needle_cline.gapopen=10
&gt;&gt;&gt; needle_cline.gapextend=0.5
&gt;&gt;&gt; needle_cline.outfile="needle.txt"
&gt;&gt;&gt; print(needle_cline)
needle -outfile=needle.txt -asequence=alpha.faa -bsequence=beta.faa -gapopen=10 -gapextend=0.5
&gt;&gt;&gt; print(needle_cline.outfile)
needle.txt
</pre><p>Next we want to use Python to run this command for us. As explained above,
for full control, we recommend you use the built in Python <span style="font-family:monospace">subprocess</span>
module, but for simple usage the wrapper object usually suffices:</p><pre class="verbatim">&gt;&gt;&gt; stdout, stderr = needle_cline()
&gt;&gt;&gt; print(stdout + stderr)
Needleman-Wunsch global alignment of two sequences
</pre><p>Next we can load the output file with <code>Bio.AlignIO</code> as
discussed earlier in this chapter, as the <span style="font-family:monospace">emboss</span> format:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import AlignIO
&gt;&gt;&gt; align = AlignIO.read("needle.txt", "emboss")
&gt;&gt;&gt; print(align)
SingleLetterAlphabet() alignment with 2 rows and 149 columns
MV-LSPADKTNVKAAWGKVGAHAGEYGAEALERMFLSFPTTKTY...KYR HBA_HUMAN
MVHLTPEEKSAVTALWGKV--NVDEVGGEALGRLLVVYPWTQRF...KYH HBB_HUMAN
</pre><p>In this example, we told EMBOSS to write the output to a file, but you
<em>can</em> tell it to write the output to stdout instead (useful if you
don&#X2019;t want a temporary output file to get rid of &#X2013; use
<span style="font-family:monospace">stdout=True</span> rather than the <span style="font-family:monospace">outfile</span> argument), and
also to read <em>one</em> of the one of the inputs from stdin (e.g.
<span style="font-family:monospace">asequence="stdin"</span>, much like in the MUSCLE example in the
section above).</p><p>This has only scratched the surface of what you can do with <span style="font-family:monospace">needle</span>
and <span style="font-family:monospace">water</span>. One useful trick is that the second file can contain
multiple sequences (say five), and then EMBOSS will do five pairwise
alignments.</p><p>Note - Biopython includes its own pairwise alignment code in the <code>Bio.pairwise2</code>
module (written in C for speed, but with a pure Python fallback available too). This
doesn&#X2019;t work with alignment objects, so we have not covered it within this chapter.
See the module&#X2019;s docstring (built in help) for details.</p>
<!--TOC chapter id="sec94" BLAST-->
<h1 id="sec94" class="chapter">Chapter&#XA0;7&#XA0;&#XA0;BLAST</h1><!--SEC END --><p>
<a id="chapter:blast"></a>
Hey, everybody loves BLAST right? I mean, geez, how can get it get any easier to do comparisons between one of your sequences and every other sequence in the known world? But, of course, this section isn&#X2019;t about how cool BLAST is, since we already know that. It is about the problem with BLAST &#X2013; it can be really difficult to deal with the volume of data generated by large runs, and to automate BLAST runs in general.</p><p>Fortunately, the Biopython folks know this only too well, so they&#X2019;ve developed lots of tools for dealing with BLAST and making things much easier. This section details how to use these tools and do useful things with them.</p><p>Dealing with BLAST can be split up into two steps, both of which can be done from within Biopython.
Firstly, running BLAST for your query sequence(s), and getting some output.
Secondly, parsing the BLAST output in Python for further analysis.</p><p>Your first introduction to running BLAST was probably via the NCBI web-service.
In fact, there are lots of ways you can run BLAST, which can be categorised several ways.
The most important distinction is running BLAST locally (on your own machine),
and running BLAST remotely (on another machine, typically the NCBI servers).
We&#X2019;re going to start this chapter by invoking the NCBI online BLAST service
from within a Python script.</p><p><em>NOTE</em>: The following Chapter&#XA0;<a href="#chapter%3Asearchio">8</a> describes
<code>Bio.SearchIO</code>, an <em>experimental</em> module in Biopython. We
intend this to ultimately replace the older <code>Bio.Blast</code> module, as it
provides a more general framework handling other related sequence
searching tools as well. However, until that is declared stable, for
production code please continue to use the <code>Bio.Blast</code> module
for dealing with NCBI BLAST.</p>
<!--TOC section id="sec95" Running BLAST over the Internet-->
<h2 id="sec95" class="section">7.1&#XA0;&#XA0;Running BLAST over the Internet</h2><!--SEC END --><p>
<a id="sec:running-www-blast"></a></p><p>We use the function <code>qblast()</code> in the <code>Bio.Blast.NCBIWWW</code> module
call the online version of BLAST. This has three non-optional arguments:
</p><ul class="itemize"><li class="li-itemize">
The first argument is the blast program to use for the search, as a
lower case string. The options and descriptions of the programs are
available at <a href="http://www.ncbi.nlm.nih.gov/BLAST/blast_program.shtml"><span style="font-family:monospace">http://www.ncbi.nlm.nih.gov/BLAST/blast_program.shtml</span></a>.
Currently <code>qblast</code> only works with blastn, blastp, blastx, tblast
and tblastx.
</li><li class="li-itemize">The second argument specifies the databases to search against. Again,
the options for this are available on the NCBI web pages at
<a href="http://www.ncbi.nlm.nih.gov/BLAST/blast_databases.shtml"><span style="font-family:monospace">http://www.ncbi.nlm.nih.gov/BLAST/blast_databases.shtml</span></a>.
</li><li class="li-itemize">The third argument is a string containing your query sequence. This
can either be the sequence itself, the sequence in fasta format,
or an identifier like a GI number.
</li></ul><p>The <code>qblast</code> function also take a number of other option arguments
which are basically analogous to the different parameters you can set
on the BLAST web page. We&#X2019;ll just highlight a few of them here:</p><ul class="itemize"><li class="li-itemize">
The <code>qblast</code> function can return the BLAST results in various
formats, which you can choose with the optional <code>format_type</code> keyword:
<code>"HTML"</code>, <code>"Text"</code>, <code>"ASN.1"</code>, or <code>"XML"</code>.
The default is <code>"XML"</code>, as that is the format expected by the parser,
described in section&#XA0;<a href="#sec%3Aparsing-blast">7.3</a> below.
</li><li class="li-itemize">The argument <code>expect</code> sets the expectation or e-value threshold.
</li></ul><p>For more about the optional BLAST arguments, we refer you to the NCBI&#X2019;s own
documentation, or that built into Biopython:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIWWW
&gt;&gt;&gt; help(NCBIWWW.qblast)
...
</pre><p>Note that the default settings on the NCBI BLAST website are not quite
the same as the defaults on QBLAST. If you get different results, you&#X2019;ll
need to check the parameters (e.g. the expectation value threshold and
the gap values).</p><p>For example, if you have a nucleotide sequence you want to search against
the nucleotide database (nt) using BLASTN, and you know the GI number of your
query sequence, you can use:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIWWW
&gt;&gt;&gt; result_handle = NCBIWWW.qblast("blastn", "nt", "8332116")
</pre><p>Alternatively, if we have our query sequence already in a FASTA formatted
file, we just need to open the file and read in this record as a string,
and use that as the query argument:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIWWW
&gt;&gt;&gt; fasta_string = open("m_cold.fasta").read()
&gt;&gt;&gt; result_handle = NCBIWWW.qblast("blastn", "nt", fasta_string)
</pre><p>We could also have read in the FASTA file as a <code>SeqRecord</code> and then
supplied just the sequence itself:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIWWW
&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; record = SeqIO.read("m_cold.fasta", format="fasta")
&gt;&gt;&gt; result_handle = NCBIWWW.qblast("blastn", "nt", record.seq)
</pre><p>Supplying just the sequence means that BLAST will assign an identifier
for your sequence automatically. You might prefer to use the
<code>SeqRecord</code> object&#X2019;s format method to make a fasta string
(which will include the existing identifier):</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIWWW
&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; record = SeqIO.read("m_cold.fasta", format="fasta")
&gt;&gt;&gt; result_handle = NCBIWWW.qblast("blastn", "nt", record.format("fasta"))
</pre><p>This approach makes more sense if you have your sequence(s) in a
non-FASTA file format which you can extract using <code>Bio.SeqIO</code>
(see Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>).</p><p>Whatever arguments you give the <code>qblast()</code> function, you should
get back your results in a handle object (by default in XML format).
The next step would be to parse the XML output into Python objects
representing the search results (Section&#XA0;<a href="#sec%3Aparsing-blast">7.3</a>),
but you might want to save a local copy of the output file first.
I find this especially useful when debugging my code that extracts
info from the BLAST results (because re-running the online search
is slow and wastes the NCBI computer time).</p><p><a id="sec:saving-blast-output"></a></p><p>We need to be a bit careful since we can use <code>result_handle.read()</code> to
read the BLAST output only once &#X2013; calling <code>result_handle.read()</code> again
returns an empty string.</p><pre class="verbatim">&gt;&gt;&gt; save_file = open("my_blast.xml", "w")
&gt;&gt;&gt; save_file.write(result_handle.read())
&gt;&gt;&gt; save_file.close()
&gt;&gt;&gt; result_handle.close()
</pre><p>After doing this, the results are in the file <code>my_blast.xml</code> and the
original handle has had all its data extracted (so we closed it). However,
the <code>parse</code> function of the BLAST parser (described
in&#XA0;<a href="#sec%3Aparsing-blast">7.3</a>) takes a file-handle-like object, so
we can just open the saved file for input:</p><pre class="verbatim">&gt;&gt;&gt; result_handle = open("my_blast.xml")
</pre><p>Now that we&#X2019;ve got the BLAST results back into a handle again, we are ready
to do something with them, so this leads us right into the parsing section
(see Section&#XA0;<a href="#sec%3Aparsing-blast">7.3</a> below). You may want to jump ahead to
that now &#X2026;.</p>
<!--TOC section id="sec96" Running BLAST locally-->
<h2 id="sec96" class="section">7.2&#XA0;&#XA0;Running BLAST locally</h2><!--SEC END --><p>
<a id="sec:running-local-blast"></a></p>
<!--TOC subsection id="sec97" Introduction-->
<h3 id="sec97" class="subsection">7.2.1&#XA0;&#XA0;Introduction</h3><!--SEC END --><p>Running BLAST locally (as opposed to over the internet, see
Section&#XA0;<a href="#sec%3Arunning-www-blast">7.1</a>) has at least major two advantages:
</p><ul class="itemize"><li class="li-itemize">
Local BLAST may be faster than BLAST over the internet;
</li><li class="li-itemize">Local BLAST allows you to make your own database to search for sequences against.
</li></ul><p>
Dealing with proprietary or unpublished sequence data can be another reason to run BLAST
locally. You may not be allowed to redistribute the sequences, so submitting them to the
NCBI as a BLAST query would not be an option.</p><p>Unfortunately, there are some major drawbacks too &#X2013; installing all the bits and getting
it setup right takes some effort:
</p><ul class="itemize"><li class="li-itemize">
Local BLAST requires command line tools to be installed.
</li><li class="li-itemize">Local BLAST requires (large) BLAST databases to be setup (and potentially kept up to date).
</li></ul><p>To further confuse matters there are several different BLAST packages available,
and there are also other tools which can produce imitation BLAST output files, such as BLAT.</p>
<!--TOC subsection id="sec98" Standalone NCBI BLAST+-->
<h3 id="sec98" class="subsection">7.2.2&#XA0;&#XA0;Standalone NCBI BLAST+</h3><!--SEC END --><p>The &#X201C;new&#X201D;
<a href="http://blast.ncbi.nlm.nih.gov/Blast.cgi?CMD=Web&PAGE_TYPE=BlastDocs&DOC_TYPE=Download">NCBI BLAST+</a> suite was released in 2009. This replaces the old NCBI &#X201C;legacy&#X201D; BLAST
package (see below).</p><p>This section will show briefly how to use these tools from within Python. If you have
already read or tried the alignment tool examples in Section&#XA0;<a href="#sec%3Aalignment-tools">6.4</a>
this should all seem quite straightforward. First, we construct a command line string
(as you would type in at the command line prompt if running standalone BLAST by hand).
Then we can execute this command from within Python.</p><p>For example, taking a FASTA file of gene nucleotide sequences, you might want to
run a BLASTX (translation) search against the non-redundant (NR) protein database.
Assuming you (or your systems administrator) has downloaded and installed the NR
database, you might run:</p><pre class="verbatim">blastx -query opuntia.fasta -db nr -out opuntia.xml -evalue 0.001 -outfmt 5
</pre><p>This should run BLASTX against the NR database, using an expectation cut-off value
of 0.001 and produce XML output to the specified file (which we can then parse).
On my computer this takes about six minutes - a good reason to save the output
to a file so you and repeat any analysis as needed.</p><p>From within Biopython we can use the NCBI BLASTX wrapper from the
<code>Bio.Blast.Applications</code> module to build the command line string,
and run it:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast.Applications import NcbiblastxCommandline
&gt;&gt;&gt; help(NcbiblastxCommandline)
...
&gt;&gt;&gt; blastx_cline = NcbiblastxCommandline(query="opuntia.fasta", db="nr", evalue=0.001,
...                                      outfmt=5, out="opuntia.xml")
&gt;&gt;&gt; blastx_cline
NcbiblastxCommandline(cmd='blastx', out='opuntia.xml', outfmt=5, query='opuntia.fasta',
db='nr', evalue=0.001)
&gt;&gt;&gt; print(blastx_cline)
blastx -out opuntia.xml -outfmt 5 -query opuntia.fasta -db nr -evalue 0.001
&gt;&gt;&gt; stdout, stderr = blastx_cline()
</pre><p>In this example there shouldn&#X2019;t be any output from BLASTX to the terminal,
so stdout and stderr should be empty. You may want to check the output file
<code>opuntia.xml</code> has been created.</p><p>As you may recall from earlier examples in the tutorial, the <code>opuntia.fasta</code>
contains seven sequences, so the BLAST XML output should contain multiple results.
Therefore use <code>Bio.Blast.NCBIXML.parse()</code> to parse it as described below in
Section&#XA0;<a href="#sec%3Aparsing-blast">7.3</a>.</p>
<!--TOC subsection id="sec99" Other versions of BLAST-->
<h3 id="sec99" class="subsection">7.2.3&#XA0;&#XA0;Other versions of BLAST</h3><!--SEC END --><p>NCBI BLAST+ (written in C++) was first released in 2009 as a replacement for
the original NCBI &#X201C;legacy&#X201D; BLAST (written in C) which is no longer being updated.
There were a lot of changes &#X2013; the old version had a single core command line
tool <code>blastall</code> which covered multiple different BLAST search types (which
are now separate commands in BLAST+), and all the command line options
were renamed.
Biopython&#X2019;s wrappers for the NCBI &#X201C;legacy&#X201D; BLAST tools have been deprecated
and will be removed in a future release.
To try to avoid confusion, we do not cover calling these old tools from Biopython
in this tutorial.</p><p>You may also come across <a href="http://blast.wustl.edu/">Washington University BLAST</a>
(WU-BLAST), and its successor, <a href="http://blast.advbiocomp.com">Advanced Biocomputing
BLAST</a> (AB-BLAST, released in 2009, not free/open source). These packages include
the command line tools <code>wu-blastall</code> and <code>ab-blastall</code>, which mimicked
<code>blastall</code> from the NCBI &#X201C;legacy&#X201D; BLAST stuie.
Biopython does not currently provide wrappers for calling these tools, but should be able
to parse any NCBI compatible output from them.</p>
<!--TOC section id="sec100" Parsing BLAST output-->
<h2 id="sec100" class="section">7.3&#XA0;&#XA0;Parsing BLAST output</h2><!--SEC END --><p>
<a id="sec:parsing-blast"></a></p><p>As mentioned above, BLAST can generate output in various formats, such as
XML, HTML, and plain text. Originally, Biopython had parsers for BLAST
plain text and HTML output, as these were the only output formats offered
at the time. Unfortunately, the BLAST output in these formats kept changing,
each time breaking the Biopython parsers. Our HTML BLAST parser has been
removed, but the plain text BLAST parser is still available (see
Section&#XA0;<a href="#sec%3Aparsing-blast-deprecated">7.5</a>). Use it at your own risk,
it may or may not work, depending on which BLAST version you&#X2019;re using.</p><p>As keeping up with changes in BLAST
became a hopeless endeavor, especially with users running different BLAST
versions, we now recommend to parse the output in XML format, which can be
generated by recent versions of BLAST. Not only is the XML output more stable
than the plain text and HTML output, it is also much easier to parse
automatically, making Biopython a whole lot more stable.</p><p>You can get BLAST output in XML format in various ways. For the parser, it
doesn&#X2019;t matter how the output was generated, as long as it is in the XML format.
</p><ul class="itemize"><li class="li-itemize">
You can use Biopython to run BLAST over the internet, as described in
section&#XA0;<a href="#sec%3Arunning-www-blast">7.1</a>.
</li><li class="li-itemize">You can use Biopython to run BLAST locally, as described in
section&#XA0;<a href="#sec%3Arunning-local-blast">7.2</a>.
</li><li class="li-itemize">You can do the BLAST search yourself on the NCBI site through your
web browser, and then save the results. You need to choose XML as the format
in which to receive the results, and save the final BLAST page you get
(you know, the one with all of the interesting results!) to a file.
</li><li class="li-itemize">You can also run BLAST locally without using Biopython, and save
the output in a file. Again, you need to choose XML as the format in which
to receive the results.
</li></ul><p>
The important point is that you do not have to use Biopython
scripts to fetch the data in order to be able to parse it.
Doing things in one of these ways, you then need to get a handle
to the results. In Python, a handle is just a nice general way of
describing input to any info source so that the info can be retrieved
using <code>read()</code> and <code>readline()</code> functions
(see Section&#XA0;sec:appendix-handles).</p><p>If you followed the code above for interacting with BLAST through a
script, then you already have <code>result_handle</code>, the handle to the
BLAST results. For example, using a GI number to do an online search:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIWWW
&gt;&gt;&gt; result_handle = NCBIWWW.qblast("blastn", "nt", "8332116")
</pre><p>If instead you ran BLAST some other way, and have the
BLAST output (in XML format) in the file <code>my_blast.xml</code>, all you
need to do is to open the file for reading:</p><pre class="verbatim">&gt;&gt;&gt; result_handle = open("my_blast.xml")
</pre><p>Now that we&#X2019;ve got a handle, we are ready to parse the output. The
code to parse it is really quite small. If you expect a single
BLAST result (i.e. you used a single query):</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIXML
&gt;&gt;&gt; blast_record = NCBIXML.read(result_handle)
</pre><p>or, if you have lots of results (i.e. multiple query sequences):</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIXML
&gt;&gt;&gt; blast_records = NCBIXML.parse(result_handle)
</pre><p>Just like <code>Bio.SeqIO</code> and <code>Bio.AlignIO</code>
(see Chapters&#XA0;<a href="#chapter%3ABio.SeqIO">5</a> and&#XA0;<a href="#chapter%3ABio.AlignIO">6</a>),
we have a pair of input functions, <code>read</code> and <code>parse</code>, where
<code>read</code> is for when you have exactly one object, and <code>parse</code>
is an iterator for when you can have lots of objects &#X2013; but instead of
getting <code>SeqRecord</code> or <code>MultipleSeqAlignment</code> objects, we
get BLAST record objects.</p><p>To be able to handle the situation where the BLAST file may be huge,
containing thousands of results, <code>NCBIXML.parse()</code> returns an
iterator. In plain English, an iterator allows you to step through
the BLAST output, retrieving BLAST records one by one for each BLAST
search result:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIXML
&gt;&gt;&gt; blast_records = NCBIXML.parse(result_handle)
&gt;&gt;&gt; blast_record = next(blast_records)
# ... do something with blast_record
&gt;&gt;&gt; blast_record = next(blast_records)
# ... do something with blast_record
&gt;&gt;&gt; blast_record = next(blast_records)
# ... do something with blast_record
&gt;&gt;&gt; blast_record = next(blast_records)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
StopIteration
# No further records
</pre><p>Or, you can use a <code>for</code>-loop:
</p><pre class="verbatim">&gt;&gt;&gt; for blast_record in blast_records:
...     # Do something with blast_record
</pre><p>Note though that you can step through the BLAST records only once.
Usually, from each BLAST record you would save the information that
you are interested in. If you want to save all returned BLAST records,
you can convert the iterator into a list:
</p><pre class="verbatim">&gt;&gt;&gt; blast_records = list(blast_records)
</pre><p>Now you can access each BLAST record in the list with an index as usual.
If your BLAST file is huge though, you may run into memory problems trying to
save them all in a list.</p><p>Usually, you&#X2019;ll be running one BLAST search at a time. Then, all you need
to do is to pick up the first (and only) BLAST record in <code>blast_records</code>:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIXML
&gt;&gt;&gt; blast_records = NCBIXML.parse(result_handle)
&gt;&gt;&gt; blast_record = next(blast_records)
</pre><p>or more elegantly:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIXML
&gt;&gt;&gt; blast_record = NCBIXML.read(result_handle)
</pre><p>I guess by now you&#X2019;re wondering what is in a BLAST record.</p>
<!--TOC section id="sec101" The BLAST record class-->
<h2 id="sec101" class="section">7.4&#XA0;&#XA0;The BLAST record class</h2><!--SEC END --><p>A BLAST Record contains everything you might ever want to extract from the
BLAST output. Right now we&#X2019;ll just show
an example of how to get some info out of the BLAST report, but if you
want something in particular that is not described here, look at the
info on the record class in detail, and take a gander into the code or
automatically generated documentation &#X2013; the docstrings have lots of
good info about what is stored in each piece of information.</p><p>To continue with our example, let&#X2019;s just print out some summary info
about all hits in our blast report greater than a particular
threshold. The following code does this:</p><pre class="verbatim">&gt;&gt;&gt; E_VALUE_THRESH = 0.04

&gt;&gt;&gt; for alignment in blast_record.alignments:
...     for hsp in alignment.hsps:
...         if hsp.expect &lt; E_VALUE_THRESH:
...             print('****Alignment****')
...             print('sequence:', alignment.title)
...             print('length:', alignment.length)
...             print('e value:', hsp.expect)
...             print(hsp.query[0:75] + '...')
...             print(hsp.match[0:75] + '...')
...             print(hsp.sbjct[0:75] + '...')
</pre><p>This will print out summary reports like the following:</p><pre class="verbatim">****Alignment****
sequence: &gt;gb|AF283004.1|AF283004 Arabidopsis thaliana cold acclimation protein WCOR413-like protein
alpha form mRNA, complete cds
length: 783
e value: 0.034
tacttgttgatattggatcgaacaaactggagaaccaacatgctcacgtcacttttagtcccttacatattcctc...
||||||||| | ||||||||||| || ||||  || || |||||||| |||||| |  | |||||||| ||| ||...
tacttgttggtgttggatcgaaccaattggaagacgaatatgctcacatcacttctcattccttacatcttcttc...
</pre><p>Basically, you can do anything you want to with the info in the BLAST
report once you have parsed it. This will, of course, depend on what
you want to use it for, but hopefully this helps you get started on
doing what you need to do!</p><p>An important consideration for extracting information from a BLAST report is the type of objects that the information is stored in. In Biopython, the parsers return <code>Record</code> objects, either <code>Blast</code> or <code>PSIBlast</code> depending on what you are parsing. These objects are defined in <code>Bio.Blast.Record</code> and are quite complete.</p><p>Here are my attempts at UML class diagrams for the <code>Blast</code> and <code>PSIBlast</code> record classes. If you are good at UML and see mistakes/improvements that can be made, please let me know. The Blast class diagram is shown in Figure&#XA0;<a href="#fig%3Ablastrecord">7.4</a>.</p><p>
<a id="fig:blastrecord"></a>
<img src="images/BlastRecord.png" width=650, height=750>
</p><p>The PSIBlast record object is similar, but has support for the rounds that are used in the iteration steps of PSIBlast. The class diagram for PSIBlast is shown in Figure&#XA0;<a href="#fig%3Apsiblastrecord">7.4</a>.</p><p>
<a id="fig:psiblastrecord"></a>
<img src="images/PSIBlastRecord.png" width=650, height=750>
</p>
<!--TOC section id="sec102" Deprecated BLAST parsers-->
<h2 id="sec102" class="section">7.5&#XA0;&#XA0;Deprecated BLAST parsers</h2><!--SEC END --><p>
<a id="sec:parsing-blast-deprecated"></a></p><p>Older versions of Biopython had parsers for BLAST output in plain text or HTML
format. Over the years, we discovered that it is very hard to maintain these
parsers in working order. Basically, any small change to the BLAST output in
newly released BLAST versions tends to cause the plain text and HTML parsers
to break. We therefore recommend parsing BLAST output in XML format, as
described in section&#XA0;<a href="#sec%3Aparsing-blast">7.3</a>.</p><p>Depending on which BLAST versions or programs you&#X2019;re using, our plain text BLAST parser may or may not work. Use it at your own risk!</p>
<!--TOC subsection id="sec103" Parsing plain-text BLAST output-->
<h3 id="sec103" class="subsection">7.5.1&#XA0;&#XA0;Parsing plain-text BLAST output</h3><!--SEC END --><p>The plain text BLAST parser is located in <code>Bio.Blast.NCBIStandalone</code>.</p><p>As with the XML parser, we need to have a handle object that we can pass to the parser. The handle must implement the <code>readline()</code> method and do this properly. The common ways to get such a handle are to either use the provided <code>blastall</code> or <code>blastpgp</code> functions to run the local blast, or to run a local blast via the command line, and then do something like the following:</p><pre class="verbatim">&gt;&gt;&gt; result_handle = open("my_file_of_blast_output.txt")
</pre><p>Well, now that we&#X2019;ve got a handle (which we&#X2019;ll call <code>result_handle</code>),
we are ready to parse it. This can be done with the following code:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIStandalone
&gt;&gt;&gt; blast_parser = NCBIStandalone.BlastParser()
&gt;&gt;&gt; blast_record = blast_parser.parse(result_handle)
</pre><p>This will parse the BLAST report into a Blast Record class (either a Blast or a PSIBlast record, depending on what you are parsing) so that you can extract the information from it. In our case, let&#X2019;s just use print out a quick summary of all of the alignments greater than some threshold value.</p><pre class="verbatim">&gt;&gt;&gt; E_VALUE_THRESH = 0.04
&gt;&gt;&gt; for alignment in blast_record.alignments:
...     for hsp in alignment.hsps:
...         if hsp.expect &lt; E_VALUE_THRESH:
...             print('****Alignment****')
...             print('sequence:', alignment.title)
...             print('length:', alignment.length)
...             print('e value:', hsp.expect)
...             print(hsp.query[0:75] + '...')
...             print(hsp.match[0:75] + '...')
...             print(hsp.sbjct[0:75] + '...')
</pre><p>If you also read the section&#XA0;<a href="#sec%3Aparsing-blast">7.3</a> on parsing BLAST XML output, you&#X2019;ll notice that the above code is identical to what is found in that section. Once you parse something into a record class you can deal with it independent of the format of the original BLAST info you were parsing. Pretty snazzy!</p><p>Sure, parsing one record is great, but I&#X2019;ve got a BLAST file with tons of records &#X2013; how can I parse them all? Well, fear not, the answer lies in the very next section.</p>
<!--TOC subsection id="sec104" Parsing a plain-text BLAST file full of BLAST runs-->
<h3 id="sec104" class="subsection">7.5.2&#XA0;&#XA0;Parsing a plain-text BLAST file full of BLAST runs</h3><!--SEC END --><p>We can do this using the blast iterator. To set up an iterator, we first set up a parser, to parse our blast reports in Blast Record objects:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIStandalone
&gt;&gt;&gt; blast_parser = NCBIStandalone.BlastParser()
</pre><p>Then we will assume we have a handle to a bunch of blast records, which we&#X2019;ll call <code>result_handle</code>. Getting a handle is described in full detail above in the blast parsing sections.</p><p>Now that we&#X2019;ve got a parser and a handle, we are ready to set up the iterator with the following command:</p><pre class="verbatim">&gt;&gt;&gt; blast_iterator = NCBIStandalone.Iterator(result_handle, blast_parser)
</pre><p>The second option, the parser, is optional. If we don&#X2019;t supply a parser, then the iterator will just return the raw BLAST reports one at a time.</p><p>Now that we&#X2019;ve got an iterator, we start retrieving blast records (generated by our parser) using <code>next()</code>:</p><pre class="verbatim">&gt;&gt;&gt; blast_record = next(blast_iterator)
</pre><p>Each call to next will return a new record that we can deal with. Now we can iterate through this records and generate our old favorite, a nice little blast report:</p><pre class="verbatim">&gt;&gt;&gt; for blast_record in blast_iterator:
...     E_VALUE_THRESH = 0.04
...     for alignment in blast_record.alignments:
...         for hsp in alignment.hsps:
...             if hsp.expect &lt; E_VALUE_THRESH:
...                 print('****Alignment****')
...                 print('sequence:', alignment.title)
...                 print('length:', alignment.length)
...                 print('e value:', hsp.expect)
...                 if len(hsp.query) &gt; 75:
...                     dots = '...'
...                 else:
...                     dots = ''
...                 print(hsp.query[0:75] + dots)
...                 print(hsp.match[0:75] + dots)
...                 print(hsp.sbjct[0:75] + dots)
</pre><p>The iterator allows you to deal with huge blast records without any memory problems, since things are read in one at a time. I have parsed tremendously huge files without any problems using this.</p>
<!--TOC subsection id="sec105" Finding a bad record somewhere in a huge plain-text BLAST file-->
<h3 id="sec105" class="subsection">7.5.3&#XA0;&#XA0;Finding a bad record somewhere in a huge plain-text BLAST file</h3><!--SEC END --><p>One really ugly problem that happens to me is that I&#X2019;ll be parsing a huge blast file for a while, and the parser will bomb out with a ValueError. This is a serious problem, since you can&#X2019;t tell if the ValueError is due to a parser problem, or a problem with the BLAST. To make it even worse, you have no idea where the parse failed, so you can&#X2019;t just ignore the error, since this could be ignoring an important data point.</p><p>We used to have to make a little script to get around this problem, but the <code>Bio.Blast</code> module now includes a <code>BlastErrorParser</code> which really helps make this easier. The <code>BlastErrorParser</code> works very similar to the regular <code>BlastParser</code>, but it adds an extra layer of work by catching ValueErrors that are generated by the parser, and attempting to diagnose the errors.</p><p>Let&#X2019;s take a look at using this parser &#X2013; first we define the file we are going to parse and the file to write the problem reports to:</p><pre class="verbatim">&gt;&gt;&gt; import os
&gt;&gt;&gt; blast_file = os.path.join(os.getcwd(), "blast_out", "big_blast.out")
&gt;&gt;&gt; error_file = os.path.join(os.getcwd(), "blast_out", "big_blast.problems")
</pre><p>Now we want to get a <code>BlastErrorParser</code>:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Blast import NCBIStandalone
&gt;&gt;&gt; error_handle = open(error_file, "w")
&gt;&gt;&gt; blast_error_parser = NCBIStandalone.BlastErrorParser(error_handle)
</pre><p>Notice that the parser take an optional argument of a handle. If a handle is passed, then the parser will write any blast records which generate a ValueError to this handle. Otherwise, these records will not be recorded.</p><p>Now we can use the <code>BlastErrorParser</code> just like a regular blast parser. Specifically, we might want to make an iterator that goes through our blast records one at a time and parses them with the error parser:</p><pre class="verbatim">&gt;&gt;&gt; result_handle = open(blast_file)
&gt;&gt;&gt; iterator = NCBIStandalone.Iterator(result_handle, blast_error_parser)
</pre><p>We can read these records one a time, but now we can catch and deal with errors that are due to problems with Blast (and not with the parser itself):</p><pre class="verbatim">&gt;&gt;&gt; try:
...     next_record = next(iterator)
... except NCBIStandalone.LowQualityBlastError as info:
...     print("LowQualityBlastError detected in id %s" % info[1])
</pre><p>The <code>next()</code> functionality is normally called indirectly via a <code>for</code>-loop.
Right now the <code>BlastErrorParser</code> can generate the following errors:</p><ul class="itemize"><li class="li-itemize">
<code>ValueError</code> &#X2013; This is the same error generated by the regular BlastParser, and is due to the parser not being able to parse a specific file. This is normally either due to a bug in the parser, or some kind of discrepancy between the version of BLAST you are using and the versions the parser is able to handle.</li><li class="li-itemize"><code>LowQualityBlastError</code> &#X2013; When BLASTing a sequence that is of really bad quality (for example, a short sequence that is basically a stretch of one nucleotide), it seems that Blast ends up masking out the entire sequence and ending up with nothing to parse. In this case it will produce a truncated report that causes the parser to generate a ValueError. <code>LowQualityBlastError</code> is reported in these cases. This error returns an info item with the following information:
<ul class="itemize"><li class="li-itemize">
<code>item[0]</code> &#X2013; The error message
</li><li class="li-itemize"><code>item[1]</code> &#X2013; The id of the input record that caused the error. This is really useful if you want to record all of the records that are causing problems.
</li></ul>
</li></ul><p>As mentioned, with each error generated, the BlastErrorParser will write the offending record to the specified <code>error_handle</code>. You can then go ahead and look and these and deal with them as you see fit. Either you will be able to debug the parser with a single blast report, or will find out problems in your blast runs. Either way, it will definitely be a useful experience!</p><p>Hopefully the <code>BlastErrorParser</code> will make it much easier to debug and deal with large Blast files.</p>
<!--TOC section id="sec106" Dealing with PSI-BLAST-->
<h2 id="sec106" class="section">7.6&#XA0;&#XA0;Dealing with PSI-BLAST</h2><!--SEC END --><p>You can run the standalone version of PSI-BLAST (the legacy NCBI command line
tool <code>blastpgp</code>, or its replacement <code>psiblast</code>) using the wrappers
in <code>Bio.Blast.Applications</code> module.</p><p>At the time of writing, the NCBI do not appear to support tools running a
PSI-BLAST search via the internet.</p><p>Note that the <code>Bio.Blast.NCBIXML</code> parser can read the XML output from
current versions of PSI-BLAST, but information like which sequences in each
iteration is new or reused isn&#X2019;t present in the XML file.
If you care about this information you may have more joy with the plain text
output and the <code>PSIBlastParser</code> in <code>Bio.Blast.NCBIStandalone</code>.</p>
<!--TOC section id="sec107" Dealing with RPS-BLAST-->
<h2 id="sec107" class="section">7.7&#XA0;&#XA0;Dealing with RPS-BLAST</h2><!--SEC END --><p>You can run the standalone version of RPS-BLAST (either the legacy NCBI
command line tool <code>rpsblast</code>, or its replacement with the same name)
using the wrappers in <code>Bio.Blast.Applications</code> module.</p><p>At the time of writing, the NCBI do not appear to support tools running an
RPS-BLAST search via the internet.</p><p>You can use the <code>Bio.Blast.NCBIXML</code> parser to read the XML output from
current versions of RPS-BLAST.</p>
<!--TOC chapter id="sec108" BLAST and other sequence search tools (<span style="font-style:italic">experimental code</span>)-->
<h1 id="sec108" class="chapter">Chapter&#XA0;8&#XA0;&#XA0;BLAST and other sequence search tools (<span style="font-style:italic">experimental code</span>)</h1><!--SEC END --><p>
<a id="chapter:searchio"></a></p><p><em>WARNING</em>: This chapter of the Tutorial describes an <em>experimental</em>
module in Biopython. It is being included in Biopython and documented
here in the tutorial in a pre-final state to allow a period of feedback
and refinement before we declare it stable. Until then the details will
probably change, and any scripts using the current <code>Bio.SearchIO</code>
would need to be updated. Please keep this in mind! For stable code
working with NCBI BLAST, please continue to use Bio.Blast described
in the preceding Chapter&#XA0;<a href="#chapter%3Ablast">7</a>.</p><p>Biological sequence identification is an integral part of bioinformatics.
Several tools are available for this, each with their own algorithms and
approaches, such as BLAST (arguably the most popular), FASTA, HMMER, and many
more. In general, these tools usually use your sequence to search a database of
potential matches. With the growing number of known sequences (hence the
growing number of potential matches), interpreting the results becomes
increasingly hard as there could be hundreds or even thousands of potential
matches. Naturally, manual interpretation of these searches&#X2019; results is out of
the question. Moreover, you often need to work with several sequence search
tools, each with its own statistics, conventions, and output format. Imagine how
daunting it would be when you need to work with multiple sequences using
multiple search tools.</p><p>We know this too well ourselves, which is why we created the <code>Bio.SearchIO</code>
submodule in Biopython. <code>Bio.SearchIO</code> allows you to extract information
from your search results in a convenient way, while also dealing with the
different standards and conventions used by different search tools.
The name <code>SearchIO</code> is a homage to BioPerl&#X2019;s module of the same name.</p><p>In this chapter, we&#X2019;ll go through the main features of <code>Bio.SearchIO</code> to
show what it can do for you. We&#X2019;ll use two popular search tools along the way:
BLAST and BLAT. They are used merely for illustrative purposes, and you should
be able to adapt the workflow to any other search tools supported by
<code>Bio.SearchIO</code> in a breeze. You&#X2019;re very welcome to follow along with the
search output files we&#X2019;ll be using. The BLAST output file can be downloaded
<a href="http://biopython.org/SRC/Tests/Tutorial/my_blast.xml">here</a>,
and the BLAT output file
<a href="http://biopython.org/SRC/Tests/Tutorial/my_blat.psl">here</a>.
Both output files were generated using this sequence:</p><pre class="verbatim">&gt;mystery_seq
CCCTCTACAGGGAAGCGCTTTCTGTTGTCTGAAAGAAAAGAAAGTGCTTCCTTTTAGAGGG
</pre><p>The BLAST result is an XML file generated using <code>blastn</code> against the NCBI
<code>refseq_rna</code> database. For BLAT, the sequence database was the February 2009
<code>hg19</code> human genome draft and the output format is PSL.</p><p>We&#X2019;ll start from an introduction to the <code>Bio.SearchIO</code> object model. The
model is the representation of your search results, thus it is core to
<code>Bio.SearchIO</code> itself. After that, we&#X2019;ll check out the main functions in
<code>Bio.SearchIO</code> that you may often use.</p><p>Now that we&#X2019;re all set, let&#X2019;s go to the first step: introducing the core
object model.</p>
<!--TOC section id="sec109" The SearchIO object model-->
<h2 id="sec109" class="section">8.1&#XA0;&#XA0;The SearchIO object model</h2><!--SEC END --><p>
<a id="sec:searchio-model"></a></p><p>Despite the wildly differing output styles among many sequence search tools,
it turns out that their underlying concept is similar:</p><ul class="itemize"><li class="li-itemize">
The output file may contain results from one or more search queries.
</li><li class="li-itemize">In each search query, you will see one or more hits from the given
search database.
</li><li class="li-itemize">In each database hit, you will see one or more regions containing the
actual sequence alignment between your query sequence and the database
sequence.
</li><li class="li-itemize">Some programs like BLAT or Exonerate may further split these regions into
several alignment fragments (or blocks in BLAT and possibly exons in
exonerate). This is not something you always see, as programs like BLAST and
HMMER do not do this.
</li></ul><p>Realizing this generality, we decided use it as base for creating the
<code>Bio.SearchIO</code> object model. The object model consists of a nested
hierarchy of Python objects, each one representing one concept outlined above.
These objects are:</p><ul class="itemize"><li class="li-itemize">
<code>QueryResult</code>, to represent a single search query.
</li><li class="li-itemize"><code>Hit</code>, to represent a single database hit. <code>Hit</code> objects are
contained within <code>QueryResult</code> and in each <code>QueryResult</code> there is
zero or more <code>Hit</code> objects.
</li><li class="li-itemize"><code>HSP</code> (short for high-scoring pair), to represent region(s) of
significant alignments between query and hit sequences. <code>HSP</code> objects
are contained within <code>Hit</code> objects and each <code>Hit</code> has one or more
<code>HSP</code> objects.
</li><li class="li-itemize"><code>HSPFragment</code>, to represent a single contiguous alignment between
query and hit sequences. <code>HSPFragment</code> objects are contained within
<code>HSP</code> objects. Most sequence search tools like BLAST and HMMER unify
<code>HSP</code> and <code>HSPFragment</code> objects as each <code>HSP</code> will only have
a single <code>HSPFragment</code>. However there are tools like BLAT and Exonerate
that produce <code>HSP</code> containing multiple <code>HSPFragment</code>. Don&#X2019;t worry
if this seems a tad confusing now, we&#X2019;ll elaborate more on these two objects
later on.
</li></ul><p>These four objects are the ones you will interact with when you use
<code>Bio.SearchIO</code>. They are created using one of the main <code>Bio.SearchIO</code>
methods: <code>read</code>, <code>parse</code>, <code>index</code>, or <code>index_db</code>. The
details of these methods are provided in later sections. For this section, we&#X2019;ll
only be using read and parse. These functions behave similarly to their
<code>Bio.SeqIO</code> and <code>Bio.AlignIO</code> counterparts:</p><ul class="itemize"><li class="li-itemize">
<code>read</code> is used for search output files with a single query and
returns a <code>QueryResult</code> object
</li><li class="li-itemize"><code>parse</code> is used for search output files with multiple queries and
returns a generator that yields <code>QueryResult</code> objects
</li></ul><p>With that settled, let&#X2019;s start probing each <code>Bio.SearchIO</code> object,
beginning with <code>QueryResult</code>.</p>
<!--TOC subsection id="sec110" QueryResult-->
<h3 id="sec110" class="subsection">8.1.1&#XA0;&#XA0;QueryResult</h3><!--SEC END --><p>
<a id="sec:searchio-qresult"></a></p><p>The QueryResult object represents a single search query and contains zero or
more Hit objects. Let&#X2019;s see what it looks like using the BLAST file we have:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SearchIO
&gt;&gt;&gt; blast_qresult = SearchIO.read('my_blast.xml', 'blast-xml')
&gt;&gt;&gt; print(blast_qresult)
Program: blastn (2.2.27+)
  Query: 42291 (61)
         mystery_seq
 Target: refseq_rna
   Hits: ----  -----  ----------------------------------------------------------
            #  # HSP  ID + description                                          
         ----  -----  ----------------------------------------------------------
            0      1  gi|262205317|ref|NR_030195.1|  Homo sapiens microRNA 52...
            1      1  gi|301171311|ref|NR_035856.1|  Pan troglodytes microRNA...
            2      1  gi|270133242|ref|NR_032573.1|  Macaca mulatta microRNA ...
            3      2  gi|301171322|ref|NR_035857.1|  Pan troglodytes microRNA...
            4      1  gi|301171267|ref|NR_035851.1|  Pan troglodytes microRNA...
            5      2  gi|262205330|ref|NR_030198.1|  Homo sapiens microRNA 52...
            6      1  gi|262205302|ref|NR_030191.1|  Homo sapiens microRNA 51...
            7      1  gi|301171259|ref|NR_035850.1|  Pan troglodytes microRNA...
            8      1  gi|262205451|ref|NR_030222.1|  Homo sapiens microRNA 51...
            9      2  gi|301171447|ref|NR_035871.1|  Pan troglodytes microRNA...
           10      1  gi|301171276|ref|NR_035852.1|  Pan troglodytes microRNA...
           11      1  gi|262205290|ref|NR_030188.1|  Homo sapiens microRNA 51...
           12      1  gi|301171354|ref|NR_035860.1|  Pan troglodytes microRNA...
           13      1  gi|262205281|ref|NR_030186.1|  Homo sapiens microRNA 52...
           14      2  gi|262205298|ref|NR_030190.1|  Homo sapiens microRNA 52...
           15      1  gi|301171394|ref|NR_035865.1|  Pan troglodytes microRNA...
           16      1  gi|262205429|ref|NR_030218.1|  Homo sapiens microRNA 51...
           17      1  gi|262205423|ref|NR_030217.1|  Homo sapiens microRNA 52...
           18      1  gi|301171401|ref|NR_035866.1|  Pan troglodytes microRNA...
           19      1  gi|270133247|ref|NR_032574.1|  Macaca mulatta microRNA ...
           20      1  gi|262205309|ref|NR_030193.1|  Homo sapiens microRNA 52...
           21      2  gi|270132717|ref|NR_032716.1|  Macaca mulatta microRNA ...
           22      2  gi|301171437|ref|NR_035870.1|  Pan troglodytes microRNA...
           23      2  gi|270133306|ref|NR_032587.1|  Macaca mulatta microRNA ...
           24      2  gi|301171428|ref|NR_035869.1|  Pan troglodytes microRNA...
           25      1  gi|301171211|ref|NR_035845.1|  Pan troglodytes microRNA...
           26      2  gi|301171153|ref|NR_035838.1|  Pan troglodytes microRNA...
           27      2  gi|301171146|ref|NR_035837.1|  Pan troglodytes microRNA...
           28      2  gi|270133254|ref|NR_032575.1|  Macaca mulatta microRNA ...
           29      2  gi|262205445|ref|NR_030221.1|  Homo sapiens microRNA 51...
           ~~~
           97      1  gi|356517317|ref|XM_003527287.1|  PREDICTED: Glycine ma...
           98      1  gi|297814701|ref|XM_002875188.1|  Arabidopsis lyrata su...
           99      1  gi|397513516|ref|XM_003827011.1|  PREDICTED: Pan panisc...
</pre><p>We&#X2019;ve just begun to scratch the surface of the object model, but you can see that
there&#X2019;s already some useful information. By invoking <code>print</code> on the
<code>QueryResult</code> object, you can see:</p><ul class="itemize"><li class="li-itemize">
The program name and version (blastn version 2.2.27+)
</li><li class="li-itemize">The query ID, description, and its sequence length (ID is 42291,
description is &#X2018;mystery_seq&#X2019;, and it is 61 nucleotides long)
</li><li class="li-itemize">The target database to search against (refseq_rna)
</li><li class="li-itemize">A quick overview of the resulting hits. For our query sequence, there are
100 potential hits (numbered 0&#X2013;99 in the table). For each hit, we can also see
how many HSPs it contains, its ID, and a snippet of its description. Notice
here that <code>Bio.SearchIO</code> truncates the hit table overview, by showing
only hits numbered 0&#X2013;29, and then 97&#X2013;99.
</li></ul><p>Now let&#X2019;s check our BLAT results using the same procedure as above:</p><pre class="verbatim">&gt;&gt;&gt; blat_qresult = SearchIO.read('my_blat.psl', 'blat-psl')
&gt;&gt;&gt; print(blat_qresult)
Program: blat (&lt;unknown version&gt;)
  Query: mystery_seq (61)
         &lt;unknown description&gt;
 Target: &lt;unknown target&gt;
   Hits: ----  -----  ----------------------------------------------------------
            #  # HSP  ID + description                                          
         ----  -----  ----------------------------------------------------------
            0     17  chr19  &lt;unknown description&gt;                              
</pre><p>You&#X2019;ll immediately notice that there are some differences. Some of these are
caused by the way PSL format stores its details, as you&#X2019;ll see. The rest are
caused by the genuine program and target database differences between our BLAST
and BLAT searches:</p><ul class="itemize"><li class="li-itemize">
The program name and version. <code>Bio.SearchIO</code> knows that the program
is BLAT, but in the output file there is no information regarding the
program version so it defaults to &#X2018;&lt;unknown version&gt;&#X2019;.
</li><li class="li-itemize">The query ID, description, and its sequence length. Notice here that these
details are slightly different from the ones we saw in BLAST. The ID is
&#X2018;mystery_seq&#X2019; instead of 42991, there is no known description, but the query
length is still 61. This is actually a difference introduced by the file
formats themselves. BLAST sometimes creates its own query IDs and uses your
original ID as the sequence description.
</li><li class="li-itemize">The target database is not known, as it is not stated in the BLAT output
file.
</li><li class="li-itemize">And finally, the list of hits we have is completely different. Here, we
see that our query sequence only hits the &#X2018;chr19&#X2019; database entry, but in it
we see 17 HSP regions. This should not be surprising however, given that we
are using a different program, each with its own target database.
</li></ul><p>All the details you saw when invoking the <code>print</code> method can be accessed
individually using Python&#X2019;s object attribute access notation (a.k.a. the dot
notation). There are also other format-specific attributes that you can access
using the same method.</p><pre class="verbatim">&gt;&gt;&gt; print("%s %s" % (blast_qresult.program, blast_qresult.version))
blastn 2.2.27+
&gt;&gt;&gt; print("%s %s" % (blat_qresult.program, blat_qresult.version))
blat &lt;unknown version&gt;
&gt;&gt;&gt; blast_qresult.param_evalue_threshold    # blast-xml specific
10.0
</pre><p>For a complete list of accessible attributes, you can check each format-specific
documentation. Here are the ones
<a href="http://biopython.org/DIST/docs/api/Bio.SearchIO.BlastIO-module.html">for BLAST</a>
and for
<a href="http://biopython.org/DIST/docs/api/Bio.SearchIO.BlatIO-module.html">BLAT</a>.</p><p>Having looked at using <code>print</code> on <code>QueryResult</code> objects, let&#X2019;s drill
down deeper. What exactly is a <code>QueryResult</code>? In terms of Python objects,
<code>QueryResult</code> is a hybrid between a list and a dictionary. In other words,
it is a container object with all the convenient features of lists and
dictionaries.</p><p>Like Python lists and dictionaries, <code>QueryResult</code> objects are iterable.
Each iteration returns a <code>Hit</code> object:</p><pre class="verbatim">&gt;&gt;&gt; for hit in blast_qresult:
...     hit
Hit(id='gi|262205317|ref|NR_030195.1|', query_id='42291', 1 hsps)
Hit(id='gi|301171311|ref|NR_035856.1|', query_id='42291', 1 hsps)
Hit(id='gi|270133242|ref|NR_032573.1|', query_id='42291', 1 hsps)
Hit(id='gi|301171322|ref|NR_035857.1|', query_id='42291', 2 hsps)
Hit(id='gi|301171267|ref|NR_035851.1|', query_id='42291', 1 hsps)
...
</pre><p>To check how many items (hits) a <code>QueryResult</code> has, you can simply invoke
Python&#X2019;s <code>len</code> method:</p><pre class="verbatim">&gt;&gt;&gt; len(blast_qresult)
100
&gt;&gt;&gt; len(blat_qresult)
1
</pre><p>Like Python lists, you can retrieve items (hits) from a <code>QueryResult</code> using
the slice notation:</p><pre class="verbatim">&gt;&gt;&gt; blast_qresult[0]        # retrieves the top hit
Hit(id='gi|262205317|ref|NR_030195.1|', query_id='42291', 1 hsps)
&gt;&gt;&gt; blast_qresult[-1]       # retrieves the last hit
Hit(id='gi|397513516|ref|XM_003827011.1|', query_id='42291', 1 hsps)
</pre><p>To retrieve multiple hits, you can slice <code>QueryResult</code> objects using the
slice notation as well. In this case, the slice will return a new
<code>QueryResult</code> object containing only the sliced hits:</p><pre class="verbatim">&gt;&gt;&gt; blast_slice = blast_qresult[:3]     # slices the first three hits
&gt;&gt;&gt; print(blast_slice)
Program: blastn (2.2.27+)
  Query: 42291 (61)
         mystery_seq
 Target: refseq_rna
   Hits: ----  -----  ----------------------------------------------------------
            #  # HSP  ID + description                                          
         ----  -----  ----------------------------------------------------------
            0      1  gi|262205317|ref|NR_030195.1|  Homo sapiens microRNA 52...
            1      1  gi|301171311|ref|NR_035856.1|  Pan troglodytes microRNA...
            2      1  gi|270133242|ref|NR_032573.1|  Macaca mulatta microRNA ...
</pre><p>Like Python dictionaries, you can also retrieve hits using the hit&#X2019;s ID. This is
particularly useful if you know a given hit ID exists within a search query
results:</p><pre class="verbatim">&gt;&gt;&gt; blast_qresult['gi|262205317|ref|NR_030195.1|']
Hit(id='gi|262205317|ref|NR_030195.1|', query_id='42291', 1 hsps)
</pre><p>You can also get a full list of <code>Hit</code> objects using <code>hits</code> and a full
list of <code>Hit</code> IDs using <code>hit_keys</code>:</p><pre class="verbatim">&gt;&gt;&gt; blast_qresult.hits
[...]       # list of all hits
&gt;&gt;&gt; blast_qresult.hit_keys
[...]       # list of all hit IDs
</pre><p>What if you just want to check whether a particular hit is present in the query
results? You can do a simple Python membership test using the <code>in</code> keyword:</p><pre class="verbatim">&gt;&gt;&gt; 'gi|262205317|ref|NR_030195.1|' in blast_qresult
True
&gt;&gt;&gt; 'gi|262205317|ref|NR_030194.1|' in blast_qresult
False
</pre><p>Sometimes, knowing whether a hit is present is not enough; you also want to know
the rank of the hit. Here, the <code>index</code> method comes to the rescue:</p><pre class="verbatim">&gt;&gt;&gt; blast_qresult.index('gi|301171437|ref|NR_035870.1|')
22
</pre><p>Remember that we&#X2019;re using Python&#X2019;s indexing style here, which is zero-based.
This means our hit above is ranked at no. 23, not 22.</p><p>Also, note that the hit rank you see here is based on the native hit ordering
present in the original search output file. Different search tools may order
these hits based on different criteria.</p><p>If the native hit ordering doesn&#X2019;t suit your taste, you can use the <code>sort</code>
method of the <code>QueryResult</code> object. It is very similar to Python&#X2019;s
<code>list.sort</code> method, with the addition of an option to create a new sorted
<code>QueryResult</code> object or not.</p><p>Here is an example of using <code>QueryResult.sort</code> to sort the hits based on
each hit&#X2019;s full sequence length. For this particular sort, we&#X2019;ll set the
<code>in_place</code> flag to <code>False</code> so that sorting will return a new
<code>QueryResult</code> object and leave our initial object unsorted. We&#X2019;ll also set
the <code>reverse</code> flag to <code>True</code> so that we sort in descending order.</p><pre class="verbatim">&gt;&gt;&gt; for hit in blast_qresult[:5]:   # id and sequence length of the first five hits
...     print("%s %i" % (hit.id, hit.seq_len))
... 
gi|262205317|ref|NR_030195.1| 61
gi|301171311|ref|NR_035856.1| 60
gi|270133242|ref|NR_032573.1| 85
gi|301171322|ref|NR_035857.1| 86
gi|301171267|ref|NR_035851.1| 80

&gt;&gt;&gt; sort_key = lambda hit: hit.seq_len
&gt;&gt;&gt; sorted_qresult = blast_qresult.sort(key=sort_key, reverse=True, in_place=False)
&gt;&gt;&gt; for hit in sorted_qresult[:5]:
...     print("%s %i" % (hit.id, hit.seq_len))
... 
gi|397513516|ref|XM_003827011.1| 6002
gi|390332045|ref|XM_776818.2| 4082
gi|390332043|ref|XM_003723358.1| 4079
gi|356517317|ref|XM_003527287.1| 3251
gi|356543101|ref|XM_003539954.1| 2936
</pre><p>The advantage of having the <code>in_place</code> flag here is that we&#X2019;re preserving
the native ordering, so we may use it again later. You should note that this is
not the default behavior of <code>QueryResult.sort</code>, however, which is why we
needed to set the <code>in_place</code> flag to <code>True</code> explicitly.</p><p>At this point, you&#X2019;ve known enough about <code>QueryResult</code> objects to make it
work for you. But before we go on to the next object in the <code>Bio.SearchIO</code>
model, let&#X2019;s take a look at two more sets of methods that could make it even
easier to work with <code>QueryResult</code> objects: the <code>filter</code> and <code>map</code>
methods.</p><p>If you&#X2019;re familiar with Python&#X2019;s list comprehensions, generator expressions
or the built in <code>filter</code> and <code>map</code> functions,
you&#X2019;ll know how useful they are for working with list-like objects (if you&#X2019;re
not, check them out!). You can use these built in methods to manipulate
<code>QueryResult</code> objects, but you&#X2019;ll end up with regular Python lists and lose
the ability to do more interesting manipulations.</p><p>That&#X2019;s why, <code>QueryResult</code> objects provide its own flavor of
<code>filter</code> and <code>map</code> methods. Analogous to <code>filter</code>, there are
<code>hit_filter</code> and <code>hsp_filter</code> methods. As their name implies, these
methods filter its <code>QueryResult</code> object either on its <code>Hit</code> objects
or <code>HSP</code> objects. Similarly, analogous to <code>map</code>, <code>QueryResult</code>
objects also provide the <code>hit_map</code> and <code>hsp_map</code> methods. These
methods apply a given function to all hits or HSPs in a <code>QueryResult</code>
object, respectively.</p><p>Let&#X2019;s see these methods in action, beginning with <code>hit_filter</code>. This method
accepts a callback function that checks whether a given <code>Hit</code> object passes
the condition you set or not. In other words, the function must accept as its
argument a single <code>Hit</code> object and returns <code>True</code> or <code>False</code>.</p><p>Here is an example of using <code>hit_filter</code> to filter out <code>Hit</code> objects
that only have one HSP:</p><pre class="verbatim">&gt;&gt;&gt; filter_func = lambda hit: len(hit.hsps) &gt; 1     # the callback function
&gt;&gt;&gt; len(blast_qresult)      # no. of hits before filtering
100
&gt;&gt;&gt; filtered_qresult = blast_qresult.hit_filter(filter_func)
&gt;&gt;&gt; len(filtered_qresult)   # no. of hits after filtering
37
&gt;&gt;&gt; for hit in filtered_qresult[:5]:    # quick check for the hit lengths
...     print("%s %i" % (hit.id, len(hit.hsps)))
gi|301171322|ref|NR_035857.1| 2
gi|262205330|ref|NR_030198.1| 2
gi|301171447|ref|NR_035871.1| 2
gi|262205298|ref|NR_030190.1| 2
gi|270132717|ref|NR_032716.1| 2
</pre><p><code>hsp_filter</code> works the same as <code>hit_filter</code>, only instead of looking
at the <code>Hit</code> objects, it performs filtering on the <code>HSP</code> objects in
each hits.</p><p>As for the <code>map</code> methods, they too accept a callback function as their
arguments. However, instead of returning <code>True</code> or <code>False</code>, the
callback function must return the modified <code>Hit</code> or <code>HSP</code> object
(depending on whether you&#X2019;re using <code>hit_map</code> or <code>hsp_map</code>).</p><p>Let&#X2019;s see an example where we&#X2019;re using <code>hit_map</code> to rename the hit IDs:</p><pre class="verbatim">&gt;&gt;&gt; def map_func(hit):
...     hit.id = hit.id.split('|')[3]   # renames 'gi|301171322|ref|NR_035857.1|' to 'NR_035857.1'
...     return hit
...
&gt;&gt;&gt; mapped_qresult = blast_qresult.hit_map(map_func)
&gt;&gt;&gt; for hit in mapped_qresult[:5]:
...     print(hit.id)
NR_030195.1
NR_035856.1
NR_032573.1
NR_035857.1
NR_035851.1
</pre><p>Again, <code>hsp_map</code> works the same as <code>hit_map</code>, but on <code>HSP</code>
objects instead of <code>Hit</code> objects.</p>
<!--TOC subsection id="sec111" Hit-->
<h3 id="sec111" class="subsection">8.1.2&#XA0;&#XA0;Hit</h3><!--SEC END --><p>
<a id="sec:searchio-hit"></a></p><p><code>Hit</code> objects represent all query results from a single database entry.
They are the second-level container in the <code>Bio.SearchIO</code> object hierarchy.
You&#X2019;ve seen that they are contained by <code>QueryResult</code> objects, but they
themselves contain <code>HSP</code> objects.</p><p>Let&#X2019;s see what they look like, beginning with our BLAST search:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SearchIO
&gt;&gt;&gt; blast_qresult = SearchIO.read('my_blast.xml', 'blast-xml')
&gt;&gt;&gt; blast_hit = blast_qresult[3]    # fourth hit from the query result
&gt;&gt;&gt; print(blast_hit)
Query: 42291
       mystery_seq
  Hit: gi|301171322|ref|NR_035857.1| (86)
       Pan troglodytes microRNA mir-520c (MIR520C), microRNA
 HSPs: ----  --------  ---------  ------  ---------------  ---------------------
          #   E-value  Bit score    Span      Query range              Hit range
       ----  --------  ---------  ------  ---------------  ---------------------
          0   8.9e-20     100.47      60           [1:61]                [13:73]
          1   3.3e-06      55.39      60           [0:60]                [13:73]
</pre><p>You see that we&#X2019;ve got the essentials covered here:</p><ul class="itemize"><li class="li-itemize">
The query ID and description is present. A hit is always tied to a query,
so we want to keep track of the originating query as well. These values can
be accessed from a hit using the <code>query_id</code> and
<code>query_description</code> attributes.
</li><li class="li-itemize">We also have the unique hit ID, description, and full sequence lengths.
They can be accessed using <code>id</code>, <code>description</code>, and
<code>seq_len</code>, respectively.
</li><li class="li-itemize">Finally, there&#X2019;s a table containing quick information about the HSPs this
hit contains. In each row, we&#X2019;ve got the important HSP details listed: the
HSP index, its e-value, its bit score, its span (the alignment length
including gaps), its query coordinates, and its hit coordinates.
</li></ul><p>Now let&#X2019;s contrast this with the BLAT search. Remember that in the BLAT search we
had one hit with 17 HSPs.</p><pre class="verbatim">&gt;&gt;&gt; blat_qresult = SearchIO.read('my_blat.psl', 'blat-psl')
&gt;&gt;&gt; blat_hit = blat_qresult[0]      # the only hit
&gt;&gt;&gt; print(blat_hit)
Query: mystery_seq
       &lt;unknown description&gt;
  Hit: chr19 (59128983)
       &lt;unknown description&gt;
 HSPs: ----  --------  ---------  ------  ---------------  ---------------------
          #   E-value  Bit score    Span      Query range              Hit range
       ----  --------  ---------  ------  ---------------  ---------------------
          0         ?          ?       ?           [0:61]    [54204480:54204541]
          1         ?          ?       ?           [0:61]    [54233104:54264463]
          2         ?          ?       ?           [0:61]    [54254477:54260071]
          3         ?          ?       ?           [1:61]    [54210720:54210780]
          4         ?          ?       ?           [0:60]    [54198476:54198536]
          5         ?          ?       ?           [0:61]    [54265610:54265671]
          6         ?          ?       ?           [0:61]    [54238143:54240175]
          7         ?          ?       ?           [0:60]    [54189735:54189795]
          8         ?          ?       ?           [0:61]    [54185425:54185486]
          9         ?          ?       ?           [0:60]    [54197657:54197717]
         10         ?          ?       ?           [0:61]    [54255662:54255723]
         11         ?          ?       ?           [0:61]    [54201651:54201712]
         12         ?          ?       ?           [8:60]    [54206009:54206061]
         13         ?          ?       ?          [10:61]    [54178987:54179038]
         14         ?          ?       ?           [8:61]    [54212018:54212071]
         15         ?          ?       ?           [8:51]    [54234278:54234321]
         16         ?          ?       ?           [8:61]    [54238143:54238196]
</pre><p>Here, we&#X2019;ve got a similar level of detail as with the BLAST hit we saw earlier.
There are some differences worth explaining, though:</p><ul class="itemize"><li class="li-itemize">
The e-value and bit score column values. As BLAT HSPs do not have e-values
and bit scores, the display defaults to &#X2018;?&#X2019;.
</li><li class="li-itemize">What about the span column? The span values is meant to display the
complete alignment length, which consists of all residues and any gaps that
may be present. The PSL format do not have this information readily available
and <code>Bio.SearchIO</code> does not attempt to try guess what it is, so we get a
&#X2018;?&#X2019; similar to the e-value and bit score columns.
</li></ul><p>In terms of Python objects, <code>Hit</code> behaves almost the same as Python lists,
but contain <code>HSP</code> objects exclusively. If you&#X2019;re familiar with lists, you
should encounter no difficulties working with the <code>Hit</code> object. </p><p>Just like Python lists, <code>Hit</code> objects are iterable, and each iteration
returns one <code>HSP</code> object it contains:</p><pre class="verbatim">&gt;&gt;&gt; for hsp in blast_hit:
...     hsp
HSP(hit_id='gi|301171322|ref|NR_035857.1|', query_id='42291', 1 fragments)
HSP(hit_id='gi|301171322|ref|NR_035857.1|', query_id='42291', 1 fragments)
</pre><p>You can invoke <code>len</code> on a <code>Hit</code> to see how many <code>HSP</code> objects it
has:</p><pre class="verbatim">&gt;&gt;&gt; len(blast_hit)
2
&gt;&gt;&gt; len(blat_hit)
17
</pre><p>You can use the slice notation on <code>Hit</code> objects, whether to retrieve single
<code>HSP</code> or multiple <code>HSP</code> objects. Like <code>QueryResult</code>, if you slice
for multiple <code>HSP</code>, a new <code>Hit</code> object will be returned containing
only the sliced <code>HSP</code> objects:</p><pre class="verbatim">&gt;&gt;&gt; blat_hit[0]                 # retrieve single items
HSP(hit_id='chr19', query_id='mystery_seq', 1 fragments)
&gt;&gt;&gt; sliced_hit = blat_hit[4:9]  # retrieve multiple items
&gt;&gt;&gt; len(sliced_hit)
5
&gt;&gt;&gt; print(sliced_hit)
Query: mystery_seq
       &lt;unknown description&gt;
  Hit: chr19 (59128983)
       &lt;unknown description&gt;
 HSPs: ----  --------  ---------  ------  ---------------  ---------------------
          #   E-value  Bit score    Span      Query range              Hit range
       ----  --------  ---------  ------  ---------------  ---------------------
          0         ?          ?       ?           [0:60]    [54198476:54198536]
          1         ?          ?       ?           [0:61]    [54265610:54265671]
          2         ?          ?       ?           [0:61]    [54238143:54240175]
          3         ?          ?       ?           [0:60]    [54189735:54189795]
          4         ?          ?       ?           [0:61]    [54185425:54185486]
</pre><p>You can also sort the <code>HSP</code> inside a <code>Hit</code>, using the exact same
arguments like the sort method you saw in the <code>QueryResult</code> object.</p><p>Finally, there are also the <code>filter</code> and <code>map</code> methods you can use
on <code>Hit</code> objects. Unlike in the <code>QueryResult</code> object, <code>Hit</code>
objects only have one variant of <code>filter</code> (<code>Hit.filter</code>) and one
variant of <code>map</code> (<code>Hit.map</code>). Both of <code>Hit.filter</code> and
<code>Hit.map</code> work on the <code>HSP</code> objects a <code>Hit</code> has.</p>
<!--TOC subsection id="sec112" HSP-->
<h3 id="sec112" class="subsection">8.1.3&#XA0;&#XA0;HSP</h3><!--SEC END --><p>
<a id="sec:searchio-hsp"></a></p><p><code>HSP</code> (high-scoring pair) represents region(s) in the hit sequence that
contains significant alignment(s) to the query sequence. It contains the actual
match between your query sequence and a database entry. As this match is
determined by the sequence search tool&#X2019;s algorithms, the <code>HSP</code> object
contains the bulk of the statistics computed by the search tool. This also makes
the distinction between <code>HSP</code> objects from different search tools more
apparent compared to the differences you&#X2019;ve seen in <code>QueryResult</code> or
<code>Hit</code> objects.</p><p>Let&#X2019;s see some examples from our BLAST and BLAT searches. We&#X2019;ll look at the
BLAST HSP first:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SearchIO
&gt;&gt;&gt; blast_qresult = SearchIO.read('my_blast.xml', 'blast-xml')
&gt;&gt;&gt; blast_hsp = blast_qresult[0][0]    # first hit, first hsp
&gt;&gt;&gt; print(blast_hsp)
      Query: 42291 mystery_seq
        Hit: gi|262205317|ref|NR_030195.1| Homo sapiens microRNA 520b (MIR520...
Query range: [0:61] (1)
  Hit range: [0:61] (1)
Quick stats: evalue 4.9e-23; bitscore 111.29
  Fragments: 1 (61 columns)
     Query - CCCTCTACAGGGAAGCGCTTTCTGTTGTCTGAAAGAAAAGAAAGTGCTTCCTTTTAGAGGG
             |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
       Hit - CCCTCTACAGGGAAGCGCTTTCTGTTGTCTGAAAGAAAAGAAAGTGCTTCCTTTTAGAGGG
</pre><p>Just like <code>QueryResult</code> and <code>Hit</code>, invoking <code>print</code> on an
<code>HSP</code> shows its general details:
</p><ul class="itemize"><li class="li-itemize">
There are the query and hit IDs and descriptions. We need these to
identify our <code>HSP</code>.
</li><li class="li-itemize">We&#X2019;ve also got the matching range of the query and hit sequences. The
slice notation we&#X2019;re using here is an indication that the range is displayed
using Python&#X2019;s indexing style (zero-based, half open). The number inside the
parenthesis denotes the strand. In this case, both sequences have the plus
strand.
</li><li class="li-itemize">Some quick statistics are available: the e-value and bitscore.
</li><li class="li-itemize">There is information about the HSP fragments. Ignore this for now; it will
be explained later on.
</li><li class="li-itemize">And finally, we have the query and hit sequence alignment itself.
</li></ul><p>These details can be accessed on their own using the dot notation, just like in
<code>QueryResult</code> and <code>Hit</code>:</p><pre class="verbatim">&gt;&gt;&gt; blast_hsp.query_range
(0, 61)
</pre><pre class="verbatim">&gt;&gt;&gt; blast_hsp.evalue
4.91307e-23
</pre><p>They&#X2019;re not the only attributes available, though. <code>HSP</code> objects come with
a default set of properties that makes it easy to probe their various
details. Here are some examples:</p><pre class="verbatim">&gt;&gt;&gt; blast_hsp.hit_start         # start coordinate of the hit sequence
0
&gt;&gt;&gt; blast_hsp.query_span        # how many residues in the query sequence
61
&gt;&gt;&gt; blast_hsp.aln_span          # how long the alignment is
61
</pre><p>Check out the <code>HSP</code>
<a href="http://biopython.org/DIST/docs/api/Bio.SearchIO._model.hsp-module.html">documentation</a>
for a full list of these predefined properties.</p><p>Furthermore, each sequence search tool usually computes its own statistics /
details for its <code>HSP</code> objects. For example, an XML BLAST search also
outputs the number of gaps and identical residues. These attributes can be
accessed like so:</p><pre class="verbatim">&gt;&gt;&gt; blast_hsp.gap_num       # number of gaps
0
&gt;&gt;&gt; blast_hsp.ident_num     # number of identical residues
61
</pre><p>These details are format-specific; they may not be present in other formats.
To see which details are available for a given sequence search tool, you
should check the format&#X2019;s documentation in <code>Bio.SearchIO</code>. Alternatively,
you may also use <code>.__dict__.keys()</code> for a quick list of what&#X2019;s available:</p><pre class="verbatim">&gt;&gt;&gt; blast_hsp.__dict__.keys()
['bitscore', 'evalue', 'ident_num', 'gap_num', 'bitscore_raw', 'pos_num', '_items']
</pre><p>Finally, you may have noticed that the <code>query</code> and <code>hit</code> attributes
of our HSP are not just regular strings:</p><pre class="verbatim">&gt;&gt;&gt; blast_hsp.query
SeqRecord(seq=Seq('CCCTCTACAGGGAAGCGCTTTCTGTTGTCTGAAAGAAAAGAAAGTGCTTCCTTT...GGG', DNAAlphabet()), id='42291', name='aligned query sequence', description='mystery_seq', dbxrefs=[])
&gt;&gt;&gt; blast_hsp.hit
SeqRecord(seq=Seq('CCCTCTACAGGGAAGCGCTTTCTGTTGTCTGAAAGAAAAGAAAGTGCTTCCTTT...GGG', DNAAlphabet()), id='gi|262205317|ref|NR_030195.1|', name='aligned hit sequence', description='Homo sapiens microRNA 520b (MIR520B), microRNA', dbxrefs=[])
</pre><p>They are <code>SeqRecord</code> objects you saw earlier in
Section&#XA0;<a href="#chapter%3ASeqRecord">4</a>! This means that you can do all sorts of
interesting things you can do with <code>SeqRecord</code> objects on <code>HSP.query</code>
and/or <code>HSP.hit</code>. </p><p>It should not surprise you now that the <code>HSP</code> object has an
<code>alignment</code> property which is a <code>MultipleSeqAlignment</code> object:</p><pre class="verbatim">&gt;&gt;&gt; print(blast_hsp.aln)
DNAAlphabet() alignment with 2 rows and 61 columns
CCCTCTACAGGGAAGCGCTTTCTGTTGTCTGAAAGAAAAGAAAG...GGG 42291
CCCTCTACAGGGAAGCGCTTTCTGTTGTCTGAAAGAAAAGAAAG...GGG gi|262205317|ref|NR_030195.1|
</pre><p>Having probed the BLAST HSP, let&#X2019;s now take a look at HSPs from our BLAT
results for a different kind of HSP. As usual, we&#X2019;ll begin by invoking
<code>print</code> on it:</p><pre class="verbatim">&gt;&gt;&gt; blat_qresult = SearchIO.read('my_blat.psl', 'blat-psl')
&gt;&gt;&gt; blat_hsp = blat_qresult[0][0]       # first hit, first hsp
&gt;&gt;&gt; print(blat_hsp)
      Query: mystery_seq &lt;unknown description&gt;
        Hit: chr19 &lt;unknown description&gt;
Query range: [0:61] (1)
  Hit range: [54204480:54204541] (1)
Quick stats: evalue ?; bitscore ?
  Fragments: 1 (? columns)
</pre><p>Some of the outputs you may have already guessed. We have the query and hit IDs
and descriptions and the sequence coordinates. Values for evalue and bitscore is
&#X2018;?&#X2019; as BLAT HSPs do not have these attributes. But The biggest difference here
is that you don&#X2019;t see any sequence alignments displayed. If you look closer, PSL
formats themselves do not have any hit or query sequences, so
<code>Bio.SearchIO</code> won&#X2019;t create any sequence or alignment objects. What happens
if you try to access <code>HSP.query</code>, <code>HSP.hit</code>, or <code>HSP.aln</code>?
You&#X2019;ll get the default values for these attributes, which is <code>None</code>:</p><pre class="verbatim">&gt;&gt;&gt; blat_hsp.hit is None
True
&gt;&gt;&gt; blat_hsp.query is None
True
&gt;&gt;&gt; blat_hsp.aln is None
True
</pre><p>This does not affect other attributes, though. For example, you can still
access the length of the query or hit alignment. Despite not displaying any
attributes, the PSL format still have this information so <code>Bio.SearchIO</code>
can extract them:</p><pre class="verbatim">&gt;&gt;&gt; blat_hsp.query_span     # length of query match
61
&gt;&gt;&gt; blat_hsp.hit_span       # length of hit match
61
</pre><p>Other format-specific attributes are still present as well:</p><pre class="verbatim">&gt;&gt;&gt; blat_hsp.score          # PSL score
61
&gt;&gt;&gt; blat_hsp.mismatch_num   # the mismatch column
0
</pre><p>So far so good? Things get more interesting when you look at another &#X2018;variant&#X2019;
of HSP present in our BLAT results. You might recall that in BLAT searches,
sometimes we get our results separated into &#X2018;blocks&#X2019;. These blocks are
essentially alignment fragments that may have some intervening sequence between
them.</p><p>Let&#X2019;s take a look at a BLAT HSP that contains multiple blocks to see how
<code>Bio.SearchIO</code> deals with this:</p><pre class="verbatim">&gt;&gt;&gt; blat_hsp2 = blat_qresult[0][1]      # first hit, second hsp
&gt;&gt;&gt; print(blat_hsp2)
      Query: mystery_seq &lt;unknown description&gt;
        Hit: chr19 &lt;unknown description&gt;
Query range: [0:61] (1)
  Hit range: [54233104:54264463] (1)
Quick stats: evalue ?; bitscore ?
  Fragments: ---  --------------  ----------------------  ----------------------
               #            Span             Query range               Hit range
             ---  --------------  ----------------------  ----------------------
               0               ?                  [0:18]     [54233104:54233122]
               1               ?                 [18:61]     [54264420:54264463]
</pre><p>What&#X2019;s happening here? We still some essential details covered: the IDs and
descriptions, the coordinates, and the quick statistics are similar to what
you&#X2019;ve seen before. But the fragments detail is all different. Instead of
showing &#X2018;Fragments: 1&#X2019;, we now have a table with two data rows.</p><p>This is how <code>Bio.SearchIO</code> deals with HSPs having multiple fragments. As
mentioned before, an HSP alignment may be separated by intervening sequences
into fragments. The intervening sequences are not part of the query-hit match,
so they should not be considered part of query nor hit sequence. However, they
do affect how we deal with sequence coordinates, so we can&#X2019;t ignore them.</p><p>Take a look at the hit coordinate of the HSP above. In the <code>Hit range:</code> field,
we see that the coordinate is <code>[54233104:54264463]</code>. But looking at the
table rows, we see that not the entire region spanned by this coordinate matches
our query. Specifically, the intervening region spans from <code>54233122</code> to
<code>54264420</code>.</p><p>Why then, is the query coordinates seem to be contiguous, you ask? This is
perfectly fine. In this case it means that the query match is contiguous (no
intervening regions), while the hit match is not.</p><p>All these attributes are accessible from the HSP directly, by the way:</p><pre class="verbatim">&gt;&gt;&gt; blat_hsp2.hit_range         # hit start and end coordinates of the entire HSP
(54233104, 54264463)
&gt;&gt;&gt; blat_hsp2.hit_range_all     # hit start and end coordinates of each fragment
[(54233104, 54233122), (54264420, 54264463)]
&gt;&gt;&gt; blat_hsp2.hit_span          # hit span of the entire HSP
31359
&gt;&gt;&gt; blat_hsp2.hit_span_all      # hit span of each fragment
[18, 43]
&gt;&gt;&gt; blat_hsp2.hit_inter_ranges  # start and end coordinates of intervening regions in the hit sequence
[(54233122, 54264420)]
&gt;&gt;&gt; blat_hsp2.hit_inter_spans   # span of intervening regions in the hit sequence
[31298]
</pre><p>Most of these attributes are not readily available from the PSL file we have,
but <code>Bio.SearchIO</code> calculates them for you on the fly when you parse the
PSL file. All it needs are the start and end coordinates of each fragment.</p><p>What about the <code>query</code>, <code>hit</code>, and <code>aln</code> attributes? If the
HSP has multiple fragments, you won&#X2019;t be able to use these attributes as they
only fetch single <code>SeqRecord</code> or <code>MultipleSeqAlignment</code> objects.
However, you can use their <code>*_all</code> counterparts: <code>query_all</code>,
<code>hit_all</code>, and <code>aln_all</code>. These properties will return a list containing
<code>SeqRecord</code> or <code>MultipleSeqAlignment</code> objects from each of the HSP
fragment. There are other attributes that behave similarly, i.e. they only work
for HSPs with one fragment. Check out the <code>HSP</code> <a href="http://biopython.org/DIST/docs/api/Bio.SearchIO._model.hsp-module.html">documentation</a>
for a full list.</p><p>Finally, to check whether you have multiple fragments or not, you can use the
<code>is_fragmented</code> property like so:</p><pre class="verbatim">&gt;&gt;&gt; blat_hsp2.is_fragmented     # BLAT HSP with 2 fragments
True
&gt;&gt;&gt; blat_hsp.is_fragmented      # BLAT HSP from earlier, with one fragment
False
</pre><p>Before we move on, you should also know that we can use the slice notation on
<code>HSP</code> objects, just like <code>QueryResult</code> or <code>Hit</code> objects. When
you use this notation, you&#X2019;ll get an <code>HSPFragment</code> object in return, the
last component of the object model.</p>
<!--TOC subsection id="sec113" HSPFragment-->
<h3 id="sec113" class="subsection">8.1.4&#XA0;&#XA0;HSPFragment</h3><!--SEC END --><p>
<a id="sec:searchio-hspfragment"></a></p><p><code>HSPFragment</code> represents a single, contiguous match between the query and
hit sequences. You could consider it the core of the object model and search
result, since it is the presence of these fragments that determine whether your
search have results or not.</p><p>In most cases, you don&#X2019;t have to deal with <code>HSPFragment</code> objects directly
since not that many sequence search tools fragment their HSPs. When you do have
to deal with them, what you should remember is that <code>HSPFragment</code> objects
were written with to be as compact as possible. In most cases, they only contain
attributes directly related to sequences: strands, reading frames, alphabets,
coordinates, the sequences themselves, and their IDs and descriptions.</p><p>These attributes are readily shown when you invoke <code>print</code> on an
<code>HSPFragment</code>. Here&#X2019;s an example, taken from our BLAST search:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SearchIO
&gt;&gt;&gt; blast_qresult = SearchIO.read('my_blast.xml', 'blast-xml')
&gt;&gt;&gt; blast_frag = blast_qresult[0][0][0]    # first hit, first hsp, first fragment
&gt;&gt;&gt; print(blast_frag)
      Query: 42291 mystery_seq
        Hit: gi|262205317|ref|NR_030195.1| Homo sapiens microRNA 520b (MIR520...
Query range: [0:61] (1)
  Hit range: [0:61] (1)
  Fragments: 1 (61 columns)
     Query - CCCTCTACAGGGAAGCGCTTTCTGTTGTCTGAAAGAAAAGAAAGTGCTTCCTTTTAGAGGG
             |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
       Hit - CCCTCTACAGGGAAGCGCTTTCTGTTGTCTGAAAGAAAAGAAAGTGCTTCCTTTTAGAGGG
</pre><p>At this level, the BLAT fragment looks quite similar to the BLAST fragment, save
for the query and hit sequences which are not present:</p><pre class="verbatim">&gt;&gt;&gt; blat_qresult = SearchIO.read('my_blat.psl', 'blat-psl')
&gt;&gt;&gt; blat_frag = blat_qresult[0][0][0]    # first hit, first hsp, first fragment
&gt;&gt;&gt; print(blat_frag)
      Query: mystery_seq &lt;unknown description&gt;
        Hit: chr19 &lt;unknown description&gt;
Query range: [0:61] (1)
  Hit range: [54204480:54204541] (1)
  Fragments: 1 (? columns)
</pre><p>In all cases, these attributes are accessible using our favorite dot notation.
Some examples:</p><pre class="verbatim">&gt;&gt;&gt; blast_frag.query_start      # query start coordinate
0
&gt;&gt;&gt; blast_frag.hit_strand       # hit sequence strand
1
&gt;&gt;&gt; blast_frag.hit              # hit sequence, as a SeqRecord object
SeqRecord(seq=Seq('CCCTCTACAGGGAAGCGCTTTCTGTTGTCTGAAAGAAAAGAAAGTGCTTCCTTT...GGG', DNAAlphabet()), id='gi|262205317|ref|NR_030195.1|', name='aligned hit sequence', description='Homo sapiens microRNA 520b (MIR520B), microRNA', dbxrefs=[])
</pre>
<!--TOC section id="sec114" A note about standards and conventions-->
<h2 id="sec114" class="section">8.2&#XA0;&#XA0;A note about standards and conventions</h2><!--SEC END --><p>
<a id="sec:searchio-standards"></a></p><p>Before we move on to the main functions, there is something you ought to know
about the standards <code>Bio.SearchIO</code> uses. If you&#X2019;ve worked with multiple
sequence search tools, you might have had to deal with the many different ways
each program deals with things like sequence coordinates. It might not have been
a pleasant experience as these search tools usually have their own standards.
For example, one tools might use one-based coordinates, while the other uses
zero-based coordinates. Or, one program might reverse the start and end
coordinates if the strand is minus, while others don&#X2019;t. In short, these often
creates unnecessary mess must be dealt with.</p><p>We realize this problem ourselves and we intend to address it in
<code>Bio.SearchIO</code>. After all, one of the goals of <code>Bio.SearchIO</code> is to
create a common, easy to use interface to deal with various search output files.
This means creating standards that extend beyond the object model you just saw.</p><p>Now, you might complain, "Not another standard!". Well, eventually we have to
choose one convention or the other, so this is necessary. Plus, we&#X2019;re not
creating something entirely new here; just adopting a standard we think is best
for a Python programmer (it is Biopython, after all).</p><p>There are three implicit standards that you can expect when working with
<code>Bio.SearchIO</code>:</p><ul class="itemize"><li class="li-itemize">
The first one pertains to sequence coordinates. In <code>Bio.SearchIO</code>,
all sequence coordinates follows Python&#X2019;s coordinate style: zero-based and
half open. For example, if in a BLAST XML output file the start and end
coordinates of an HSP are 10 and 28, they would become 9 and 28 in
<code>Bio.SearchIO</code>. The start coordinate becomes 9 because Python indices
start from zero, while the end coordinate remains 28 as Python slices omit
the last item in an interval.
</li><li class="li-itemize">The second is on sequence coordinate orders. In <code>Bio.SearchIO</code>, start
coordinates are always less than or equal to end coordinates. This isn&#X2019;t
always the case with all sequence search tools, as some of them have larger
start coordinates when the sequence strand is minus.
</li><li class="li-itemize">The last one is on strand and reading frame values. For strands, there are
only four valid choices: <code>1</code> (plus strand), <code>-1</code> (minus strand),
<code>0</code> (protein sequences), and <code>None</code> (no strand). For reading
frames, the valid choices are integers from <code>-3</code> to <code>3</code> and
<code>None</code>.
</li></ul><p>Note that these standards only exist in <code>Bio.SearchIO</code> objects. If you
write <code>Bio.SearchIO</code> objects into an output format, <code>Bio.SearchIO</code>
will use the format&#X2019;s standard for the output. It does not force its standard
over to your output file.</p>
<!--TOC section id="sec115" Reading search output files-->
<h2 id="sec115" class="section">8.3&#XA0;&#XA0;Reading search output files</h2><!--SEC END --><p>
<a id="sec:searchio-input"></a></p><p>There are two functions you can use for reading search output files into
<code>Bio.SearchIO</code> objects: <code>read</code> and <code>parse</code>. They&#X2019;re essentially
similar to <code>read</code> and <code>parse</code> functions in other submodules like
<code>Bio.SeqIO</code> or <code>Bio.AlignIO</code>. In both cases, you need to supply the
search output file name and the file format name, both as Python strings. You
can check the documentation for a list of format names <code>Bio.SearchIO</code>
recognizes.</p><p><code>Bio.SearchIO.read</code> is used for reading search output files with only one
query and returns a <code>QueryResult</code> object. You&#X2019;ve seen <code>read</code> used in
our previous examples. What you haven&#X2019;t seen is that <code>read</code> may also accept
additional keyword arguments, depending on the file format.</p><p>Here are some examples. In the first one, we use <code>read</code> just like
previously to read a BLAST tabular output file. In the second one, we use a
keyword argument to modify so it parses the BLAST tabular variant with comments
in it:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SearchIO
&gt;&gt;&gt; qresult = SearchIO.read('tab_2226_tblastn_003.txt', 'blast-tab')
&gt;&gt;&gt; qresult
QueryResult(id='gi|16080617|ref|NP_391444.1|', 3 hits)
&gt;&gt;&gt; qresult2 = SearchIO.read('tab_2226_tblastn_007.txt', 'blast-tab', comments=True)
&gt;&gt;&gt; qresult2
QueryResult(id='gi|16080617|ref|NP_391444.1|', 3 hits)
</pre><p>These keyword arguments differs among file formats. Check the format
documentation to see if it has keyword arguments that modifies its parser&#X2019;s
behavior.</p><p>As for the <code>Bio.SearchIO.parse</code>, it is used for reading search output
files with any number of queries. The function returns a generator object that
yields a <code>QueryResult</code> object in each iteration. Like
<code>Bio.SearchIO.read</code>, it also accepts format-specific keyword arguments:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SearchIO
&gt;&gt;&gt; qresults = SearchIO.parse('tab_2226_tblastn_001.txt', 'blast-tab')
&gt;&gt;&gt; for qresult in qresults:
...     print(qresult.id)
gi|16080617|ref|NP_391444.1|
gi|11464971:4-101
&gt;&gt;&gt; qresults2 = SearchIO.parse('tab_2226_tblastn_005.txt', 'blast-tab', comments=True)
&gt;&gt;&gt; for qresult in qresults2:
...     print(qresult.id)
random_s00
gi|16080617|ref|NP_391444.1|
gi|11464971:4-101
</pre>
<!--TOC section id="sec116" Dealing with large search output files with indexing-->
<h2 id="sec116" class="section">8.4&#XA0;&#XA0;Dealing with large search output files with indexing</h2><!--SEC END --><p>
<a id="sec:searchio-index"></a></p><p>Sometimes, you&#X2019;re handed a search output file containing hundreds or thousands
of queries that you need to parse. You can of course use
<code>Bio.SearchIO.parse</code> for this file, but that would be grossly inefficient
if you need to access only a few of the queries. This is because <code>parse</code>
will parse all queries it sees before it fetches your query of interest.</p><p>In this case, the ideal choice would be to index the file using
<code>Bio.SearchIO.index</code> or <code>Bio.SearchIO.index_db</code>. If the names sound
familiar, it&#X2019;s because you&#X2019;ve seen them before in Section&#XA0;<a href="#sec%3ASeqIO-index">5.4.2</a>.
These functions also behave similarly to their <code>Bio.SeqIO</code> counterparts,
with the addition of format-specific keyword arguments.</p><p>Here are some examples. You can use <code>index</code> with just the filename and
format name:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SearchIO
&gt;&gt;&gt; idx = SearchIO.index('tab_2226_tblastn_001.txt', 'blast-tab')
&gt;&gt;&gt; sorted(idx.keys())
['gi|11464971:4-101', 'gi|16080617|ref|NP_391444.1|']
&gt;&gt;&gt; idx['gi|16080617|ref|NP_391444.1|']
QueryResult(id='gi|16080617|ref|NP_391444.1|', 3 hits)
&gt;&gt;&gt; idx.close()
</pre><p>Or also with the format-specific keyword argument:</p><pre class="verbatim">&gt;&gt;&gt; idx = SearchIO.index('tab_2226_tblastn_005.txt', 'blast-tab', comments=True)
&gt;&gt;&gt; sorted(idx.keys())
['gi|11464971:4-101', 'gi|16080617|ref|NP_391444.1|', 'random_s00']
&gt;&gt;&gt; idx['gi|16080617|ref|NP_391444.1|']
QueryResult(id='gi|16080617|ref|NP_391444.1|', 3 hits)
&gt;&gt;&gt; idx.close()
</pre><p>Or with the <code>key_function</code> argument, as in <code>Bio.SeqIO</code>:</p><pre class="verbatim">&gt;&gt;&gt; key_function = lambda id: id.upper()    # capitalizes the keys
&gt;&gt;&gt; idx = SearchIO.index('tab_2226_tblastn_001.txt', 'blast-tab', key_function=key_function)
&gt;&gt;&gt; sorted(idx.keys())
['GI|11464971:4-101', 'GI|16080617|REF|NP_391444.1|']
&gt;&gt;&gt; idx['GI|16080617|REF|NP_391444.1|']
QueryResult(id='gi|16080617|ref|NP_391444.1|', 3 hits)
&gt;&gt;&gt; idx.close()
</pre><p><code>Bio.SearchIO.index_db</code> works like as <code>index</code>, only it writes the
query offsets into an SQLite database file.</p>
<!--TOC section id="sec117" Writing and converting search output files-->
<h2 id="sec117" class="section">8.5&#XA0;&#XA0;Writing and converting search output files</h2><!--SEC END --><p>
<a id="sec:searchio-write"></a></p><p>It is occasionally useful to be able to manipulate search results from an output
file and write it again to a new file. <code>Bio.SearchIO</code> provides a
<code>write</code> function that lets you do exactly this. It takes as its arguments
an iterable returning <code>QueryResult</code> objects, the output filename to write
to, the format name to write to, and optionally some format-specific keyword
arguments. It returns a four-item tuple, which denotes the number or
<code>QueryResult</code>, <code>Hit</code>, <code>HSP</code>, and <code>HSPFragment</code> objects that
were written.</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SearchIO
&gt;&gt;&gt; qresults = SearchIO.parse('mirna.xml', 'blast-xml')     # read XML file
&gt;&gt;&gt; SearchIO.write(qresults, 'results.tab', 'blast-tab')    # write to tabular file
(3, 239, 277, 277)
</pre><p>You should note different file formats require different attributes of the
<code>QueryResult</code>, <code>Hit</code>, <code>HSP</code> and <code>HSPFragment</code> objects. If
these attributes are not present, writing won&#X2019;t work. In other words, you can&#X2019;t
always write to the output format that you want. For example, if you read a
BLAST XML file, you wouldn&#X2019;t be able to write the results to a PSL file as PSL
files require attributes not calculated by BLAST (e.g. the number of repeat
matches). You can always set these attributes manually, if you really want to
write to PSL, though.</p><p>Like <code>read</code>, <code>parse</code>, <code>index</code>, and <code>index_db</code>, <code>write</code>
also accepts format-specific keyword arguments. Check out the documentation for
a complete list of formats <code>Bio.SearchIO</code> can write to and their arguments.</p><p>Finally, <code>Bio.SearchIO</code> also provides a <code>convert</code> function, which is
simply a shortcut for <code>Bio.SearchIO.parse</code> and <code>Bio.SearchIO.write</code>.
Using the convert function, our example above would be:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SearchIO
&gt;&gt;&gt; SearchIO.convert('mirna.xml', 'blast-xml', 'results.tab', 'blast-tab')
(3, 239, 277, 277)
</pre><p>As <code>convert</code> uses <code>write</code>, it is only limited to format conversions
that have all the required attributes. Here, the BLAST XML file provides all the
default values a BLAST tabular file requires, so it works just fine. However,
other format conversions are less likely to work since you need to manually
assign the required attributes first.</p>
<!--TOC chapter id="sec118" Accessing NCBI&#X2019;s Entrez databases-->
<h1 id="sec118" class="chapter">Chapter&#XA0;9&#XA0;&#XA0;Accessing NCBI&#X2019;s Entrez databases</h1><!--SEC END --><p>
<a id="chapter:entrez"></a></p><p>Entrez (<a href="http://www.ncbi.nlm.nih.gov/Entrez"><span style="font-family:monospace">http://www.ncbi.nlm.nih.gov/Entrez</span></a>) is a data retrieval system that provides users access to NCBI&#X2019;s databases such as PubMed, GenBank, GEO, and many others. You can access Entrez from a web browser to manually enter queries, or you can use Biopython&#X2019;s <code>Bio.Entrez</code> module for programmatic access to Entrez. The latter allows you for example to search PubMed or download GenBank records from within a Python script.</p><p>The <code>Bio.Entrez</code> module makes use of the Entrez Programming Utilities (also known as EUtils), consisting of eight tools that are described in detail on NCBI&#X2019;s page at <a href="http://www.ncbi.nlm.nih.gov/entrez/utils/"><span style="font-family:monospace">http://www.ncbi.nlm.nih.gov/entrez/utils/</span></a>.
Each of these tools corresponds to one Python function in the <code>Bio.Entrez</code> module, as described in the sections below. This module makes sure that the correct URL is used for the queries, and that not more than one request is made every three seconds, as required by NCBI.</p><p>The output returned by the Entrez Programming Utilities is typically in XML format. To parse such output, you have several options:
</p><ol class="enumerate" type=1><li class="li-enumerate">
Use <code>Bio.Entrez</code>&#X2019;s parser to parse the XML output into a Python object;
</li><li class="li-enumerate">Use the DOM (Document Object Model) parser in Python&#X2019;s standard library;
</li><li class="li-enumerate">Use the SAX (Simple API for XML) parser in Python&#X2019;s standard library;
</li><li class="li-enumerate">Read the XML output as raw text, and parse it by string searching and manipulation.
</li></ol><p>
For the DOM and SAX parsers, see the Python documentation. The parser in <code>Bio.Entrez</code> is discussed below.</p><p>NCBI uses DTD (Document Type Definition) files to describe the structure of the information contained in XML files. Most of the DTD files used by NCBI are included in the Biopython distribution. The <code>Bio.Entrez</code> parser makes use of the DTD files when parsing an XML file returned by NCBI Entrez.</p><p>Occasionally, you may find that the DTD file associated with a specific XML file is missing in the Biopython distribution. In particular, this may happen when NCBI updates its DTD files. If this happens, <code>Entrez.read</code> will show a warning message with the name and URL of the missing DTD file. The parser will proceed to access the missing DTD file through the internet, allowing the parsing of the XML file to continue. However, the parser is much faster if the DTD file is available locally. For this purpose, please download the DTD file from the URL in the warning message and place it in the directory <code>...site-packages/Bio/Entrez/DTDs</code>, containing the other DTD files. If you don&#X2019;t have write access to this directory, you can also place the DTD file in <code>~/.biopython/Bio/Entrez/DTDs</code>, where <code>~</code> represents your home directory. Since this directory is read before the directory <code>...site-packages/Bio/Entrez/DTDs</code>, you can also put newer versions of DTD files there if the ones in <code>...site-packages/Bio/Entrez/DTDs</code> become outdated. Alternatively, if you installed Biopython from source, you can add the DTD file to the source code&#X2019;s <code>Bio/Entrez/DTDs</code> directory, and reinstall Biopython. This will install the new DTD file in the correct location together with the other DTD files.</p><p>The Entrez Programming Utilities can also generate output in other formats, such as the Fasta or GenBank file formats for sequence databases, or the MedLine format for the literature database, discussed in Section&#XA0;<a href="#sec%3Aentrez-specialized-parsers">9.12</a>.</p>
<!--TOC section id="sec119" Entrez Guidelines-->
<h2 id="sec119" class="section">9.1&#XA0;&#XA0;Entrez Guidelines</h2><!--SEC END --><p>
<a id="sec:entrez-guidelines"></a>
Before using Biopython to access the NCBI&#X2019;s online resources (via <code>Bio.Entrez</code> or some of the other modules), please read the
<a href="http://www.ncbi.nlm.nih.gov/books/NBK25497/#chapter2.Usage_Guidelines_and_Requiremen">NCBI&#X2019;s Entrez User Requirements</a>.
If the NCBI finds you are abusing their systems, they can and will ban your access! </p><p>To paraphrase:</p><ul class="itemize"><li class="li-itemize">
For any series of more than 100 requests, do this at weekends or outside USA peak times. This is up to you to obey.
</li><li class="li-itemize">Use the <a href="http://eutils.ncbi.nlm.nih.gov"><span style="font-family:monospace">http://eutils.ncbi.nlm.nih.gov</span></a> address, not the standard NCBI Web address. Biopython uses this web address.
</li><li class="li-itemize">Make no more than three requests every seconds (relaxed from at most one request every three seconds in early 2009). This is automatically enforced by Biopython.
</li><li class="li-itemize">Use the optional email parameter so the NCBI can contact you if there is a problem. You can either explicitly set this as a parameter with each call to Entrez (e.g. include <span style="font-family:monospace">email="A.N.Other@example.com"</span> in the argument list), or you can set a global email address:
<pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"
</pre><span style="font-family:monospace">Bio.Entrez</span> will then use this email address with each call to Entrez. The <span style="font-family:monospace">example.com</span> address is a reserved domain name specifically for documentation (RFC 2606). Please DO NOT use a random email &#X2013; it&#X2019;s better not to give an email at all. The email parameter will be mandatory from June 1, 2010. In case of excessive usage, NCBI will attempt to contact a user at the e-mail address provided prior to blocking access to the E-utilities.
</li><li class="li-itemize">If you are using Biopython within some larger software suite, use the tool parameter to specify this. You can either explicitly set the tool name as a parameter with each call to Entrez (e.g. include <span style="font-family:monospace">tool="MyLocalScript"</span> in the argument list), or you can set a global tool name:
<pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.tool = "MyLocalScript"
</pre>The tool parameter will default to Biopython.
</li><li class="li-itemize">For large queries, the NCBI also recommend using their session history feature (the WebEnv session cookie string, see Section&#XA0;<a href="#sec%3Aentrez-webenv">9.15</a>). This is only slightly more complicated.
</li></ul><p>In conclusion, be sensible with your usage levels. If you plan to download lots of data, consider other options. For example, if you want easy access to all the human genes, consider fetching each chromosome by FTP as a GenBank file, and importing these into your own BioSQL database (see Section&#XA0;<a href="#sec%3ABioSQL">18.5</a>).</p>
<!--TOC section id="sec120" EInfo: Obtaining information about the Entrez databases-->
<h2 id="sec120" class="section">9.2&#XA0;&#XA0;EInfo: Obtaining information about the Entrez databases</h2><!--SEC END --><p>
<a id="sec:entrez-einfo"></a>
EInfo provides field index term counts, last update, and available links for each of NCBI&#X2019;s databases. In addition, you can use EInfo to obtain a list of all database names accessible through the Entrez utilities:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
&gt;&gt;&gt; handle = Entrez.einfo()
&gt;&gt;&gt; result = handle.read()
</pre><p>The variable <code>result</code> now contains a list of databases in XML format:
</p><pre class="verbatim">&gt;&gt;&gt; print(result)
&lt;?xml version="1.0"?&gt;
&lt;!DOCTYPE eInfoResult PUBLIC "-//NLM//DTD eInfoResult, 11 May 2002//EN"
 "http://www.ncbi.nlm.nih.gov/entrez/query/DTD/eInfo_020511.dtd"&gt;
&lt;eInfoResult&gt;
&lt;DbList&gt;
        &lt;DbName&gt;pubmed&lt;/DbName&gt;
        &lt;DbName&gt;protein&lt;/DbName&gt;
        &lt;DbName&gt;nucleotide&lt;/DbName&gt;
        &lt;DbName&gt;nuccore&lt;/DbName&gt;
        &lt;DbName&gt;nucgss&lt;/DbName&gt;
        &lt;DbName&gt;nucest&lt;/DbName&gt;
        &lt;DbName&gt;structure&lt;/DbName&gt;
        &lt;DbName&gt;genome&lt;/DbName&gt;
        &lt;DbName&gt;books&lt;/DbName&gt;
        &lt;DbName&gt;cancerchromosomes&lt;/DbName&gt;
        &lt;DbName&gt;cdd&lt;/DbName&gt;
        &lt;DbName&gt;gap&lt;/DbName&gt;
        &lt;DbName&gt;domains&lt;/DbName&gt;
        &lt;DbName&gt;gene&lt;/DbName&gt;
        &lt;DbName&gt;genomeprj&lt;/DbName&gt;
        &lt;DbName&gt;gensat&lt;/DbName&gt;
        &lt;DbName&gt;geo&lt;/DbName&gt;
        &lt;DbName&gt;gds&lt;/DbName&gt;
        &lt;DbName&gt;homologene&lt;/DbName&gt;
        &lt;DbName&gt;journals&lt;/DbName&gt;
        &lt;DbName&gt;mesh&lt;/DbName&gt;
        &lt;DbName&gt;ncbisearch&lt;/DbName&gt;
        &lt;DbName&gt;nlmcatalog&lt;/DbName&gt;
        &lt;DbName&gt;omia&lt;/DbName&gt;
        &lt;DbName&gt;omim&lt;/DbName&gt;
        &lt;DbName&gt;pmc&lt;/DbName&gt;
        &lt;DbName&gt;popset&lt;/DbName&gt;
        &lt;DbName&gt;probe&lt;/DbName&gt;
        &lt;DbName&gt;proteinclusters&lt;/DbName&gt;
        &lt;DbName&gt;pcassay&lt;/DbName&gt;
        &lt;DbName&gt;pccompound&lt;/DbName&gt;
        &lt;DbName&gt;pcsubstance&lt;/DbName&gt;
        &lt;DbName&gt;snp&lt;/DbName&gt;
        &lt;DbName&gt;taxonomy&lt;/DbName&gt;
        &lt;DbName&gt;toolkit&lt;/DbName&gt;
        &lt;DbName&gt;unigene&lt;/DbName&gt;
        &lt;DbName&gt;unists&lt;/DbName&gt;
&lt;/DbList&gt;
&lt;/eInfoResult&gt;
</pre><p>Since this is a fairly simple XML file, we could extract the information it contains simply by string searching. Using <code>Bio.Entrez</code>&#X2019;s parser instead, we can directly parse this XML file into a Python object:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; handle = Entrez.einfo()
&gt;&gt;&gt; record = Entrez.read(handle)
</pre><p>Now <code>record</code> is a dictionary with exactly one key:
</p><pre class="verbatim">&gt;&gt;&gt; record.keys()
[u'DbList']
</pre><p>The values stored in this key is the list of database names shown in the XML above:
</p><pre class="verbatim">&gt;&gt;&gt; record["DbList"]
['pubmed', 'protein', 'nucleotide', 'nuccore', 'nucgss', 'nucest',
 'structure', 'genome', 'books', 'cancerchromosomes', 'cdd', 'gap',
 'domains', 'gene', 'genomeprj', 'gensat', 'geo', 'gds', 'homologene',
 'journals', 'mesh', 'ncbisearch', 'nlmcatalog', 'omia', 'omim', 'pmc',
 'popset', 'probe', 'proteinclusters', 'pcassay', 'pccompound',
 'pcsubstance', 'snp', 'taxonomy', 'toolkit', 'unigene', 'unists']
</pre><p>For each of these databases, we can use EInfo again to obtain more information:
</p><pre class="verbatim">&gt;&gt;&gt; handle = Entrez.einfo(db="pubmed")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; record["DbInfo"]["Description"]
'PubMed bibliographic record'
&gt;&gt;&gt; record["DbInfo"]["Count"]
'17989604'
&gt;&gt;&gt; record["DbInfo"]["LastUpdate"]
'2008/05/24 06:45'
</pre><p>Try <code>record["DbInfo"].keys()</code> for other information stored in this record.
One of the most useful is a list of possible search fields for use with ESearch:</p><pre class="verbatim">&gt;&gt;&gt; for field in record["DbInfo"]["FieldList"]:
...     print("%(Name)s, %(FullName)s, %(Description)s" % field)
ALL, All Fields, All terms from all searchable fields
UID, UID, Unique number assigned to publication
FILT, Filter, Limits the records
TITL, Title, Words in title of publication
WORD, Text Word, Free text associated with publication
MESH, MeSH Terms, Medical Subject Headings assigned to publication
MAJR, MeSH Major Topic, MeSH terms of major importance to publication
AUTH, Author, Author(s) of publication
JOUR, Journal, Journal abbreviation of publication
AFFL, Affiliation, Author's institutional affiliation and address
...
</pre><p>That&#X2019;s a long list, but indirectly this tells you that for the PubMed
database, you can do things like <span style="font-family:monospace">Jones[AUTH]</span> to search the
author field, or <span style="font-family:monospace">Sanger[AFFL]</span> to restrict to authors at the
Sanger Centre. This can be very handy - especially if you are not so
familiar with a particular database.</p>
<!--TOC section id="sec121" ESearch: Searching the Entrez databases-->
<h2 id="sec121" class="section">9.3&#XA0;&#XA0;ESearch: Searching the Entrez databases</h2><!--SEC END --><p>
<a id="sec:entrez-esearch"></a>
To search any of these databases, we use <code>Bio.Entrez.esearch()</code>. For example, let&#X2019;s search in PubMed for publications related to Biopython:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
&gt;&gt;&gt; handle = Entrez.esearch(db="pubmed", term="biopython")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; record["IdList"]
['19304878', '18606172', '16403221', '16377612', '14871861', '14630660', '12230038']
</pre><p>In this output, you see seven PubMed IDs (including 19304878 which is the PMID for the Biopython application note), which can be retrieved by EFetch (see section <a href="#sec%3Aefetch">9.6</a>).</p><p>You can also use ESearch to search GenBank. Here we&#X2019;ll do a quick
search for the <em>matK</em> gene in <em>Cypripedioideae</em> orchids
(see Section&#XA0;<a href="#sec%3Aentrez-einfo">9.2</a> about EInfo for one way to
find out which fields you can search in each Entrez database):</p><pre class="verbatim">&gt;&gt;&gt; handle = Entrez.esearch(db="nucleotide", term="Cypripedioideae[Orgn] AND matK[Gene]")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; record["Count"]
'25'
&gt;&gt;&gt; record["IdList"]
['126789333', '37222967', '37222966', '37222965', ..., '61585492']
</pre><p>Each of the IDs (126789333, 37222967, 37222966, &#X2026;) is a GenBank identifier.
See section&#XA0;<a href="#sec%3Aefetch">9.6</a> for information on how to actually download these GenBank records.</p><p>Note that instead of a species name like <span style="font-family:monospace">Cypripedioideae[Orgn]</span>, you can restrict the search using an NCBI taxon identifier, here this would be <span style="font-family:monospace">txid158330[Orgn]</span>. This isn&#X2019;t currently documented on the ESearch help page - the NCBI explained this in reply to an email query. You can often deduce the search term formatting by playing with the Entrez web interface. For example, including <span style="font-family:monospace">complete[prop]</span> in a genome search restricts to just completed genomes.</p><p>As a final example, let&#X2019;s get a list of computational journal titles:
</p><pre class="verbatim">&gt;&gt;&gt; handle = Entrez.esearch(db="journals", term="computational")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; record["Count"]
'16'
&gt;&gt;&gt; record["IdList"]
['30367', '33843', '33823', '32989', '33190', '33009', '31986',
 '34502', '8799', '22857', '32675', '20258', '33859', '32534',
 '32357', '32249']
</pre><p>Again, we could use EFetch to obtain more information for each of these journal IDs.</p><p>ESearch has many useful options &#X2014; see the <a href="http://www.ncbi.nlm.nih.gov/entrez/query/static/esearch_help.html">ESearch help page</a> for more information.</p>
<!--TOC section id="sec122" EPost: Uploading a list of identifiers-->
<h2 id="sec122" class="section">9.4&#XA0;&#XA0;EPost: Uploading a list of identifiers</h2><!--SEC END --><p>
EPost uploads a list of UIs for use in subsequent search strategies; see the
<a href="http://www.ncbi.nlm.nih.gov/entrez/query/static/epost_help.html">EPost help page</a> for more information. It is available from Biopython through
the <code>Bio.Entrez.epost()</code> function.</p><p>To give an example of when this is useful, suppose you have a long list of IDs
you want to download using EFetch (maybe sequences, maybe citations &#X2013;
anything). When you make a request with EFetch your list of IDs, the database
etc, are all turned into a long URL sent to the server. If your list of IDs is
long, this URL gets long, and long URLs can break (e.g. some proxies don&#X2019;t
cope well).</p><p>Instead, you can break this up into two steps, first uploading the list of IDs
using EPost (this uses an &#X201C;HTML post&#X201D; internally, rather than an &#X201C;HTML get&#X201D;,
getting round the long URL problem). With the history support, you can then
refer to this long list of IDs, and download the associated data with EFetch.</p><p>Let&#X2019;s look at a simple example to see how EPost works &#X2013; uploading some PubMed identifiers:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
&gt;&gt;&gt; id_list = ["19304878", "18606172", "16403221", "16377612", "14871861", "14630660"]
&gt;&gt;&gt; print(Entrez.epost("pubmed", id=",".join(id_list)).read())
&lt;?xml version="1.0"?&gt;
&lt;!DOCTYPE ePostResult PUBLIC "-//NLM//DTD ePostResult, 11 May 2002//EN"
 "http://www.ncbi.nlm.nih.gov/entrez/query/DTD/ePost_020511.dtd"&gt;
&lt;ePostResult&gt;
 &lt;QueryKey&gt;1&lt;/QueryKey&gt;
 &lt;WebEnv&gt;NCID_01_206841095_130.14.22.101_9001_1242061629&lt;/WebEnv&gt;
&lt;/ePostResult&gt;
</pre><p>The returned XML includes two important strings, <code>QueryKey</code> and <code>WebEnv</code> which together define your history session.
You would extract these values for use with another Entrez call such as EFetch:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
&gt;&gt;&gt; id_list = ["19304878", "18606172", "16403221", "16377612", "14871861", "14630660"]
&gt;&gt;&gt; search_results = Entrez.read(Entrez.epost("pubmed", id=",".join(id_list)))
&gt;&gt;&gt; webenv = search_results["WebEnv"]
&gt;&gt;&gt; query_key = search_results["QueryKey"] 
</pre><p>Section&#XA0;<a href="#sec%3Aentrez-webenv">9.15</a> shows how to use the history feature.</p>
<!--TOC section id="sec123" ESummary: Retrieving summaries from primary IDs-->
<h2 id="sec123" class="section">9.5&#XA0;&#XA0;ESummary: Retrieving summaries from primary IDs</h2><!--SEC END --><p>
ESummary retrieves document summaries from a list of primary IDs (see the <a href="http://www.ncbi.nlm.nih.gov/entrez/query/static/esummary_help.html">ESummary help page</a> for more information). In Biopython, ESummary is available as <code>Bio.Entrez.esummary()</code>. Using the search result above, we can for example find out more about the journal with ID 30367:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
&gt;&gt;&gt; handle = Entrez.esummary(db="journals", id="30367")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; record[0]["Id"]
'30367'
&gt;&gt;&gt; record[0]["Title"]
'Computational biology and chemistry'
&gt;&gt;&gt; record[0]["Publisher"]
'Pergamon,'
</pre>
<!--TOC section id="sec124" EFetch: Downloading full records from Entrez-->
<h2 id="sec124" class="section">9.6&#XA0;&#XA0;EFetch: Downloading full records from Entrez</h2><!--SEC END --><p>
<a id="sec:efetch"></a></p><p>EFetch is what you use when you want to retrieve a full record from Entrez.
This covers several possible databases, as described on the main <a href="http://eutils.ncbi.nlm.nih.gov/entrez/query/static/efetch_help.html">EFetch Help page</a>.</p><p>For most of their databases, the NCBI support several different file formats. Requesting a specific file format from Entrez using <code>Bio.Entrez.efetch()</code> requires specifying the <code>rettype</code> and/or <code>retmode</code> optional arguments. The different combinations are described for each database type on the pages linked to on <a href="http://www.ncbi.nlm.nih.gov/entrez/query/static/efetch_help.html">NCBI efetch webpage</a> (e.g. <a href="http://eutils.ncbi.nlm.nih.gov/corehtml/query/static/efetchlit_help.html">literature</a>, <a href="http://eutils.ncbi.nlm.nih.gov/corehtml/query/static/efetchseq_help.html">sequences</a> and <a href="http://eutils.ncbi.nlm.nih.gov/corehtml/query/static/efetchtax_help.html">taxonomy</a>).</p><p>One common usage is downloading sequences in the FASTA or GenBank/GenPept plain text formats (which can then be parsed with <code>Bio.SeqIO</code>, see Sections&#XA0;<a href="#sec%3ASeqIO_GenBank_Online">5.3.1</a> and&#XA0;<a href="#sec%3Aefetch">9.6</a>). From the <em>Cypripedioideae</em> example above, we can download GenBank record 186972394 using <code>Bio.Entrez.efetch</code>:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
&gt;&gt;&gt; handle = Entrez.efetch(db="nucleotide", id="186972394", rettype="gb", retmode="text")
&gt;&gt;&gt; print(handle.read())
LOCUS       EU490707                1302 bp    DNA     linear   PLN 05-MAY-2008
DEFINITION  Selenipedium aequinoctiale maturase K (matK) gene, partial cds;
            chloroplast.
ACCESSION   EU490707
VERSION     EU490707.1  GI:186972394
KEYWORDS    .
SOURCE      chloroplast Selenipedium aequinoctiale
  ORGANISM  Selenipedium aequinoctiale
            Eukaryota; Viridiplantae; Streptophyta; Embryophyta; Tracheophyta;
            Spermatophyta; Magnoliophyta; Liliopsida; Asparagales; Orchidaceae;
            Cypripedioideae; Selenipedium.
REFERENCE   1  (bases 1 to 1302)
  AUTHORS   Neubig,K.M., Whitten,W.M., Carlsward,B.S., Blanco,M.A.,
            Endara,C.L., Williams,N.H. and Moore,M.J.
  TITLE     Phylogenetic utility of ycf1 in orchids
  JOURNAL   Unpublished
REFERENCE   2  (bases 1 to 1302)
  AUTHORS   Neubig,K.M., Whitten,W.M., Carlsward,B.S., Blanco,M.A.,
            Endara,C.L., Williams,N.H. and Moore,M.J.
  TITLE     Direct Submission
  JOURNAL   Submitted (14-FEB-2008) Department of Botany, University of
            Florida, 220 Bartram Hall, Gainesville, FL 32611-8526, USA
FEATURES             Location/Qualifiers
     source          1..1302
                     /organism="Selenipedium aequinoctiale"
                     /organelle="plastid:chloroplast"
                     /mol_type="genomic DNA"
                     /specimen_voucher="FLAS:Blanco 2475"
                     /db_xref="taxon:256374"
     gene            &lt;1..&gt;1302
                     /gene="matK"
     CDS             &lt;1..&gt;1302
                     /gene="matK"
                     /codon_start=1
                     /transl_table=11
                     /product="maturase K"
                     /protein_id="ACC99456.1"
                     /db_xref="GI:186972395"
                     /translation="IFYEPVEIFGYDNKSSLVLVKRLITRMYQQNFLISSVNDSNQKG
                     FWGHKHFFSSHFSSQMVSEGFGVILEIPFSSQLVSSLEEKKIPKYQNLRSIHSIFPFL
                     EDKFLHLNYVSDLLIPHPIHLEILVQILQCRIKDVPSLHLLRLLFHEYHNLNSLITSK
                     KFIYAFSKRKKRFLWLLYNSYVYECEYLFQFLRKQSSYLRSTSSGVFLERTHLYVKIE
                     HLLVVCCNSFQRILCFLKDPFMHYVRYQGKAILASKGTLILMKKWKFHLVNFWQSYFH
                     FWSQPYRIHIKQLSNYSFSFLGYFSSVLENHLVVRNQMLENSFIINLLTKKFDTIAPV
                     ISLIGSLSKAQFCTVLGHPISKPIWTDFSDSDILDRFCRICRNLCRYHSGSSKKQVLY
                     RIKYILRLSCARTLARKHKSTVRTFMRRLGSGLLEEFFMEEE"
ORIGIN      
        1 attttttacg aacctgtgga aatttttggt tatgacaata aatctagttt agtacttgtg
       61 aaacgtttaa ttactcgaat gtatcaacag aattttttga tttcttcggt taatgattct
      121 aaccaaaaag gattttgggg gcacaagcat tttttttctt ctcatttttc ttctcaaatg
      181 gtatcagaag gttttggagt cattctggaa attccattct cgtcgcaatt agtatcttct
      241 cttgaagaaa aaaaaatacc aaaatatcag aatttacgat ctattcattc aatatttccc
      301 tttttagaag acaaattttt acatttgaat tatgtgtcag atctactaat accccatccc
      361 atccatctgg aaatcttggt tcaaatcctt caatgccgga tcaaggatgt tccttctttg
      421 catttattgc gattgctttt ccacgaatat cataatttga atagtctcat tacttcaaag
      481 aaattcattt acgccttttc aaaaagaaag aaaagattcc tttggttact atataattct
      541 tatgtatatg aatgcgaata tctattccag tttcttcgta aacagtcttc ttatttacga
      601 tcaacatctt ctggagtctt tcttgagcga acacatttat atgtaaaaat agaacatctt
      661 ctagtagtgt gttgtaattc ttttcagagg atcctatgct ttctcaagga tcctttcatg
      721 cattatgttc gatatcaagg aaaagcaatt ctggcttcaa agggaactct tattctgatg
      781 aagaaatgga aatttcatct tgtgaatttt tggcaatctt attttcactt ttggtctcaa
      841 ccgtatagga ttcatataaa gcaattatcc aactattcct tctcttttct ggggtatttt
      901 tcaagtgtac tagaaaatca tttggtagta agaaatcaaa tgctagagaa ttcatttata
      961 ataaatcttc tgactaagaa attcgatacc atagccccag ttatttctct tattggatca
     1021 ttgtcgaaag ctcaattttg tactgtattg ggtcatccta ttagtaaacc gatctggacc
     1081 gatttctcgg attctgatat tcttgatcga ttttgccgga tatgtagaaa tctttgtcgt
     1141 tatcacagcg gatcctcaaa aaaacaggtt ttgtatcgta taaaatatat acttcgactt
     1201 tcgtgtgcta gaactttggc acggaaacat aaaagtacag tacgcacttt tatgcgaaga
     1261 ttaggttcgg gattattaga agaattcttt atggaagaag aa
//
</pre><p>The arguments <code>rettype="gb"</code> and <code>retmode="text"</code> let us download this record in the GenBank format.</p><p>Note that until Easter 2009, the Entrez EFetch API let you use &#X201C;genbank&#X201D; as the
return type, however the NCBI now insist on using the official return types of
&#X201C;gb&#X201D; or &#X201C;gbwithparts&#X201D; (or &#X201C;gp&#X201D; for proteins) as described on online.
Also not that until Feb 2012, the Entrez EFetch API would default to returning
plain text files, but now defaults to XML.</p><p>Alternatively, you could for example use <code>rettype="fasta"</code> to get the Fasta-format; see the <a href="http://www.ncbi.nlm.nih.gov/entrez/query/static/efetchseq_help.html">EFetch Sequences Help page</a> for other options. Remember &#X2013; the available formats depend on which database you are downloading from - see the main <a href="http://eutils.ncbi.nlm.nih.gov/entrez/query/static/efetch_help.html">EFetch Help page</a>.</p><p>If you fetch the record in one of the formats accepted by <code>Bio.SeqIO</code> (see Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>), you could directly parse it into a <code>SeqRecord</code>:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez, SeqIO
&gt;&gt;&gt; handle = Entrez.efetch(db="nucleotide", id="186972394", rettype="gb", retmode="text")
&gt;&gt;&gt; record = SeqIO.read(handle, "genbank")
&gt;&gt;&gt; handle.close()
&gt;&gt;&gt; print(record)
ID: EU490707.1
Name: EU490707
Description: Selenipedium aequinoctiale maturase K (matK) gene, partial cds; chloroplast.
Number of features: 3
...
Seq('ATTTTTTACGAACCTGTGGAAATTTTTGGTTATGACAATAAATCTAGTTTAGTA...GAA', IUPACAmbiguousDNA())
</pre><p>Note that a more typical use would be to save the sequence data to a local file, and <em>then</em> parse it with <code>Bio.SeqIO</code>. This can save you having to re-download the same file repeatedly while working on your script, and places less load on the NCBI&#X2019;s servers. For example:</p><pre class="verbatim">import os
from Bio import SeqIO
from Bio import Entrez
Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
filename = "gi_186972394.gbk"
if not os.path.isfile(filename):
    # Downloading...
    net_handle = Entrez.efetch(db="nucleotide",id="186972394",rettype="gb", retmode="text")
    out_handle = open(filename, "w")
    out_handle.write(net_handle.read())
    out_handle.close()
    net_handle.close()
    print("Saved")

print("Parsing...")
record = SeqIO.read(filename, "genbank")
print(record)
</pre><p>To get the output in XML format, which you can parse using the <code>Bio.Entrez.read()</code> function, use <code>retmode="xml"</code>:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; handle = Entrez.efetch(db="nucleotide", id="186972394", retmode="xml")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; handle.close()
&gt;&gt;&gt; record[0]["GBSeq_definition"] 
'Selenipedium aequinoctiale maturase K (matK) gene, partial cds; chloroplast'
&gt;&gt;&gt; record[0]["GBSeq_source"] 
'chloroplast Selenipedium aequinoctiale'
</pre><p>So, that dealt with sequences. For examples of parsing file formats specific to the other databases (e.g. the <code>MEDLINE</code> format used in PubMed), see Section&#XA0;<a href="#sec%3Aentrez-specialized-parsers">9.12</a>.</p><p>If you want to perform a search with <code>Bio.Entrez.esearch()</code>, and then download the records with <code>Bio.Entrez.efetch()</code>, you should use the WebEnv history feature &#X2013; see Section&#XA0;<a href="#sec%3Aentrez-webenv">9.15</a>.</p>
<!--TOC section id="sec125" ELink: Searching for related items in NCBI Entrez-->
<h2 id="sec125" class="section">9.7&#XA0;&#XA0;ELink: Searching for related items in NCBI Entrez</h2><!--SEC END --><p>
<a id="sec:elink"></a></p><p>ELink, available from Biopython as <code>Bio.Entrez.elink()</code>, can be used to find related items in the NCBI Entrez databases. For example, you can us this to find nucleotide entries for an entry in the gene database,
and other cool stuff.</p><p>Let&#X2019;s use ELink to find articles related to the Biopython application note published in <span style="font-style:italic">Bioinformatics</span> in 2009. The PubMed ID of this article is 19304878:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"
&gt;&gt;&gt; pmid = "19304878"
&gt;&gt;&gt; record = Entrez.read(Entrez.elink(dbfrom="pubmed", id=pmid))
</pre><p>The <code>record</code> variable consists of a Python list, one for each database in which we searched. Since we specified only one PubMed ID to search for, <code>record</code> contains only one item. This item is a dictionary containing information about our search term, as well as all the related items that were found:</p><pre class="verbatim">&gt;&gt;&gt; record[0]["DbFrom"]
'pubmed'
&gt;&gt;&gt; record[0]["IdList"]
['19304878']
</pre><p>The <code>"LinkSetDb"</code> key contains the search results, stored as a list consisting of one item for each target database. In our search results, we only find hits in the PubMed database (although sub-divided into categories):</p><pre class="verbatim">&gt;&gt;&gt; len(record[0]["LinkSetDb"])
5
&gt;&gt;&gt; for linksetdb in record[0]["LinkSetDb"]:
...     print(linksetdb["DbTo"], linksetdb["LinkName"], len(linksetdb["Link"]))
... 
pubmed pubmed_pubmed 110
pubmed pubmed_pubmed_combined 6
pubmed pubmed_pubmed_five 6
pubmed pubmed_pubmed_reviews 5
pubmed pubmed_pubmed_reviews_five 5
</pre><p>The actual search results are stored as under the <code>"Link"</code> key. In total, 110 items were found under
standard search.
Let&#X2019;s now at the first search result:
</p><pre class="verbatim">&gt;&gt;&gt; record[0]["LinkSetDb"][0]["Link"][0]
{u'Id': '19304878'}
</pre><p>This is the article we searched for, which doesn&#X2019;t help us much, so let&#X2019;s look at the second search result:</p><pre class="verbatim">&gt;&gt;&gt; record[0]["LinkSetDb"][0]["Link"][1]
{u'Id': '14630660'}
</pre><p>This paper, with PubMed ID 14630660, is about the Biopython PDB parser.</p><p>We can use a loop to print out all PubMed IDs:
</p><pre class="verbatim">&gt;&gt;&gt; for link in record[0]["LinkSetDb"][0]["Link"]:
...     print(link["Id"])
19304878
14630660
18689808
17121776
16377612
12368254
......
</pre><p>Now that was nice, but personally I am often more interested to find out if a paper has been cited.
Well, ELink can do that too &#X2013; at least for journals in Pubmed Central (see Section&#XA0;<a href="#sec%3Aelink-citations">9.15.3</a>).</p><p>For help on ELink, see the <a href="http://www.ncbi.nlm.nih.gov/entrez/query/static/elink_help.html">ELink help page</a>.
There is an entire sub-page just for the <a href="http://eutils.ncbi.nlm.nih.gov/corehtml/query/static/entrezlinks.html">link names</a>, describing how different databases can be cross referenced.</p>
<!--TOC section id="sec126" EGQuery: Global Query - counts for search terms-->
<h2 id="sec126" class="section">9.8&#XA0;&#XA0;EGQuery: Global Query - counts for search terms</h2><!--SEC END --><p>
EGQuery provides counts for a search term in each of the Entrez databases (i.e. a global query). This is particularly useful to find out how many items your search terms would find in each database without actually performing lots of separate searches with ESearch (see the example in <a href="#subsec%3Aentrez_example_genbank">9.14.2</a> below).</p><p>In this example, we use <code>Bio.Entrez.egquery()</code> to obtain the counts for &#X201C;Biopython&#X201D;:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
&gt;&gt;&gt; handle = Entrez.egquery(term="biopython")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; for row in record["eGQueryResult"]:
...     print(row["DbName"], row["Count"])
... 
pubmed 6
pmc 62
journals 0
...
</pre><p>See the <a href="http://www.ncbi.nlm.nih.gov/entrez/query/static/egquery_help.html">EGQuery help page</a> for more information.</p>
<!--TOC section id="sec127" ESpell: Obtaining spelling suggestions-->
<h2 id="sec127" class="section">9.9&#XA0;&#XA0;ESpell: Obtaining spelling suggestions</h2><!--SEC END --><p>
ESpell retrieves spelling suggestions. In this example, we use <code>Bio.Entrez.espell()</code> to obtain the correct spelling of Biopython:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
&gt;&gt;&gt; handle = Entrez.espell(term="biopythooon")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; record["Query"]
'biopythooon'
&gt;&gt;&gt; record["CorrectedQuery"]
'biopython'
</pre><p>See the <a href="http://www.ncbi.nlm.nih.gov/entrez/query/static/espell_help.html">ESpell help page</a> for more information.
The main use of this is for GUI tools to provide automatic suggestions for search terms.</p>
<!--TOC section id="sec128" Parsing huge Entrez XML files-->
<h2 id="sec128" class="section">9.10&#XA0;&#XA0;Parsing huge Entrez XML files</h2><!--SEC END --><p>The <code>Entrez.read</code> function reads the entire XML file returned by Entrez into a single Python object, which is kept in memory. To parse Entrez XML files too large to fit in memory, you can use the function <code>Entrez.parse</code>. This is a generator function that reads records in the XML file one by one. This function is only useful if the XML file reflects a Python list object (in other words, if <code>Entrez.read</code> on a computer with infinite memory resources would return a Python list).</p><p>For example, you can download the entire Entrez Gene database for a given organism as a file from NCBI&#X2019;s ftp site. These files can be very large. As an example, on September 4, 2009, the file <code>Homo_sapiens.ags.gz</code>, containing the Entrez Gene database for human, had a size of 116576 kB. This file, which is in the <code>ASN</code> format, can be converted into an XML file using NCBI&#X2019;s <code>gene2xml</code> program (see NCBI&#X2019;s ftp site for more information):</p><pre class="verbatim">gene2xml -b T -i Homo_sapiens.ags -o Homo_sapiens.xml
</pre><p>The resulting XML file has a size of 6.1 GB. Attempting <code>Entrez.read</code> on this file will result in a <code>MemoryError</code> on many computers.</p><p>The XML file <code>Homo_sapiens.xml</code> consists of a list of Entrez gene records, each corresponding to one Entrez gene in human. <code>Entrez.parse</code> retrieves these gene records one by one. You can then print out or store the relevant information in each record by iterating over the records. For example, this script iterates over the Entrez gene records and prints out the gene numbers and names for all current genes:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; handle = open("Homo_sapiens.xml")
&gt;&gt;&gt; records = Entrez.parse(handle)

&gt;&gt;&gt; for record in records:
...     status = record['Entrezgene_track-info']['Gene-track']['Gene-track_status']
...     if status.attributes['value']=='discontinued':
...         continue
...     geneid = record['Entrezgene_track-info']['Gene-track']['Gene-track_geneid']
...     genename = record['Entrezgene_gene']['Gene-ref']['Gene-ref_locus']
...     print(geneid, genename)
</pre><p>This will print:
</p><pre class="verbatim">1 A1BG
2 A2M
3 A2MP
8 AA
9 NAT1
10 NAT2
11 AACP
12 SERPINA3
13 AADAC
14 AAMP
15 AANAT
16 AARS
17 AAVS1
...
</pre>
<!--TOC section id="sec129" Handling errors-->
<h2 id="sec129" class="section">9.11&#XA0;&#XA0;Handling errors</h2><!--SEC END --><p>Three things can go wrong when parsing an XML file:
</p><ul class="itemize"><li class="li-itemize">
The file may not be an XML file to begin with;
</li><li class="li-itemize">The file may end prematurely or otherwise be corrupted;
</li><li class="li-itemize">The file may be correct XML, but contain items that are not represented in the associated DTD.
</li></ul><p>The first case occurs if, for example, you try to parse a Fasta file as if it were an XML file:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; handle = open("NC_005816.fna") # a Fasta file
&gt;&gt;&gt; record = Entrez.read(handle)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/usr/local/lib/python2.7/site-packages/Bio/Entrez/__init__.py", line 257, in read
    record = handler.read(handle)
  File "/usr/local/lib/python2.7/site-packages/Bio/Entrez/Parser.py", line 164, in read
    raise NotXMLError(e)
Bio.Entrez.Parser.NotXMLError: Failed to parse the XML data (syntax error: line 1, column 0). Please make sure that the input data are in XML format.
</pre><p>Here, the parser didn&#X2019;t find the <code>&lt;?xml ...</code> tag with which an XML file is supposed to start, and therefore decides (correctly) that the file is not an XML file.</p><p>When your file is in the XML format but is corrupted (for example, by ending prematurely), the parser will raise a CorruptedXMLError.
Here is an example of an XML file that ends prematurely:
</p><pre class="verbatim">&lt;?xml version="1.0"?&gt;
&lt;!DOCTYPE eInfoResult PUBLIC "-//NLM//DTD eInfoResult, 11 May 2002//EN" "http://www.ncbi.nlm.nih.gov/entrez/query/DTD/eInfo_020511.dtd"&gt;
&lt;eInfoResult&gt;
&lt;DbList&gt;
        &lt;DbName&gt;pubmed&lt;/DbName&gt;
        &lt;DbName&gt;protein&lt;/DbName&gt;
        &lt;DbName&gt;nucleotide&lt;/DbName&gt;
        &lt;DbName&gt;nuccore&lt;/DbName&gt;
        &lt;DbName&gt;nucgss&lt;/DbName&gt;
        &lt;DbName&gt;nucest&lt;/DbName&gt;
        &lt;DbName&gt;structure&lt;/DbName&gt;
        &lt;DbName&gt;genome&lt;/DbName&gt;
        &lt;DbName&gt;books&lt;/DbName&gt;
        &lt;DbName&gt;cancerchromosomes&lt;/DbName&gt;
        &lt;DbName&gt;cdd&lt;/DbName&gt;
</pre><p>which will generate the following traceback:
</p><pre class="verbatim">&gt;&gt;&gt; Entrez.read(handle)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/usr/local/lib/python2.7/site-packages/Bio/Entrez/__init__.py", line 257, in read
    record = handler.read(handle)
  File "/usr/local/lib/python2.7/site-packages/Bio/Entrez/Parser.py", line 160, in read
    raise CorruptedXMLError(e)
Bio.Entrez.Parser.CorruptedXMLError: Failed to parse the XML data (no element found: line 16, column 0). Please make sure that the input data are not corrupted.

&gt;&gt;&gt;
</pre><p>Note that the error message tells you at what point in the XML file the error was detected.</p><p>The third type of error occurs if the XML file contains tags that do not have a description in the corresponding DTD file. This is an example of such an XML file:</p><pre class="verbatim">&lt;?xml version="1.0"?&gt;
&lt;!DOCTYPE eInfoResult PUBLIC "-//NLM//DTD eInfoResult, 11 May 2002//EN" "http://www.ncbi.nlm.nih.gov/entrez/query/DTD/eInfo_020511.dtd"&gt;
&lt;eInfoResult&gt;
        &lt;DbInfo&gt;
        &lt;DbName&gt;pubmed&lt;/DbName&gt;
        &lt;MenuName&gt;PubMed&lt;/MenuName&gt;
        &lt;Description&gt;PubMed bibliographic record&lt;/Description&gt;
        &lt;Count&gt;20161961&lt;/Count&gt;
        &lt;LastUpdate&gt;2010/09/10 04:52&lt;/LastUpdate&gt;
        &lt;FieldList&gt;
                &lt;Field&gt;
...
                &lt;/Field&gt;
        &lt;/FieldList&gt;
        &lt;DocsumList&gt;
                &lt;Docsum&gt;
                        &lt;DsName&gt;PubDate&lt;/DsName&gt;
                        &lt;DsType&gt;4&lt;/DsType&gt;
                        &lt;DsTypeName&gt;string&lt;/DsTypeName&gt;
                &lt;/Docsum&gt;
                &lt;Docsum&gt;
                        &lt;DsName&gt;EPubDate&lt;/DsName&gt;
...
        &lt;/DbInfo&gt;
&lt;/eInfoResult&gt;
</pre><p>In this file, for some reason the tag <code>&lt;DocsumList&gt;</code> (and several others) are not listed in the DTD file <code>eInfo_020511.dtd</code>, which is specified on the second line as the DTD for this XML file. By default, the parser will stop and raise a ValidationError if it cannot find some tag in the DTD:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; handle = open("einfo3.xml")
&gt;&gt;&gt; record = Entrez.read(handle)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/usr/local/lib/python2.7/site-packages/Bio/Entrez/__init__.py", line 257, in read
    record = handler.read(handle)
  File "/usr/local/lib/python2.7/site-packages/Bio/Entrez/Parser.py", line 154, in read
    self.parser.ParseFile(handle)
  File "/usr/local/lib/python2.7/site-packages/Bio/Entrez/Parser.py", line 246, in startElementHandler
    raise ValidationError(name)
Bio.Entrez.Parser.ValidationError: Failed to find tag 'DocsumList' in the DTD. To skip all tags that are not represented in the DTD, please call Bio.Entrez.read or Bio.Entrez.parse with validate=False.
</pre><p>Optionally, you can instruct the parser to skip such tags instead of raising a ValidationError. This is done by calling <code>Entrez.read</code> or <code>Entrez.parse</code> with the argument <code>validate</code> equal to False:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; handle = open("einfo3.xml")
&gt;&gt;&gt; record = Entrez.read(handle, validate=False)
&gt;&gt;&gt;
</pre><p>Of course, the information contained in the XML tags that are not in the DTD are not present in the record returned by <code>Entrez.read</code>.</p>
<!--TOC section id="sec130" Specialized parsers-->
<h2 id="sec130" class="section">9.12&#XA0;&#XA0;Specialized parsers</h2><!--SEC END --><p>
<a id="sec:entrez-specialized-parsers"></a></p><p>The <code>Bio.Entrez.read()</code> function can parse most (if not all) XML output returned by Entrez. Entrez typically allows you to retrieve records in other formats, which may have some advantages compared to the XML format in terms of readability (or download size).</p><p>To request a specific file format from Entrez using <code>Bio.Entrez.efetch()</code> requires specifying the <code>rettype</code> and/or <code>retmode</code> optional arguments. The different combinations are described for each database type on the <a href="http://www.ncbi.nlm.nih.gov/entrez/query/static/efetch_help.html">NCBI efetch webpage</a>.</p><p>One obvious case is you may prefer to download sequences in the FASTA or GenBank/GenPept plain text formats (which can then be parsed with <code>Bio.SeqIO</code>, see Sections&#XA0;<a href="#sec%3ASeqIO_GenBank_Online">5.3.1</a> and&#XA0;<a href="#sec%3Aefetch">9.6</a>). For the literature databases, Biopython contains a parser for the <code>MEDLINE</code> format used in PubMed.</p>
<!--TOC subsection id="sec131" Parsing Medline records-->
<h3 id="sec131" class="subsection">9.12.1&#XA0;&#XA0;Parsing Medline records</h3><!--SEC END --><p>
<a id="subsec:entrez-and-medline"></a>
You can find the Medline parser in <code>Bio.Medline</code>. Suppose we want to parse the file <code>pubmed_result1.txt</code>, containing one Medline record. You can find this file in Biopython&#X2019;s <code>Tests\Medline</code> directory. The file looks like this:</p><pre class="verbatim">PMID- 12230038
OWN - NLM
STAT- MEDLINE
DA  - 20020916
DCOM- 20030606
LR  - 20041117
PUBM- Print
IS  - 1467-5463 (Print)
VI  - 3
IP  - 3
DP  - 2002 Sep
TI  - The Bio* toolkits--a brief overview.
PG  - 296-302
AB  - Bioinformatics research is often difficult to do with commercial software. The
      Open Source BioPerl, BioPython and Biojava projects provide toolkits with
...
</pre><p>We first open the file and then parse it:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Medline
&gt;&gt;&gt; with open("pubmed_result1.txt") as handle:
...    record = Medline.read(handle)
...
</pre><p>The <code>record</code> now contains the Medline record as a Python dictionary:
</p><pre class="verbatim">&gt;&gt;&gt; record["PMID"]
'12230038'
</pre><pre class="verbatim">&gt;&gt;&gt; record["AB"]
'Bioinformatics research is often difficult to do with commercial software.
The Open Source BioPerl, BioPython and Biojava projects provide toolkits with
multiple functionality that make it easier to create customised pipelines or
analysis. This review briefly compares the quirks of the underlying languages
and the functionality, documentation, utility and relative advantages of the
Bio counterparts, particularly from the point of view of the beginning
biologist programmer.'
</pre><p>The key names used in a Medline record can be rather obscure; use
</p><pre class="verbatim">&gt;&gt;&gt; help(record)
</pre><p>for a brief summary.</p><p>To parse a file containing multiple Medline records, you can use the <code>parse</code> function instead:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Medline
&gt;&gt;&gt; with open("pubmed_result2.txt") as handle:
...     for record in Medline.parse(handle):
...         print(record["TI"])
...
A high level interface to SCOP and ASTRAL implemented in python.
GenomeDiagram: a python package for the visualization of large-scale genomic data.
Open source clustering software.
PDB file parser and structure class implemented in Python.
</pre><p>Instead of parsing Medline records stored in files, you can also parse Medline records downloaded by <code>Bio.Entrez.efetch</code>. For example, let&#X2019;s look at all Medline records in PubMed related to Biopython:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
&gt;&gt;&gt; handle = Entrez.esearch(db="pubmed", term="biopython")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; record["IdList"]
['19304878', '18606172', '16403221', '16377612', '14871861', '14630660', '12230038']
</pre><p>We now use <code>Bio.Entrez.efetch</code> to download these Medline records:
</p><pre class="verbatim">&gt;&gt;&gt; idlist = record["IdList"]
&gt;&gt;&gt; handle = Entrez.efetch(db="pubmed", id=idlist, rettype="medline", retmode="text")
</pre><p>Here, we specify <code>rettype="medline", retmode="text"</code> to obtain the Medline records in plain-text Medline format. Now we use <code>Bio.Medline</code> to parse these records:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Medline
&gt;&gt;&gt; records = Medline.parse(handle)
&gt;&gt;&gt; for record in records:
...     print(record["AU"])
['Cock PJ', 'Antao T', 'Chang JT', 'Chapman BA', 'Cox CJ', 'Dalke A', ..., 'de Hoon MJ']
['Munteanu CR', 'Gonzalez-Diaz H', 'Magalhaes AL']
['Casbon JA', 'Crooks GE', 'Saqi MA']
['Pritchard L', 'White JA', 'Birch PR', 'Toth IK']
['de Hoon MJ', 'Imoto S', 'Nolan J', 'Miyano S']
['Hamelryck T', 'Manderick B']
['Mangalam H']
</pre><p>For comparison, here we show an example using the XML format:
</p><pre class="verbatim">&gt;&gt;&gt; idlist = record["IdList"]
&gt;&gt;&gt; handle = Entrez.efetch(db="pubmed", id=idlist, rettype="medline", retmode="xml")
&gt;&gt;&gt; records = Entrez.read(handle)
&gt;&gt;&gt; for record in records:
...     print(record["MedlineCitation"]["Article"]["ArticleTitle"])
Biopython: freely available Python tools for computational molecular biology and
 bioinformatics.
Enzymes/non-enzymes classification model complexity based on composition, sequence,
 3D and topological indices.
A high level interface to SCOP and ASTRAL implemented in python.
GenomeDiagram: a python package for the visualization of large-scale genomic data.
Open source clustering software.
PDB file parser and structure class implemented in Python.
The Bio* toolkits--a brief overview.
</pre><p>Note that in both of these examples, for simplicity we have naively combined ESearch and EFetch.
In this situation, the NCBI would expect you to use their history feature,
as illustrated in Section&#XA0;<a href="#sec%3Aentrez-webenv">9.15</a>.</p>
<!--TOC subsection id="sec132" Parsing GEO records-->
<h3 id="sec132" class="subsection">9.12.2&#XA0;&#XA0;Parsing GEO records</h3><!--SEC END --><p>GEO (<a href="http://www.ncbi.nlm.nih.gov/geo/">Gene Expression Omnibus</a>)
is a data repository of high-throughput gene expression and hybridization
array data. The <code>Bio.Geo</code> module can be used to parse GEO-formatted
data.</p><p>The following code fragment shows how to parse the example GEO file
<code>GSE16.txt</code> into a record and print the record:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Geo
&gt;&gt;&gt; handle = open("GSE16.txt")
&gt;&gt;&gt; records = Geo.parse(handle)
&gt;&gt;&gt; for record in records:
...     print(record)
</pre><p>You can search the &#X201C;gds&#X201D; database (GEO datasets) with ESearch:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com" # Always tell NCBI who you are
&gt;&gt;&gt; handle = Entrez.esearch(db="gds", term="GSE16")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; record["Count"]
2
&gt;&gt;&gt; record["IdList"]
['200000016', '100000028']
</pre><p>From the Entrez website, UID &#X201C;200000016&#X201D; is GDS16 while the other hit
&#X201C;100000028&#X201D; is for the associated platform, GPL28. Unfortunately, at the
time of writing the NCBI don&#X2019;t seem to support downloading GEO files using
Entrez (not as XML, nor in the <span style="font-style:italic">Simple Omnibus Format in Text</span> (SOFT)
format).</p><p>However, it is actually pretty straight forward to download the GEO files by FTP
from <a href="ftp://ftp.ncbi.nih.gov/pub/geo/"><span style="font-family:monospace">ftp://ftp.ncbi.nih.gov/pub/geo/</span></a> instead. In this case you might want
<a href="ftp://ftp.ncbi.nih.gov/pub/geo/DATA/SOFT/by_series/GSE16/GSE16_family.soft.gz"><span style="font-family:monospace">ftp://ftp.ncbi.nih.gov/pub/geo/DATA/SOFT/by_series/GSE16/GSE16_family.soft.gz</span></a>
(a compressed file, see the Python module gzip).</p>
<!--TOC subsection id="sec133" Parsing UniGene records-->
<h3 id="sec133" class="subsection">9.12.3&#XA0;&#XA0;Parsing UniGene records</h3><!--SEC END --><p>UniGene is an NCBI database of the transcriptome, with each UniGene record showing the set of transcripts that are associated with a particular gene in a specific organism. A typical UniGene record looks like this:</p><pre class="verbatim">ID          Hs.2
TITLE       N-acetyltransferase 2 (arylamine N-acetyltransferase)
GENE        NAT2
CYTOBAND    8p22
GENE_ID     10
LOCUSLINK   10
HOMOL       YES
EXPRESS      bone| connective tissue| intestine| liver| liver tumor| normal| soft tissue/muscle tissue tumor| adult
RESTR_EXPR   adult
CHROMOSOME  8
STS         ACC=PMC310725P3 UNISTS=272646
STS         ACC=WIAF-2120 UNISTS=44576
STS         ACC=G59899 UNISTS=137181
...
STS         ACC=GDB:187676 UNISTS=155563
PROTSIM     ORG=10090; PROTGI=6754794; PROTID=NP_035004.1; PCT=76.55; ALN=288
PROTSIM     ORG=9796; PROTGI=149742490; PROTID=XP_001487907.1; PCT=79.66; ALN=288
PROTSIM     ORG=9986; PROTGI=126722851; PROTID=NP_001075655.1; PCT=76.90; ALN=288
...
PROTSIM     ORG=9598; PROTGI=114619004; PROTID=XP_519631.2; PCT=98.28; ALN=288

SCOUNT      38
SEQUENCE    ACC=BC067218.1; NID=g45501306; PID=g45501307; SEQTYPE=mRNA
SEQUENCE    ACC=NM_000015.2; NID=g116295259; PID=g116295260; SEQTYPE=mRNA
SEQUENCE    ACC=D90042.1; NID=g219415; PID=g219416; SEQTYPE=mRNA
SEQUENCE    ACC=D90040.1; NID=g219411; PID=g219412; SEQTYPE=mRNA
SEQUENCE    ACC=BC015878.1; NID=g16198419; PID=g16198420; SEQTYPE=mRNA
SEQUENCE    ACC=CR407631.1; NID=g47115198; PID=g47115199; SEQTYPE=mRNA
SEQUENCE    ACC=BG569293.1; NID=g13576946; CLONE=IMAGE:4722596; END=5'; LID=6989; SEQTYPE=EST; TRACE=44157214
...
SEQUENCE    ACC=AU099534.1; NID=g13550663; CLONE=HSI08034; END=5'; LID=8800; SEQTYPE=EST
//
</pre><p>This particular record shows the set of transcripts (shown in the <code>SEQUENCE</code> lines) that originate from the human gene NAT2, encoding en N-acetyltransferase. The <code>PROTSIM</code> lines show proteins with significant similarity to NAT2, whereas the <code>STS</code> lines show the corresponding sequence-tagged sites in the genome.</p><p>To parse UniGene files, use the <code>Bio.UniGene</code> module:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import UniGene
&gt;&gt;&gt; input = open("myunigenefile.data")
&gt;&gt;&gt; record = UniGene.read(input)
</pre><p>The <code>record</code> returned by <code>UniGene.read</code> is a Python object with attributes corresponding to the fields in the UniGene record. For example,
</p><pre class="verbatim">&gt;&gt;&gt; record.ID
"Hs.2"
&gt;&gt;&gt; record.title
"N-acetyltransferase 2 (arylamine N-acetyltransferase)"
</pre><p>The <code>EXPRESS</code> and <code>RESTR_EXPR</code> lines are stored as Python lists of strings:
</p><pre class="verbatim">['bone', 'connective tissue', 'intestine', 'liver', 'liver tumor', 'normal', 'soft tissue/muscle tissue tumor', 'adult']
</pre><p>Specialized objects are returned for the <code>STS</code>, <code>PROTSIM</code>, and <code>SEQUENCE</code> lines, storing the keys shown in each line as attributes:
</p><pre class="verbatim">&gt;&gt;&gt; record.sts[0].acc
'PMC310725P3'
&gt;&gt;&gt; record.sts[0].unists
'272646'
</pre><p>and similarly for the <code>PROTSIM</code> and <code>SEQUENCE</code> lines.</p><p>To parse a file containing more than one UniGene record, use the <code>parse</code> function in <code>Bio.UniGene</code>:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import UniGene
&gt;&gt;&gt; input = open("unigenerecords.data")
&gt;&gt;&gt; records = UniGene.parse(input)
&gt;&gt;&gt; for record in records:
...     print(record.ID)
</pre>
<!--TOC section id="sec134" Using a proxy-->
<h2 id="sec134" class="section">9.13&#XA0;&#XA0;Using a proxy</h2><!--SEC END --><p>Normally you won&#X2019;t have to worry about using a proxy, but if this is an issue
on your network here is how to deal with it. Internally, <code>Bio.Entrez</code>
uses the standard Python library <code>urllib</code> for accessing the NCBI servers.
This will check an environment variable called <code>http_proxy</code> to configure
any simple proxy automatically. Unfortunately this module does not support
the use of proxies which require authentication.</p><p>You may choose to set the <code>http_proxy</code> environment variable once (how you
do this will depend on your operating system). Alternatively you can set this
within Python at the start of your script, for example:</p><pre class="verbatim">import os
os.environ["http_proxy"] = "http://proxyhost.example.com:8080"
</pre><p>See the <a href="http://www.python.org/doc/lib/module-urllib.html">urllib documentation</a> for more details.</p>
<!--TOC section id="sec135" Examples-->
<h2 id="sec135" class="section">9.14&#XA0;&#XA0;Examples</h2><!--SEC END --><p>
<a id="sec:entrez_examples"></a></p>
<!--TOC subsection id="sec136" PubMed and Medline-->
<h3 id="sec136" class="subsection">9.14.1&#XA0;&#XA0;PubMed and Medline</h3><!--SEC END --><p>
<a id="subsec:pub_med"></a></p><p>If you are in the medical field or interested in human issues (and many times even if you are not!), PubMed (<a href="http://www.ncbi.nlm.nih.gov/PubMed/"><span style="font-family:monospace">http://www.ncbi.nlm.nih.gov/PubMed/</span></a>) is an excellent source of all kinds of goodies. So like other things, we&#X2019;d like to be able to grab information from it and use it in Python scripts.</p><p>In this example, we will query PubMed for all articles having to do with orchids (see section&#XA0;<a href="#sec%3Aorchids">2.3</a> for our motivation). We first check how many of such articles there are:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
&gt;&gt;&gt; handle = Entrez.egquery(term="orchid")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; for row in record["eGQueryResult"]:
...     if row["DbName"]=="pubmed":
...         print(row["Count"])
463
</pre><p>Now we use the <code>Bio.Entrez.efetch</code> function to download the PubMed IDs of these 463 articles:
</p><pre class="verbatim">&gt;&gt;&gt; handle = Entrez.esearch(db="pubmed", term="orchid", retmax=463)
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; idlist = record["IdList"]
&gt;&gt;&gt; print(idlist)
</pre><p>This returns a Python list containing all of the PubMed IDs of articles related to orchids:
</p><pre class="verbatim">['18680603', '18665331', '18661158', '18627489', '18627452', '18612381',
'18594007', '18591784', '18589523', '18579475', '18575811', '18575690',
...
</pre><p>Now that we&#X2019;ve got them, we obviously want to get the corresponding Medline records and extract the information from them. Here, we&#X2019;ll download the Medline records in the Medline flat-file format, and use the <code>Bio.Medline</code> module to parse them:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Medline
&gt;&gt;&gt; handle = Entrez.efetch(db="pubmed", id=idlist, rettype="medline",
                           retmode="text")
&gt;&gt;&gt; records = Medline.parse(handle)
</pre><p>NOTE - We&#X2019;ve just done a separate search and fetch here, the NCBI much prefer you to take advantage of their history support in this situation. See Section&#XA0;<a href="#sec%3Aentrez-webenv">9.15</a>.</p><p>Keep in mind that <code>records</code> is an iterator, so you can iterate through the records only once. If you want to save the records, you can convert them to a list:
</p><pre class="verbatim">&gt;&gt;&gt; records = list(records)
</pre><p>Let&#X2019;s now iterate over the records to print out some information about each record:
</p><pre class="verbatim">&gt;&gt;&gt; for record in records:
...     print("title:", record.get("TI", "?"))
...     print("authors:", record.get("AU", "?"))
...     print("source:", record.get("SO", "?"))
...     print("")
</pre><p>The output for this looks like:
</p><pre class="verbatim">title: Sex pheromone mimicry in the early spider orchid (ophrys sphegodes):
patterns of hydrocarbons as the key mechanism for pollination by sexual
deception [In Process Citation]
authors: ['Schiestl FP', 'Ayasse M', 'Paulus HF', 'Lofstedt C', 'Hansson BS',
'Ibarra F', 'Francke W']
source: J Comp Physiol [A] 2000 Jun;186(6):567-74
</pre><p>Especially interesting to note is the list of authors, which is returned as a standard Python list. This makes it easy to manipulate and search using standard Python tools. For instance, we could loop through a whole bunch of entries searching for a particular author with code like the following:
</p><pre class="verbatim">&gt;&gt;&gt; search_author = "Waits T"

&gt;&gt;&gt; for record in records:
...     if not "AU" in record:
...         continue
...     if search_author in record["AU"]:
...         print("Author %s found: %s" % (search_author, record["SO"]))
</pre><p>Hopefully this section gave you an idea of the power and flexibility of the Entrez and Medline interfaces and how they can be used together.</p>
<!--TOC subsection id="sec137" Searching, downloading, and parsing Entrez Nucleotide records-->
<h3 id="sec137" class="subsection">9.14.2&#XA0;&#XA0;Searching, downloading, and parsing Entrez Nucleotide records</h3><!--SEC END --><p>
<a id="subsec:entrez_example_genbank"></a></p><p>Here we&#X2019;ll show a simple example of performing a remote Entrez query. In section&#XA0;<a href="#sec%3Aorchids">2.3</a> of the parsing examples, we talked about using NCBI&#X2019;s Entrez website to search the NCBI nucleotide databases for info on Cypripedioideae, our friends the lady slipper orchids. Now, we&#X2019;ll look at how to automate that process using a Python script. In this example, we&#X2019;ll just show how to connect, get the results, and parse them, with the Entrez module doing all of the work.</p><p>First, we use EGQuery to find out the number of results we will get before actually downloading them. EGQuery will tell us how many search results were found in each of the databases, but for this example we are only interested in nucleotides:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
&gt;&gt;&gt; handle = Entrez.egquery(term="Cypripedioideae")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; for row in record["eGQueryResult"]:
...     if row["DbName"]=="nuccore":
...         print(row["Count"])
814
</pre><p>So, we expect to find 814 Entrez Nucleotide records (this is the number I obtained in 2008; it is likely to increase in the future). If you find some ridiculously high number of hits, you may want to reconsider if you really want to download all of them, which is our next step:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; handle = Entrez.esearch(db="nucleotide", term="Cypripedioideae", retmax=814)
&gt;&gt;&gt; record = Entrez.read(handle)
</pre><p>Here, <code>record</code> is a Python dictionary containing the search results and some auxiliary information. Just for information, let&#X2019;s look at what is stored in this dictionary:
</p><pre class="verbatim">&gt;&gt;&gt; print(record.keys())
[u'Count', u'RetMax', u'IdList', u'TranslationSet', u'RetStart', u'QueryTranslation']
</pre><p>First, let&#X2019;s check how many results were found:
</p><pre class="verbatim">&gt;&gt;&gt; print(record["Count"])
'814'
</pre><p>which is the number we expected. The 814 results are stored in <code>record['IdList']</code>:
</p><pre class="verbatim">&gt;&gt;&gt; len(record["IdList"])
814
</pre><p>Let&#X2019;s look at the first five results:
</p><pre class="verbatim">&gt;&gt;&gt; record["IdList"][:5]
['187237168', '187372713', '187372690', '187372688', '187372686']
</pre><p><a id="sec:entrez-batched-efetch"></a>
We can download these records using <code>efetch</code>.
While you could download these records one by one, to reduce the load on NCBI&#X2019;s servers, it is better to fetch a bunch of records at the same time, shown below.
However, in this situation you should ideally be using the history feature described later in Section&#XA0;<a href="#sec%3Aentrez-webenv">9.15</a>.</p><pre class="verbatim">&gt;&gt;&gt; idlist = ",".join(record["IdList"][:5])
&gt;&gt;&gt; print(idlist)
187237168,187372713,187372690,187372688,187372686
&gt;&gt;&gt; handle = Entrez.efetch(db="nucleotide", id=idlist, retmode="xml")
&gt;&gt;&gt; records = Entrez.read(handle)
&gt;&gt;&gt; len(records)
5
</pre><p>Each of these records corresponds to one GenBank record.
</p><pre class="verbatim">&gt;&gt;&gt; print(records[0].keys())
[u'GBSeq_moltype', u'GBSeq_source', u'GBSeq_sequence',
 u'GBSeq_primary-accession', u'GBSeq_definition', u'GBSeq_accession-version',
 u'GBSeq_topology', u'GBSeq_length', u'GBSeq_feature-table',
 u'GBSeq_create-date', u'GBSeq_other-seqids', u'GBSeq_division',
 u'GBSeq_taxonomy', u'GBSeq_references', u'GBSeq_update-date',
 u'GBSeq_organism', u'GBSeq_locus', u'GBSeq_strandedness']

&gt;&gt;&gt; print(records[0]["GBSeq_primary-accession"])
DQ110336

&gt;&gt;&gt; print(records[0]["GBSeq_other-seqids"])
['gb|DQ110336.1|', 'gi|187237168']

&gt;&gt;&gt; print(records[0]["GBSeq_definition"])
Cypripedium calceolus voucher Davis 03-03 A maturase (matR) gene, partial cds;
mitochondrial

&gt;&gt;&gt; print(records[0]["GBSeq_organism"])
Cypripedium calceolus
</pre><p>You could use this to quickly set up searches &#X2013; but for heavy usage, see Section&#XA0;<a href="#sec%3Aentrez-webenv">9.15</a>.</p>
<!--TOC subsection id="sec138" Searching, downloading, and parsing GenBank records-->
<h3 id="sec138" class="subsection">9.14.3&#XA0;&#XA0;Searching, downloading, and parsing GenBank records</h3><!--SEC END --><p>
<a id="sec:entrez-search-fetch-genbank"></a></p><p>The GenBank record format is a very popular method of holding information about sequences, sequence features, and other associated sequence information. The format is a good way to get information from the NCBI databases at <a href="http://www.ncbi.nlm.nih.gov/"><span style="font-family:monospace">http://www.ncbi.nlm.nih.gov/</span></a>.</p><p>In this example we&#X2019;ll show how to query the NCBI databases,to retrieve the records from the query, and then parse them using <code>Bio.SeqIO</code> - something touched on in Section&#XA0;<a href="#sec%3ASeqIO_GenBank_Online">5.3.1</a>.
For simplicity, this example <em>does not</em> take advantage of the WebEnv history feature &#X2013; see Section&#XA0;<a href="#sec%3Aentrez-webenv">9.15</a> for this.</p><p>First, we want to make a query and find out the ids of the records to retrieve. Here we&#X2019;ll do a quick search for one of our favorite organisms, <em>Opuntia</em> (prickly-pear cacti). We can do quick search and get back the GIs (GenBank identifiers) for all of the corresponding records. First we check how many records there are:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
&gt;&gt;&gt; handle = Entrez.egquery(term="Opuntia AND rpl16")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; for row in record["eGQueryResult"]:
...     if row["DbName"]=="nuccore":
...         print(row["Count"])
...
9
</pre><p>Now we download the list of GenBank identifiers:
</p><pre class="verbatim">&gt;&gt;&gt; handle = Entrez.esearch(db="nuccore", term="Opuntia AND rpl16")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; gi_list = record["IdList"]
&gt;&gt;&gt; gi_list
['57240072', '57240071', '6273287', '6273291', '6273290', '6273289', '6273286',
'6273285', '6273284']
</pre><p>Now we use these GIs to download the GenBank records - note that with older versions of Biopython you had to supply a comma separated list of GI numbers to Entrez, as of Biopython 1.59 you can pass a list and this is converted for you:</p><pre class="verbatim">&gt;&gt;&gt; gi_str = ",".join(gi_list)
&gt;&gt;&gt; handle = Entrez.efetch(db="nuccore", id=gi_str, rettype="gb", retmode="text")
</pre><p>If you want to look at the raw GenBank files, you can read from this handle and print out the result:</p><pre class="verbatim">&gt;&gt;&gt; text = handle.read()
&gt;&gt;&gt; print(text)
LOCUS       AY851612                 892 bp    DNA     linear   PLN 10-APR-2007
DEFINITION  Opuntia subulata rpl16 gene, intron; chloroplast.
ACCESSION   AY851612
VERSION     AY851612.1  GI:57240072
KEYWORDS    .
SOURCE      chloroplast Austrocylindropuntia subulata
  ORGANISM  Austrocylindropuntia subulata
            Eukaryota; Viridiplantae; Streptophyta; Embryophyta; Tracheophyta;
            Spermatophyta; Magnoliophyta; eudicotyledons; core eudicotyledons;
            Caryophyllales; Cactaceae; Opuntioideae; Austrocylindropuntia.
REFERENCE   1  (bases 1 to 892)
  AUTHORS   Butterworth,C.A. and Wallace,R.S.
...
</pre><p>In this case, we are just getting the raw records. To get the records in a more Python-friendly form, we can use <code>Bio.SeqIO</code> to parse the GenBank data into <code>SeqRecord</code> objects, including <code>SeqFeature</code> objects (see Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>):</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; handle = Entrez.efetch(db="nuccore", id=gi_str, rettype="gb", retmode="text")
&gt;&gt;&gt; records = SeqIO.parse(handle, "gb")
</pre><p>We can now step through the records and look at the information we are interested in:
</p><pre class="verbatim">&gt;&gt;&gt; for record in records: 
&gt;&gt;&gt; ...    print("%s, length %i, with %i features" \
&gt;&gt;&gt; ...           % (record.name, len(record), len(record.features)))
AY851612, length 892, with 3 features
AY851611, length 881, with 3 features
AF191661, length 895, with 3 features
AF191665, length 902, with 3 features
AF191664, length 899, with 3 features
AF191663, length 899, with 3 features
AF191660, length 893, with 3 features
AF191659, length 894, with 3 features
AF191658, length 896, with 3 features
</pre><p>Using these automated query retrieval functionality is a big plus over doing things by hand. Although the module should obey the NCBI&#X2019;s max three queries per second rule, the NCBI have other recommendations like avoiding peak hours. See Section&#XA0;<a href="#sec%3Aentrez-guidelines">9.1</a>.
In particular, please note that for simplicity, this example does not use the WebEnv history feature. You should use this for any non-trivial search and download work, see Section&#XA0;<a href="#sec%3Aentrez-webenv">9.15</a>.</p><p>Finally, if plan to repeat your analysis, rather than downloading the files from the NCBI and parsing them immediately (as shown in this example), you should just download the records <em>once</em> and save them to your hard disk, and then parse the local file.</p>
<!--TOC subsection id="sec139" Finding the lineage of an organism-->
<h3 id="sec139" class="subsection">9.14.4&#XA0;&#XA0;Finding the lineage of an organism</h3><!--SEC END --><p>Staying with a plant example, let&#X2019;s now find the lineage of the Cypripedioideae orchid family. First, we search the Taxonomy database for Cypripedioideae, which yields exactly one NCBI taxonomy identifier:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"     # Always tell NCBI who you are
&gt;&gt;&gt; handle = Entrez.esearch(db="Taxonomy", term="Cypripedioideae")
&gt;&gt;&gt; record = Entrez.read(handle)
&gt;&gt;&gt; record["IdList"]
['158330']
&gt;&gt;&gt; record["IdList"][0]
'158330'
</pre><p>Now, we use <code>efetch</code> to download this entry in the Taxonomy database, and then parse it:
</p><pre class="verbatim">&gt;&gt;&gt; handle = Entrez.efetch(db="Taxonomy", id="158330", retmode="xml")
&gt;&gt;&gt; records = Entrez.read(handle)
</pre><p>Again, this record stores lots of information:
</p><pre class="verbatim">&gt;&gt;&gt; records[0].keys()
[u'Lineage', u'Division', u'ParentTaxId', u'PubDate', u'LineageEx',
 u'CreateDate', u'TaxId', u'Rank', u'GeneticCode', u'ScientificName',
 u'MitoGeneticCode', u'UpdateDate']
</pre><p>We can get the lineage directly from this record:
</p><pre class="verbatim">&gt;&gt;&gt; records[0]["Lineage"]
'cellular organisms; Eukaryota; Viridiplantae; Streptophyta; Streptophytina;
 Embryophyta; Tracheophyta; Euphyllophyta; Spermatophyta; Magnoliophyta;
 Liliopsida; Asparagales; Orchidaceae'
</pre><p>The record data contains much more than just the information shown here - for example look under <span style="font-family:monospace">"LineageEx"</span> instead of <span style="font-family:monospace">"Lineage"</span> and you&#X2019;ll get the NCBI taxon identifiers of the lineage entries too.</p>
<!--TOC section id="sec140" Using the history and WebEnv-->
<h2 id="sec140" class="section">9.15&#XA0;&#XA0;Using the history and WebEnv</h2><!--SEC END --><p>
<a id="sec:entrez-webenv"></a></p><p>Often you will want to make a series of linked queries. Most typically,
running a search, perhaps refining the search, and then retrieving detailed
search results. You <em>can</em> do this by making a series of separate calls
to Entrez. However, the NCBI prefer you to take advantage of their history
support - for example combining ESearch and EFetch.</p><p>Another typical use of the history support would be to combine EPost and
EFetch. You use EPost to upload a list of identifiers, which starts a new
history session. You then download the records with EFetch by referring
to the session (instead of the identifiers).</p>
<!--TOC subsection id="sec141" Searching for and downloading sequences using the history-->
<h3 id="sec141" class="subsection">9.15.1&#XA0;&#XA0;Searching for and downloading sequences using the history</h3><!--SEC END --><p>
Suppose we want to search and download all the <span style="font-style:italic">Opuntia</span> rpl16
nucleotide sequences, and store them in a FASTA file. As shown in
Section&#XA0;<a href="#sec%3Aentrez-search-fetch-genbank">9.14.3</a>, we can naively combine
<code>Bio.Entrez.esearch()</code> to get a list of GI numbers, and then call
<code>Bio.Entrez.efetch()</code> to download them all.</p><p>However, the approved approach is to run the search with the history
feature. Then, we can fetch the results by reference to the search
results - which the NCBI can anticipate and cache.</p><p>To do this, call <code>Bio.Entrez.esearch()</code> as normal, but with the
additional argument of <code>usehistory="y"</code>,</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "history.user@example.com"
&gt;&gt;&gt; search_handle = Entrez.esearch(db="nucleotide",term="Opuntia[orgn] and rpl16",
                                   usehistory="y")
&gt;&gt;&gt; search_results = Entrez.read(search_handle)
&gt;&gt;&gt; search_handle.close()
</pre><p>When you get the XML output back, it will still include the usual search results:</p><pre class="verbatim">&gt;&gt;&gt; gi_list = search_results["IdList"]
&gt;&gt;&gt; count = int(search_results["Count"])
&gt;&gt;&gt; assert count == len(gi_list)
</pre><p>However, you also get given two additional pieces of information, the <span style="font-family:monospace">WebEnv</span> session cookie, and the <span style="font-family:monospace">QueryKey</span>:</p><pre class="verbatim">&gt;&gt;&gt; webenv = search_results["WebEnv"]
&gt;&gt;&gt; query_key = search_results["QueryKey"] 
</pre><p>Having stored these values in variables <span style="font-family:monospace">session_cookie</span> and <span style="font-family:monospace">query_key</span> we can use them as parameters to <code>Bio.Entrez.efetch()</code> instead of giving the GI numbers as identifiers. </p><p>While for small searches you might be OK downloading everything at once, it is better to download in batches. You use the <span style="font-family:monospace">retstart</span> and <span style="font-family:monospace">retmax</span> parameters to specify which range of search results you want returned (starting entry using zero-based counting, and maximum number of results to return). For example,</p><pre class="verbatim">batch_size = 3
out_handle = open("orchid_rpl16.fasta", "w")
for start in range(0,count,batch_size):
    end = min(count, start+batch_size)
    print("Going to download record %i to %i" % (start+1, end))
    fetch_handle = Entrez.efetch(db="nucleotide", rettype="fasta", retmode="text",
                                 retstart=start, retmax=batch_size,
                                 webenv=webenv, query_key=query_key)
    data = fetch_handle.read()
    fetch_handle.close()
    out_handle.write(data)
out_handle.close()
</pre><p>For illustrative purposes, this example downloaded the FASTA records in batches of three. Unless you are downloading genomes or chromosomes, you would normally pick a larger batch size.</p>
<!--TOC subsection id="sec142" Searching for and downloading abstracts using the history-->
<h3 id="sec142" class="subsection">9.15.2&#XA0;&#XA0;Searching for and downloading abstracts using the history</h3><!--SEC END --><p>
Here is another history example, searching for papers published in the last year about the <span style="font-style:italic">Opuntia</span>, and then downloading them into a file in MedLine format:</p><pre class="verbatim">from Bio import Entrez
Entrez.email = "history.user@example.com"
search_results = Entrez.read(Entrez.esearch(db="pubmed",
                                            term="Opuntia[ORGN]",
                                            reldate=365, datetype="pdat",
                                            usehistory="y"))
count = int(search_results["Count"])
print("Found %i results" % count)

batch_size = 10
out_handle = open("recent_orchid_papers.txt", "w")
for start in range(0,count,batch_size):
    end = min(count, start+batch_size)
    print("Going to download record %i to %i" % (start+1, end))
    fetch_handle = Entrez.efetch(db="pubmed",
                                 rettype="medline", retmode="text",
                                 retstart=start, retmax=batch_size,
                                 webenv=search_results["WebEnv"],
                                 query_key=search_results["QueryKey"])
    data = fetch_handle.read()
    fetch_handle.close()
    out_handle.write(data)
out_handle.close()
</pre><p>At the time of writing, this gave 28 matches - but because this is a date dependent search, this will of course vary. As described in Section&#XA0;<a href="#subsec%3Aentrez-and-medline">9.12.1</a> above, you can then use <code>Bio.Medline</code> to parse the saved records.</p>
<!--TOC subsection id="sec143" Searching for citations-->
<h3 id="sec143" class="subsection">9.15.3&#XA0;&#XA0;Searching for citations</h3><!--SEC END --><p>
<a id="sec:elink-citations"></a></p><p>Back in Section&#XA0;<a href="#sec%3Aelink">9.7</a> we mentioned ELink can be used to search for citations of a given paper.
Unfortunately this only covers journals indexed for PubMed Central
(doing it for all the journals in PubMed would mean a lot more work for the NIH).
Let&#X2019;s try this for the Biopython PDB parser paper, PubMed ID 14630660:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Entrez
&gt;&gt;&gt; Entrez.email = "A.N.Other@example.com"
&gt;&gt;&gt; pmid = "14630660"
&gt;&gt;&gt; results = Entrez.read(Entrez.elink(dbfrom="pubmed", db="pmc",
...                                    LinkName="pubmed_pmc_refs", from_uid=pmid))
&gt;&gt;&gt; pmc_ids = [link["Id"] for link in results[0]["LinkSetDb"][0]["Link"]]
&gt;&gt;&gt; pmc_ids
['2744707', '2705363', '2682512', ..., '1190160']
</pre><p>Great - eleven articles. But why hasn&#X2019;t the Biopython application note been
found (PubMed ID 19304878)? Well, as you might have guessed from the variable
names, there are not actually PubMed IDs, but PubMed Central IDs. Our
application note is the third citing paper in that list, PMCID 2682512.</p><p>So, what if (like me) you&#X2019;d rather get back a list of PubMed IDs? Well we
can call ELink again to translate them. This becomes a two step process,
so by now you should expect to use the history feature to accomplish it
(Section&#XA0;<a href="#sec%3Aentrez-webenv">9.15</a>).</p><p>But first, taking the more straightforward approach of making a second
(separate) call to ELink:</p><pre class="verbatim">&gt;&gt;&gt; results2 = Entrez.read(Entrez.elink(dbfrom="pmc", db="pubmed", LinkName="pmc_pubmed",
...                                     from_uid=",".join(pmc_ids)))
&gt;&gt;&gt; pubmed_ids = [link["Id"] for link in results2[0]["LinkSetDb"][0]["Link"]]
&gt;&gt;&gt; pubmed_ids
['19698094', '19450287', '19304878', ..., '15985178']
</pre><p>This time you can immediately spot the Biopython application note
as the third hit (PubMed ID 19304878).</p><p>Now, let&#X2019;s do that all again but with the history &#X2026;<span style="font-style:italic">TODO</span>.</p><p>And finally, don&#X2019;t forget to include your <em>own</em> email address in the Entrez calls.</p>
<!--TOC chapter id="sec144" Swiss-Prot and ExPASy-->
<h1 id="sec144" class="chapter">Chapter&#XA0;10&#XA0;&#XA0;Swiss-Prot and ExPASy</h1><!--SEC END --><p>
<a id="chapter:swiss_prot"></a></p>
<!--TOC section id="sec145" Parsing Swiss-Prot files-->
<h2 id="sec145" class="section">10.1&#XA0;&#XA0;Parsing Swiss-Prot files</h2><!--SEC END --><p>Swiss-Prot (<a href="http://www.expasy.org/sprot"><span style="font-family:monospace">http://www.expasy.org/sprot</span></a>) is a hand-curated database of protein sequences. Biopython can parse the &#X201C;plain text&#X201D; Swiss-Prot file format, which is still used for the UniProt Knowledgebase which combined Swiss-Prot, TrEMBL and PIR-PSD. We do not (yet) support the UniProtKB XML file format.</p>
<!--TOC subsection id="sec146" Parsing Swiss-Prot records-->
<h3 id="sec146" class="subsection">10.1.1&#XA0;&#XA0;Parsing Swiss-Prot records</h3><!--SEC END --><p>In Section&#XA0;<a href="#sec%3ASeqIO_ExPASy_and_SwissProt">5.3.2</a>, we described how to extract the sequence of a Swiss-Prot record as a <code>SeqRecord</code> object. Alternatively, you can store the Swiss-Prot record in a <code>Bio.SwissProt.Record</code> object, which in fact stores the complete information contained in the Swiss-Prot record. In this Section, we describe how to extract <code>Bio.SwissProt.Record</code> objects from a Swiss-Prot file.</p><p>To parse a Swiss-Prot record, we first get a handle to a Swiss-Prot record. There are several ways to do so, depending on where and how the Swiss-Prot record is stored:
</p><ul class="itemize"><li class="li-itemize">
Open a Swiss-Prot file locally: <br>
<code>&gt;&gt;&gt; handle = open("myswissprotfile.dat")</code>
</li><li class="li-itemize">Open a gzipped Swiss-Prot file:
<pre class="verbatim">&gt;&gt;&gt; import gzip
&gt;&gt;&gt; handle = gzip.open("myswissprotfile.dat.gz")
</pre></li><li class="li-itemize">Open a Swiss-Prot file over the internet:
<pre class="verbatim">&gt;&gt;&gt; import urllib
&gt;&gt;&gt; handle = urllib.urlopen("http://www.somelocation.org/data/someswissprotfile.dat")
</pre></li><li class="li-itemize">Open a Swiss-Prot file over the internet from the ExPASy database
(see section <a href="#subsec%3Aexpasy_swissprot">10.5.1</a>):
<pre class="verbatim">&gt;&gt;&gt; from Bio import ExPASy
&gt;&gt;&gt; handle = ExPASy.get_sprot_raw(myaccessionnumber)
</pre></li></ul><p>
The key point is that for the parser, it doesn&#X2019;t matter how the handle was created, as long as it points to data in the Swiss-Prot format.</p><p>We can use <code>Bio.SeqIO</code> as described in Section&#XA0;<a href="#sec%3ASeqIO_ExPASy_and_SwissProt">5.3.2</a> to get file format agnostic <code>SeqRecord</code> objects. Alternatively, we can use <code>Bio.SwissProt</code> get <code>Bio.SwissProt.Record</code> objects, which are a much closer match to the underlying file format.</p><p>To read one Swiss-Prot record from the handle, we use the function <code>read()</code>:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SwissProt
&gt;&gt;&gt; record = SwissProt.read(handle)
</pre><p>This function should be used if the handle points to exactly one Swiss-Prot record. It raises a <code>ValueError</code> if no Swiss-Prot record was found, and also if more than one record was found.</p><p>We can now print out some information about this record:
</p><pre class="verbatim">&gt;&gt;&gt; print(record.description)
'RecName: Full=Chalcone synthase 3; EC=2.3.1.74; AltName: Full=Naringenin-chalcone synthase 3;'
&gt;&gt;&gt; for ref in record.references:
...     print("authors:", ref.authors)
...     print("title:", ref.title)
... 
authors: Liew C.F., Lim S.H., Loh C.S., Goh C.J.;
title: "Molecular cloning and sequence analysis of chalcone synthase cDNAs of
Bromheadia finlaysoniana.";
&gt;&gt;&gt; print(record.organism_classification)
['Eukaryota', 'Viridiplantae', 'Streptophyta', 'Embryophyta', ..., 'Bromheadia']
</pre><p>To parse a file that contains more than one Swiss-Prot record, we use the <code>parse</code> function instead. This function allows us to iterate over the records in the file.</p><p>For example, let&#X2019;s parse the full Swiss-Prot database and collect all the descriptions.
You can download this from the <a href="ftp://ftp.expasy.org/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.dat.gz">ExPAYs FTP site</a> as a single gzipped-file <code>uniprot_sprot.dat.gz</code> (about 300MB). This is a compressed file containing a single file, <code>uniprot_sprot.dat</code> (over 1.5GB).</p><p>As described at the start of this section, you can use the Python library <code>gzip</code> to open and uncompress a <span style="font-family:monospace">.gz</span> file, like this:</p><pre class="verbatim">&gt;&gt;&gt; import gzip
&gt;&gt;&gt; handle = gzip.open("uniprot_sprot.dat.gz")
</pre><p>However, uncompressing a large file takes time, and each time you open the file for reading in this way, it has to be decompressed on the fly. So, if you can spare the disk space you&#X2019;ll save time in the long run if you first decompress the file to disk, to get the <code>uniprot_sprot.dat</code> file inside. Then you can open the file for reading as usual:</p><pre class="verbatim">&gt;&gt;&gt; handle = open("uniprot_sprot.dat")
</pre><p>As of June 2009, the full Swiss-Prot database downloaded from ExPASy contained 468851 Swiss-Prot records. One concise way to build up a list of the record descriptions is with a list comprehension: 
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SwissProt
&gt;&gt;&gt; handle = open("uniprot_sprot.dat")
&gt;&gt;&gt; descriptions = [record.description for record in SwissProt.parse(handle)]
&gt;&gt;&gt; len(descriptions)
468851
&gt;&gt;&gt; descriptions[:5]
['RecName: Full=Protein MGF 100-1R;',
 'RecName: Full=Protein MGF 100-1R;',
 'RecName: Full=Protein MGF 100-1R;',
 'RecName: Full=Protein MGF 100-1R;',
 'RecName: Full=Protein MGF 100-2L;']

</pre><p>Or, using a for loop over the record iterator:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SwissProt
&gt;&gt;&gt; descriptions = []
&gt;&gt;&gt; handle = open("uniprot_sprot.dat")
&gt;&gt;&gt; for record in SwissProt.parse(handle):
...     descriptions.append(record.description)
...
&gt;&gt;&gt; len(descriptions)
468851
</pre><p>Because this is such a large input file, either way takes about eleven minutes on my new desktop computer (using the uncompressed <code>uniprot_sprot.dat</code> file as input).</p><p>It is equally easy to extract any kind of information you&#X2019;d like from Swiss-Prot records. To see the members of a Swiss-Prot record, use
</p><pre class="verbatim">&gt;&gt;&gt; dir(record)
['__doc__', '__init__', '__module__', 'accessions', 'annotation_update',
'comments', 'created', 'cross_references', 'data_class', 'description',
'entry_name', 'features', 'gene_name', 'host_organism', 'keywords',
'molecule_type', 'organelle', 'organism', 'organism_classification',
'references', 'seqinfo', 'sequence', 'sequence_length',
'sequence_update', 'taxonomy_id']
</pre>
<!--TOC subsection id="sec147" Parsing the Swiss-Prot keyword and category list-->
<h3 id="sec147" class="subsection">10.1.2&#XA0;&#XA0;Parsing the Swiss-Prot keyword and category list</h3><!--SEC END --><p>Swiss-Prot also distributes a file <code>keywlist.txt</code>, which lists the keywords and categories used in Swiss-Prot. The file contains entries in the following form:</p><pre class="verbatim">ID   2Fe-2S.
AC   KW-0001
DE   Protein which contains at least one 2Fe-2S iron-sulfur cluster: 2 iron
DE   atoms complexed to 2 inorganic sulfides and 4 sulfur atoms of
DE   cysteines from the protein.
SY   Fe2S2; [2Fe-2S] cluster; [Fe2S2] cluster; Fe2/S2 (inorganic) cluster;
SY   Di-mu-sulfido-diiron; 2 iron, 2 sulfur cluster binding.
GO   GO:0051537; 2 iron, 2 sulfur cluster binding
HI   Ligand: Iron; Iron-sulfur; 2Fe-2S.
HI   Ligand: Metal-binding; 2Fe-2S.
CA   Ligand.
//
ID   3D-structure.
AC   KW-0002
DE   Protein, or part of a protein, whose three-dimensional structure has
DE   been resolved experimentally (for example by X-ray crystallography or
DE   NMR spectroscopy) and whose coordinates are available in the PDB
DE   database. Can also be used for theoretical models.
HI   Technical term: 3D-structure.
CA   Technical term.
//
ID   3Fe-4S.
...
</pre><p>The entries in this file can be parsed by the <code>parse</code> function in the <code>Bio.SwissProt.KeyWList</code> module. Each entry is then stored as a <code>Bio.SwissProt.KeyWList.Record</code>, which is a Python dictionary.</p><pre class="verbatim">&gt;&gt;&gt; from Bio.SwissProt import KeyWList
&gt;&gt;&gt; handle = open("keywlist.txt")
&gt;&gt;&gt; records = KeyWList.parse(handle)
&gt;&gt;&gt; for record in records:
...     print(record['ID'])
...     print(record['DE'])
</pre><p>This prints
</p><pre class="verbatim">2Fe-2S.
Protein which contains at least one 2Fe-2S iron-sulfur cluster: 2 iron atoms
complexed to 2 inorganic sulfides and 4 sulfur atoms of cysteines from the
protein.
...
</pre>
<!--TOC section id="sec148" Parsing Prosite records-->
<h2 id="sec148" class="section">10.2&#XA0;&#XA0;Parsing Prosite records</h2><!--SEC END --><p>Prosite is a database containing protein domains, protein families, functional sites, as well as the patterns and profiles to recognize them. Prosite was developed in parallel with Swiss-Prot. In Biopython, a Prosite record is represented by the <code>Bio.ExPASy.Prosite.Record</code> class, whose members correspond to the different fields in a Prosite record.</p><p>In general, a Prosite file can contain more than one Prosite records. For example, the full set of Prosite records, which can be downloaded as a single file (<code>prosite.dat</code>) from the <a href="ftp://ftp.expasy.org/databases/prosite/prosite.dat">ExPASy FTP site</a>, contains 2073 records (version 20.24 released on 4 December 2007). To parse such a file, we again make use of an iterator:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.ExPASy import Prosite
&gt;&gt;&gt; handle = open("myprositefile.dat")
&gt;&gt;&gt; records = Prosite.parse(handle)
</pre><p>We can now take the records one at a time and print out some information. For example, using the file containing the complete Prosite database, we&#X2019;d find
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.ExPASy import Prosite
&gt;&gt;&gt; handle = open("prosite.dat")
&gt;&gt;&gt; records = Prosite.parse(handle)
&gt;&gt;&gt; record = next(records)
&gt;&gt;&gt; record.accession
'PS00001'
&gt;&gt;&gt; record.name
'ASN_GLYCOSYLATION'
&gt;&gt;&gt; record.pdoc
'PDOC00001'
&gt;&gt;&gt; record = next(records)
&gt;&gt;&gt; record.accession
'PS00004'
&gt;&gt;&gt; record.name
'CAMP_PHOSPHO_SITE'
&gt;&gt;&gt; record.pdoc
'PDOC00004'
&gt;&gt;&gt; record = next(records)
&gt;&gt;&gt; record.accession
'PS00005'
&gt;&gt;&gt; record.name
'PKC_PHOSPHO_SITE'
&gt;&gt;&gt; record.pdoc
'PDOC00005'
</pre><p>and so on. If you&#X2019;re interested in how many Prosite records there are, you could use
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.ExPASy import Prosite
&gt;&gt;&gt; handle = open("prosite.dat")
&gt;&gt;&gt; records = Prosite.parse(handle)
&gt;&gt;&gt; n = 0
&gt;&gt;&gt; for record in records: n+=1
...
&gt;&gt;&gt; n
2073
</pre><p>To read exactly one Prosite from the handle, you can use the <code>read</code> function:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.ExPASy import Prosite
&gt;&gt;&gt; handle = open("mysingleprositerecord.dat")
&gt;&gt;&gt; record = Prosite.read(handle)
</pre><p>This function raises a ValueError if no Prosite record is found, and also if more than one Prosite record is found.</p>
<!--TOC section id="sec149" Parsing Prosite documentation records-->
<h2 id="sec149" class="section">10.3&#XA0;&#XA0;Parsing Prosite documentation records</h2><!--SEC END --><p>In the Prosite example above, the <code>record.pdoc</code> accession numbers <code>'PDOC00001'</code>, <code>'PDOC00004'</code>, <code>'PDOC00005'</code> and so on refer to Prosite documentation. The Prosite documentation records are available from ExPASy as individual files, and as one file (<code>prosite.doc</code>) containing all Prosite documentation records.</p><p>We use the parser in <code>Bio.ExPASy.Prodoc</code> to parse Prosite documentation records. For example, to create a list of all accession numbers of Prosite documentation record, you can use</p><pre class="verbatim">&gt;&gt;&gt; from Bio.ExPASy import Prodoc
&gt;&gt;&gt; handle = open("prosite.doc")
&gt;&gt;&gt; records = Prodoc.parse(handle)
&gt;&gt;&gt; accessions = [record.accession for record in records]
</pre><p>Again a <code>read()</code> function is provided to read exactly one Prosite documentation record from the handle.</p>
<!--TOC section id="sec150" Parsing Enzyme records-->
<h2 id="sec150" class="section">10.4&#XA0;&#XA0;Parsing Enzyme records</h2><!--SEC END --><p>ExPASy&#X2019;s Enzyme database is a repository of information on enzyme nomenclature. A typical Enzyme record looks as follows:</p><pre class="verbatim">ID   3.1.1.34
DE   Lipoprotein lipase.
AN   Clearing factor lipase.
AN   Diacylglycerol lipase.
AN   Diglyceride lipase.
CA   Triacylglycerol + H(2)O = diacylglycerol + a carboxylate.
CC   -!- Hydrolyzes triacylglycerols in chylomicrons and very low-density
CC       lipoproteins (VLDL).
CC   -!- Also hydrolyzes diacylglycerol.
PR   PROSITE; PDOC00110;
DR   P11151, LIPL_BOVIN ;  P11153, LIPL_CAVPO ;  P11602, LIPL_CHICK ;
DR   P55031, LIPL_FELCA ;  P06858, LIPL_HUMAN ;  P11152, LIPL_MOUSE ;
DR   O46647, LIPL_MUSVI ;  P49060, LIPL_PAPAN ;  P49923, LIPL_PIG   ;
DR   Q06000, LIPL_RAT   ;  Q29524, LIPL_SHEEP ;
//
</pre><p>In this example, the first line shows the EC (Enzyme Commission) number of lipoprotein lipase (second line). Alternative names of lipoprotein lipase are "clearing factor lipase", "diacylglycerol lipase", and "diglyceride lipase" (lines 3 through 5). The line starting with "CA" shows the catalytic activity of this enzyme. Comment lines start with "CC". The "PR" line shows references to the Prosite Documentation records, and the "DR" lines show references to Swiss-Prot records. Not of these entries are necessarily present in an Enzyme record.</p><p>In Biopython, an Enzyme record is represented by the <code>Bio.ExPASy.Enzyme.Record</code> class. This record derives from a Python dictionary and has keys corresponding to the two-letter codes used in Enzyme files. To read an Enzyme file containing one Enzyme record, use the <code>read</code> function in <code>Bio.ExPASy.Enzyme</code>:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.ExPASy import Enzyme
&gt;&gt;&gt; with open("lipoprotein.txt") as handle:
...     record = Enzyme.read(handle)
...
&gt;&gt;&gt; record["ID"]
'3.1.1.34'
&gt;&gt;&gt; record["DE"]
'Lipoprotein lipase.'
&gt;&gt;&gt; record["AN"]
['Clearing factor lipase.', 'Diacylglycerol lipase.', 'Diglyceride lipase.']
&gt;&gt;&gt; record["CA"]
'Triacylglycerol + H(2)O = diacylglycerol + a carboxylate.'
&gt;&gt;&gt; record["PR"]
['PDOC00110']
</pre><pre class="verbatim">&gt;&gt;&gt; record["CC"]
['Hydrolyzes triacylglycerols in chylomicrons and very low-density lipoproteins
(VLDL).', 'Also hydrolyzes diacylglycerol.']
&gt;&gt;&gt; record["DR"]
[['P11151', 'LIPL_BOVIN'], ['P11153', 'LIPL_CAVPO'], ['P11602', 'LIPL_CHICK'],
['P55031', 'LIPL_FELCA'], ['P06858', 'LIPL_HUMAN'], ['P11152', 'LIPL_MOUSE'],
['O46647', 'LIPL_MUSVI'], ['P49060', 'LIPL_PAPAN'], ['P49923', 'LIPL_PIG'],
['Q06000', 'LIPL_RAT'], ['Q29524', 'LIPL_SHEEP']]
</pre><p>The <code>read</code> function raises a ValueError if no Enzyme record is found, and also if more than one Enzyme record is found.</p><p>The full set of Enzyme records can be downloaded as a single file (<code>enzyme.dat</code>) from the <a href="ftp://ftp.expasy.org/databases/enzyme/enzyme.dat">ExPASy FTP site</a>, containing 4877 records (release of 3 March 2009). To parse such a file containing multiple Enzyme records, use the <code>parse</code> function in <code>Bio.ExPASy.Enzyme</code> to obtain an iterator:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.ExPASy import Enzyme
&gt;&gt;&gt; handle = open("enzyme.dat")
&gt;&gt;&gt; records = Enzyme.parse(handle)
</pre><p>We can now iterate over the records one at a time. For example, we can make a list of all EC numbers for which an Enzyme record is available:
</p><pre class="verbatim">&gt;&gt;&gt; ecnumbers = [record["ID"] for record in records]
</pre>
<!--TOC section id="sec151" Accessing the ExPASy server-->
<h2 id="sec151" class="section">10.5&#XA0;&#XA0;Accessing the ExPASy server</h2><!--SEC END --><p>Swiss-Prot, Prosite, and Prosite documentation records can be downloaded from the ExPASy web server at <a href="http://www.expasy.org"><span style="font-family:monospace">http://www.expasy.org</span></a>. Six kinds of queries are available from ExPASy:
</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold">get_prodoc_entry</span></dt><dd class="dd-description">To download a Prosite documentation record in HTML format
</dd><dt class="dt-description"><span style="font-weight:bold">get_prosite_entry</span></dt><dd class="dd-description">To download a Prosite record in HTML format
</dd><dt class="dt-description"><span style="font-weight:bold">get_prosite_raw</span></dt><dd class="dd-description">To download a Prosite or Prosite documentation record in raw format
</dd><dt class="dt-description"><span style="font-weight:bold">get_sprot_raw</span></dt><dd class="dd-description">To download a Swiss-Prot record in raw format
</dd><dt class="dt-description"><span style="font-weight:bold">sprot_search_ful</span></dt><dd class="dd-description">To search for a Swiss-Prot record
</dd><dt class="dt-description"><span style="font-weight:bold">sprot_search_de</span></dt><dd class="dd-description">To search for a Swiss-Prot record
</dd></dl><p>
To access this web server from a Python script, we use the <code>Bio.ExPASy</code> module.</p>
<!--TOC subsection id="sec152" Retrieving a Swiss-Prot record-->
<h3 id="sec152" class="subsection">10.5.1&#XA0;&#XA0;Retrieving a Swiss-Prot record</h3><!--SEC END --><p>
<a id="subsec:expasy_swissprot"></a></p><p>Let&#X2019;s say we are looking at chalcone synthases for Orchids (see section&#XA0;<a href="#sec%3Aorchids">2.3</a> for some justification for looking for interesting things about orchids). Chalcone synthase is involved in flavanoid biosynthesis in plants, and flavanoids make lots of cool things like pigment colors and UV protectants. </p><p>If you do a search on Swiss-Prot, you can find three orchid proteins for Chalcone Synthase, id numbers O23729, O23730, O23731. Now, let&#X2019;s write a script which grabs these, and parses out some interesting information.</p><p>First, we grab the records, using the <code>get_sprot_raw()</code> function of <code>Bio.ExPASy</code>. This function is very nice since you can feed it an id and get back a handle to a raw text record (no html to mess with!). We can the use <code>Bio.SwissProt.read</code> to pull out the Swiss-Prot record, or <code>Bio.SeqIO.read</code> to get a SeqRecord. The following code accomplishes what I just wrote:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import ExPASy
&gt;&gt;&gt; from Bio import SwissProt

&gt;&gt;&gt; accessions = ["O23729", "O23730", "O23731"]
&gt;&gt;&gt; records = []

&gt;&gt;&gt; for accession in accessions:
...     handle = ExPASy.get_sprot_raw(accession)
...     record = SwissProt.read(handle)
...     records.append(record)
</pre><p>If the accession number you provided to <code>ExPASy.get_sprot_raw</code> does not exist, then <code>SwissProt.read(handle)</code> will raise a <code>ValueError</code>. You can catch <code>ValueException</code> exceptions to detect invalid accession numbers:</p><pre class="verbatim">&gt;&gt;&gt; for accession in accessions:
...     handle = ExPASy.get_sprot_raw(accession)
...     try:
...         record = SwissProt.read(handle)
...     except ValueException:
...         print("WARNING: Accession %s not found" % accession)
...     records.append(record)
</pre>
<!--TOC subsection id="sec153" Searching Swiss-Prot-->
<h3 id="sec153" class="subsection">10.5.2&#XA0;&#XA0;Searching Swiss-Prot</h3><!--SEC END --><p>Now, you may remark that I knew the records&#X2019; accession numbers
beforehand. Indeed, <code>get_sprot_raw()</code> needs either the entry name
or an accession number. When you don&#X2019;t have them handy, you can use
one of the <code>sprot_search_de()</code> or <code>sprot_search_ful()</code>
functions.</p><p><code>sprot_search_de()</code> searches in the ID, DE, GN, OS and OG lines;
<code>sprot_search_ful()</code> searches in (nearly) all the fields. They
are detailed on
<a href="http://www.expasy.org/cgi-bin/sprot-search-de"><span style="font-family:monospace">http://www.expasy.org/cgi-bin/sprot-search-de</span></a> and
<a href="http://www.expasy.org/cgi-bin/sprot-search-ful"><span style="font-family:monospace">http://www.expasy.org/cgi-bin/sprot-search-ful</span></a>
respectively. Note that they don&#X2019;t search in TrEMBL by default
(argument <code>trembl</code>). Note also that they return html pages;
however, accession numbers are quite easily extractable:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import ExPASy
&gt;&gt;&gt; import re

&gt;&gt;&gt; handle = ExPASy.sprot_search_de("Orchid Chalcone Synthase")
&gt;&gt;&gt; # or:
&gt;&gt;&gt; # handle = ExPASy.sprot_search_ful("Orchid and {Chalcone Synthase}")
&gt;&gt;&gt; html_results = handle.read()
&gt;&gt;&gt; if "Number of sequences found" in html_results:
...     ids = re.findall(r'HREF="/uniprot/(\w+)"', html_results)
... else:
...     ids = re.findall(r'href="/cgi-bin/niceprot\.pl\?(\w+)"', html_results)
</pre>
<!--TOC subsection id="sec154" Retrieving Prosite and Prosite documentation records-->
<h3 id="sec154" class="subsection">10.5.3&#XA0;&#XA0;Retrieving Prosite and Prosite documentation records</h3><!--SEC END --><p>Prosite and Prosite documentation records can be retrieved either in HTML format, or in raw format. To parse Prosite and Prosite documentation records with Biopython, you should retrieve the records in raw format. For other purposes, however, you may be interested in these records in HTML format.</p><p>To retrieve a Prosite or Prosite documentation record in raw format, use <code>get_prosite_raw()</code>. For example, to download a Prosite record and print it out in raw text format, use</p><pre class="verbatim">&gt;&gt;&gt; from Bio import ExPASy
&gt;&gt;&gt; handle = ExPASy.get_prosite_raw('PS00001')
&gt;&gt;&gt; text = handle.read()
&gt;&gt;&gt; print(text)
</pre><p>To retrieve a Prosite record and parse it into a <code>Bio.Prosite.Record</code> object, use</p><pre class="verbatim">&gt;&gt;&gt; from Bio import ExPASy
&gt;&gt;&gt; from Bio import Prosite
&gt;&gt;&gt; handle = ExPASy.get_prosite_raw('PS00001')
&gt;&gt;&gt; record = Prosite.read(handle)
</pre><p>The same function can be used to retrieve a Prosite documentation record and parse it into a <code>Bio.ExPASy.Prodoc.Record</code> object:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import ExPASy
&gt;&gt;&gt; from Bio.ExPASy import Prodoc
&gt;&gt;&gt; handle = ExPASy.get_prosite_raw('PDOC00001')
&gt;&gt;&gt; record = Prodoc.read(handle)
</pre><p>For non-existing accession numbers, <code>ExPASy.get_prosite_raw</code> returns a handle to an emptry string. When faced with an empty string, <code>Prosite.read</code> and <code>Prodoc.read</code> will raise a ValueError. You can catch these exceptions to detect invalid accession numbers.</p><p>The functions <code>get_prosite_entry()</code> and <code>get_prodoc_entry()</code> are used to download Prosite and Prosite documentation records in HTML format. To create a web page showing one Prosite record, you can use</p><pre class="verbatim">&gt;&gt;&gt; from Bio import ExPASy
&gt;&gt;&gt; handle = ExPASy.get_prosite_entry('PS00001')
&gt;&gt;&gt; html = handle.read()
&gt;&gt;&gt; output = open("myprositerecord.html", "w")
&gt;&gt;&gt; output.write(html)
&gt;&gt;&gt; output.close()
</pre><p>and similarly for a Prosite documentation record:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import ExPASy
&gt;&gt;&gt; handle = ExPASy.get_prodoc_entry('PDOC00001')
&gt;&gt;&gt; html = handle.read()
&gt;&gt;&gt; output = open("myprodocrecord.html", "w")
&gt;&gt;&gt; output.write(html)
&gt;&gt;&gt; output.close()
</pre><p>For these functions, an invalid accession number returns an error message in HTML format.</p>
<!--TOC section id="sec155" Scanning the Prosite database-->
<h2 id="sec155" class="section">10.6&#XA0;&#XA0;Scanning the Prosite database</h2><!--SEC END --><p><a href="http://www.expasy.org/tools/scanprosite/">ScanProsite</a> allows you to scan protein sequences online against the Prosite database by providing a UniProt or PDB sequence identifier or the sequence itself. For more information about ScanProsite, please see the <a href="http://www.expasy.org/tools/scanprosite/scanprosite-doc.html">ScanProsite documentation</a> as well as the <a href="http://www.expasy.org/tools/scanprosite/ScanPrositeREST.html">documentation for programmatic access of ScanProsite</a>.</p><p>You can use Biopython&#X2019;s <code>Bio.ExPASy.ScanProsite</code> module to scan the Prosite database from Python. This module both helps you to access ScanProsite programmatically, and to parse the results returned by ScanProsite. To scan for Prosite patterns in the following protein sequence:</p><pre class="verbatim">MEHKEVVLLLLLFLKSGQGEPLDDYVNTQGASLFSVTKKQLGAGSIEECAAKCEEDEEFT
CRAFQYHSKEQQCVIMAENRKSSIIIRMRDVVLFEKKVYLSECKTGNGKNYRGTMSKTKN
</pre><p>you can use the following code:</p><pre class="verbatim">&gt;&gt;&gt; sequence = "MEHKEVVLLLLLFLKSGQGEPLDDYVNTQGASLFSVTKKQLGAGSIEECAAKCEEDEEFT
CRAFQYHSKEQQCVIMAENRKSSIIIRMRDVVLFEKKVYLSECKTGNGKNYRGTMSKTKN"
&gt;&gt;&gt; from Bio.ExPASy import ScanProsite
&gt;&gt;&gt; handle = ScanProsite.scan(seq=sequence)
</pre><p>By executing <code>handle.read()</code>, you can obtain the search results in raw XML format. Instead, let&#X2019;s use <code>Bio.ExPASy.ScanProsite.read</code> to parse the raw XML into a Python object:</p><pre class="verbatim">&gt;&gt;&gt; result = ScanProsite.read(handle)
&gt;&gt;&gt; type(result)
&lt;class 'Bio.ExPASy.ScanProsite.Record'&gt;
</pre><p>A <code>Bio.ExPASy.ScanProsite.Record</code> object is derived from a list, with each element in the list storing one ScanProsite hit. This object also stores the number of hits, as well as the number of search sequences, as returned by ScanProsite. This ScanProsite search resulted in six hits:</p><pre class="verbatim">&gt;&gt;&gt; result.n_seq
1
&gt;&gt;&gt; result.n_match
6
&gt;&gt;&gt; len(result)
6
&gt;&gt;&gt; result[0]
{'signature_ac': u'PS50948', 'level': u'0', 'stop': 98, 'sequence_ac': u'USERSEQ1', 'start': 16, 'score': u'8.873'}
&gt;&gt;&gt; result[1]
{'start': 37, 'stop': 39, 'sequence_ac': u'USERSEQ1', 'signature_ac': u'PS00005'}
&gt;&gt;&gt; result[2]
{'start': 45, 'stop': 48, 'sequence_ac': u'USERSEQ1', 'signature_ac': u'PS00006'}
&gt;&gt;&gt; result[3]
{'start': 60, 'stop': 62, 'sequence_ac': u'USERSEQ1', 'signature_ac': u'PS00005'}
&gt;&gt;&gt; result[4]
{'start': 80, 'stop': 83, 'sequence_ac': u'USERSEQ1', 'signature_ac': u'PS00004'}
&gt;&gt;&gt; result[5]
{'start': 106, 'stop': 111, 'sequence_ac': u'USERSEQ1', 'signature_ac': u'PS00008'}
</pre><p>Other ScanProsite parameters can be passed as keyword arguments; see the <a href="http://www.expasy.org/tools/scanprosite/ScanPrositeREST.html">documentation for programmatic access of ScanProsite</a> for more information. As an example, passing <code>lowscore=1</code> to include matches with low level scores lets use find one additional hit:</p><pre class="verbatim">&gt;&gt;&gt; handle = ScanProsite.scan(seq=sequence, lowscore=1)
&gt;&gt;&gt; result = ScanProsite.read(handle)
&gt;&gt;&gt; result.n_match
7
</pre>
<!--TOC chapter id="sec156" Going 3D: The PDB module-->
<h1 id="sec156" class="chapter">Chapter&#XA0;11&#XA0;&#XA0;Going 3D: The PDB module</h1><!--SEC END --><p>Bio.PDB is a Biopython module that focuses on working with crystal structures of biological macromolecules. Among other things, Bio.PDB includes a PDBParser class that produces a Structure object, which can be used to access the atomic data in the file in a convenient manner. There is limited support for parsing the information contained in the PDB header.</p>
<!--TOC section id="sec157" Reading and writing crystal structure files-->
<h2 id="sec157" class="section">11.1&#XA0;&#XA0;Reading and writing crystal structure files</h2><!--SEC END -->
<!--TOC subsection id="sec158" Reading a PDB file-->
<h3 id="sec158" class="subsection">11.1.1&#XA0;&#XA0;Reading a PDB file</h3><!--SEC END --><p>First we create a <span style="font-family:monospace">PDBParser</span> object:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.PDB.PDBParser import PDBParser
&gt;&gt;&gt; p = PDBParser(PERMISSIVE=1)
</pre><p>The <span style="font-family:monospace">PERMISSIVE</span> flag indicates that a number of common problems (see <a href="#problem%20structures">11.7.1</a>) associated with PDB files will be ignored (but note that some atoms and/or residues will be missing). If the flag is not present a <span style="font-family:monospace">PDBConstructionException</span> will be generated if any problems are detected during the parse operation.</p><p>The Structure object is then produced by letting the <span style="font-family:monospace">PDBParser</span> object parse a PDB file (the PDB file in this case is called &#X2019;pdb1fat.ent&#X2019;, &#X2019;1fat&#X2019; is a user defined name for the structure):</p><pre class="verbatim">&gt;&gt;&gt; structure_id = "1fat"
&gt;&gt;&gt; filename = "pdb1fat.ent"
&gt;&gt;&gt; s = p.get_structure(structure_id, filename)
</pre><p>You can extract the header and trailer (simple lists of strings) of the PDB
file from the PDBParser object with the <span style="font-family:monospace">get_header</span> and <span style="font-family:monospace">get_trailer</span>
methods. Note however that many PDB files contain headers with
incomplete or erroneous information. Many of the errors have been
fixed in the equivalent mmCIF files. <em>Hence, if you are interested
in the header information, it is a good idea to extract information
from mmCIF files using the</em> <span style="font-family:monospace"><em>MMCIF2Dict</em></span> <em>tool
described below, instead of parsing the PDB header. </em></p><p>Now that is clarified, let&#X2019;s return to parsing the PDB header. The
structure object has an attribute called <span style="font-family:monospace">header</span> which is
a Python dictionary that maps header records to their values.</p><p>Example:</p><pre class="verbatim">&gt;&gt;&gt; resolution = structure.header['resolution']
&gt;&gt;&gt; keywords = structure.header['keywords']
</pre><p>The available keys are <code>name</code>, <code>head</code>, <code>deposition_date</code>, <code>release_date</code>, <code>structure_method</code>, <code>resolution</code>, <code>structure_reference</code> (which maps to a list of references), <code>journal_reference</code>, <code>author</code>, and <code>compound</code> (which maps to a dictionary with various information about the crystallized compound).</p><p>The dictionary can also be created without creating a <span style="font-family:monospace">Structure</span>
object, ie. directly from the PDB file:</p><pre class="verbatim">&gt;&gt;&gt; file = open(filename, 'r')
&gt;&gt;&gt; header_dict = parse_pdb_header(file)
&gt;&gt;&gt; file.close()
</pre>
<!--TOC subsection id="sec159" Reading an mmCIF file-->
<h3 id="sec159" class="subsection">11.1.2&#XA0;&#XA0;Reading an mmCIF file</h3><!--SEC END --><p>Similarly to the case the case of PDB files, first create an <span style="font-family:monospace">MMCIFParser</span> object:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.PDB.MMCIFParser import MMCIFParser
&gt;&gt;&gt; parser = MMCIFParser()
</pre><p>Then use this parser to create a structure object from the mmCIF file:
</p><pre class="verbatim">&gt;&gt;&gt; structure = parser.get_structure('1fat', '1fat.cif')
</pre><p>To have some more low level access to an mmCIF file, you can use the <code>MMCIF2Dict</code> class to create a Python dictionary that maps all mmCIF
tags in an mmCIF file to their values. If there are multiple values
(like in the case of tag <code>_atom_site.Cartn_y</code>, which holds
the <span style="font-style:italic">y</span> coordinates of all atoms), the tag is mapped to a list of values.
The dictionary is created from the mmCIF file as follows:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.PDB.MMCIF2Dict import MMCIF2Dict
&gt;&gt;&gt; mmcif_dict = MMCIF2Dict('1FAT.cif')
</pre><p>Example: get the solvent content from an mmCIF file:
</p><pre class="verbatim">&gt;&gt;&gt; sc = mmcif_dict['_exptl_crystal.density_percent_sol']
</pre><p>Example: get the list of the <span style="font-style:italic">y</span> coordinates of all atoms
</p><pre class="verbatim">&gt;&gt;&gt; y_list = mmcif_dict['_atom_site.Cartn_y']
</pre>
<!--TOC subsection id="sec160" Reading files in the PDB XML format-->
<h3 id="sec160" class="subsection">11.1.3&#XA0;&#XA0;Reading files in the PDB XML format</h3><!--SEC END --><p>That&#X2019;s not yet supported, but we are definitely planning to support that
in the future (it&#X2019;s not a lot of work). Contact the Biopython developers
(<a href="mailto:biopython-dev@biopython.org">biopython-dev@biopython.org</a>) if you need this).</p>
<!--TOC subsection id="sec161" Writing PDB files-->
<h3 id="sec161" class="subsection">11.1.4&#XA0;&#XA0;Writing PDB files</h3><!--SEC END --><p>Use the PDBIO class for this. It&#X2019;s easy to write out specific parts
of a structure too, of course.</p><p>Example: saving a structure</p><pre class="verbatim">&gt;&gt;&gt; io = PDBIO()
&gt;&gt;&gt; io.set_structure(s)
&gt;&gt;&gt; io.save('out.pdb')
</pre><p>If you want to write out a part of the structure, make use of the
<span style="font-family:monospace">Select</span> class (also in <span style="font-family:monospace">PDBIO</span>). Select has four methods:</p><ul class="itemize"><li class="li-itemize">
<code>accept_model(model)</code>
</li><li class="li-itemize"><code>accept_chain(chain)</code>
</li><li class="li-itemize"><code>accept_residue(residue)</code>
</li><li class="li-itemize"><code>accept_atom(atom)</code>
</li></ul><p>
By default, every method returns 1 (which means the model/chain/residue/atom
is included in the output). By subclassing <span style="font-family:monospace">Select</span> and returning
0 when appropriate you can exclude models, chains, etc. from the output.
Cumbersome maybe, but very powerful. The following code only writes
out glycine residues:</p><pre class="verbatim">&gt;&gt;&gt; class GlySelect(Select):
...     def accept_residue(self, residue):
...         if residue.get_name()=='GLY':
...             return True
...         else:
...             return False
... 
&gt;&gt;&gt; io = PDBIO()
&gt;&gt;&gt; io.set_structure(s)
&gt;&gt;&gt; io.save('gly_only.pdb', GlySelect())
</pre><p>If this is all too complicated for you, the <span style="font-family:monospace">Dice</span> module contains
a handy <span style="font-family:monospace">extract</span> function that writes out all residues in
a chain between a start and end residue. </p>
<!--TOC section id="sec162" Structure representation-->
<h2 id="sec162" class="section">11.2&#XA0;&#XA0;Structure representation</h2><!--SEC END --><p>The overall layout of a <span style="font-family:monospace">Structure</span> object follows the so-called SMCRA
(Structure/Model/Chain/Residue/Atom) architecture:</p><ul class="itemize"><li class="li-itemize">
A structure consists of models
</li><li class="li-itemize">A model consists of chains
</li><li class="li-itemize">A chain consists of residues
</li><li class="li-itemize">A residue consists of atoms
</li></ul><p>
This is the way many structural biologists/bioinformaticians think
about structure, and provides a simple but efficient way to deal with
structure. Additional stuff is essentially added when needed. A UML
diagram of the <span style="font-family:monospace">Structure</span> object (forget about the <span style="font-family:monospace">Disordered</span>
classes for now) is shown in Fig. <a href="#fig%3Asmcra">11.1</a>. Such a data structure is not
necessarily best suited for the representation of the macromolecular content of
a structure, but it is absolutely necessary for a good interpretation of the
data present in a file that describes the structure (typically a PDB or MMCIF
file). If this hierarchy cannot represent the contents of a structure file, it
is fairly certain that the file contains an error or at least does not describe
the structure unambiguously. If a SMCRA data structure cannot be generated,
there is reason to suspect a problem. Parsing a PDB file can thus be used to
detect likely problems. We will give several examples of this in section
<a href="#problem%20structures">11.7.1</a>.</p><blockquote class="figure"><div class="center"><hr style="width:80%;height:2"></div>

<img src="images/smcra.png" width=650, height=750>


<div class="caption"><table style="border-spacing:6px;border-collapse:separate;" class="cellpading0"><tr><td style="vertical-align:top;text-align:left;" >Figure 11.1: UML diagram of SMCRA architecture of the <span style="font-family:monospace">Structure</span> class used to represent a macromolecular structure.
Full lines with diamonds denote aggregation, full lines with
arrows denote referencing, full lines with triangles denote inheritance
and dashed lines with triangles denote interface realization.</td></tr>
</table></div>
<a id="fig:smcra"></a>
<div class="center"><hr style="width:80%;height:2"></div></blockquote><p>Structure, Model, Chain and Residue are all subclasses of the Entity base class.
The Atom class only (partly) implements the Entity interface (because an Atom
does not have children).</p><p>For each Entity subclass, you can extract a child by using a unique id for that
child as a key (e.g. you can extract an Atom object from a Residue object by
using an atom name string as a key, you can extract a Chain object from a Model
object by using its chain identifier as a key).</p><p>Disordered atoms and residues are represented by DisorderedAtom and DisorderedResidue
classes, which are both subclasses of the DisorderedEntityWrapper base class.
They hide the complexity associated with disorder and behave exactly as Atom
and Residue objects.</p><p>In general, a child Entity object (i.e. Atom, Residue, Chain, Model) can be
extracted from its parent (i.e. Residue, Chain, Model, Structure, respectively)
by using an id as a key.</p><pre class="verbatim">&gt;&gt;&gt; child_entity = parent_entity[child_id]
</pre><p>You can also get a list of all child Entities of a parent Entity object. Note
that this list is sorted in a specific way (e.g. according to chain identifier
for Chain objects in a Model object).</p><pre class="verbatim">&gt;&gt;&gt; child_list = parent_entity.get_list()
</pre><p>You can also get the parent from a child:
</p><pre class="verbatim">&gt;&gt;&gt; parent_entity = child_entity.get_parent()
</pre><p>At all levels of the SMCRA hierarchy, you can also extract a <em>full id</em>.
The full id is a tuple containing all id&#X2019;s starting from the top object (Structure)
down to the current object. A full id for a Residue object e.g. is something
like:</p><pre class="verbatim">&gt;&gt;&gt; full_id = residue.get_full_id()
&gt;&gt;&gt; print(full_id)
("1abc", 0, "A", ("", 10, "A"))
</pre><p>This corresponds to:</p><ul class="itemize"><li class="li-itemize">
The Structure with id "1abc"
</li><li class="li-itemize">The Model with id 0
</li><li class="li-itemize">The Chain with id "A"
</li><li class="li-itemize">The Residue with id (" ", 10, "A").
</li></ul><p>
The Residue id indicates that the residue is not a hetero-residue (nor a water)
because it has a blank hetero field, that its sequence identifier is 10 and
that its insertion code is "A".</p><p>To get the entity&#X2019;s id, use the <code>get_id</code> method:
</p><pre class="verbatim">&gt;&gt;&gt; entity.get_id()
</pre><p>You can check if the entity has a child with a given id by using the <code>has_id</code> method:
</p><pre class="verbatim">&gt;&gt;&gt; entity.has_id(entity_id)
</pre><p>The length of an entity is equal to its number of children:
</p><pre class="verbatim">&gt;&gt;&gt; nr_children = len(entity)
</pre><p>It is possible to delete, rename, add, etc. child entities from a parent entity,
but this does not include any sanity checks (e.g. it is possible to add two
residues with the same id to one chain). This really should be done via a nice
Decorator class that includes integrity checking, but you can take a look at
the code (Entity.py) if you want to use the raw interface.</p>
<!--TOC subsection id="sec163" Structure-->
<h3 id="sec163" class="subsection">11.2.1&#XA0;&#XA0;Structure</h3><!--SEC END --><p>The Structure object is at the top of the hierarchy. Its id is a user given
string. The Structure contains a number of Model children. Most crystal structures
(but not all) contain a single model, while NMR structures typically consist
of several models. Disorder in crystal structures of large parts of molecules
can also result in several models.</p>
<!--TOC subsection id="sec164" Model-->
<h3 id="sec164" class="subsection">11.2.2&#XA0;&#XA0;Model</h3><!--SEC END --><p>The id of the Model object is an integer, which is derived from the position
of the model in the parsed file (they are automatically numbered starting from
0).
Crystal structures generally have only one model (with id 0), while NMR files usually have several models. Whereas many PDB parsers assume that there is only one model, the <code>Structure</code> class in <code>Bio.PDB</code> is designed such that it can easily handle PDB files with more than one model.</p><p>As an example, to get the first model from a Structure object, use
</p><pre class="verbatim">&gt;&gt;&gt; first_model = structure[0]
</pre><p>The Model object stores a list of Chain children.</p>
<!--TOC subsection id="sec165" Chain-->
<h3 id="sec165" class="subsection">11.2.3&#XA0;&#XA0;Chain</h3><!--SEC END --><p>The id of a Chain object is derived from the chain identifier in the PDB/mmCIF
file, and is a single character (typically a letter). Each Chain in a Model object has a unique id. As an example, to get the Chain object with identifier &#X201C;A&#X201D; from a Model object, use
</p><pre class="verbatim">&gt;&gt;&gt; chain_A = model["A"]
</pre><p>The Chain object stores a list of Residue children.</p>
<!--TOC subsection id="sec166" Residue-->
<h3 id="sec166" class="subsection">11.2.4&#XA0;&#XA0;Residue</h3><!--SEC END --><p>A residue id is a tuple with three elements:</p><ul class="itemize"><li class="li-itemize">
The <span style="font-weight:bold">hetero-field</span> (hetfield): this is
<ul class="itemize"><li class="li-itemize">
<code>'W'</code> in the case of a water molecule;
</li><li class="li-itemize"><code>'H_'</code> followed by the residue name for other hetero residues (e.g. <code>'H_GLC'</code> in the case of a glucose molecule);
</li><li class="li-itemize">blank for standard amino and nucleic acids.
</li></ul>
This scheme is adopted for reasons described in section <a href="#hetero%20problems">11.4.1</a>.
</li><li class="li-itemize">The <span style="font-weight:bold">sequence identifier</span> (resseq), an integer describing the position of the residue in the chain (e.g., 100);
</li><li class="li-itemize">The <span style="font-weight:bold">insertion code</span> (icode); a string, e.g. &#X2019;A&#X2019;. The insertion code is sometimes used to preserve a certain desirable residue numbering scheme. A Ser 80 insertion mutant (inserted e.g. between a Thr 80 and an Asn 81
residue) could e.g. have sequence identifiers and insertion codes
as follows: Thr 80 A, Ser 80 B, Asn 81. In this way the residue numbering
scheme stays in tune with that of the wild type structure.
</li></ul><p>
The id of the above glucose residue would thus be <span style="font-family:monospace">(&#X2019;H_GLC&#X2019;,
100, &#X2019;A&#X2019;)</span>. If the hetero-flag and insertion code are blank, the sequence
identifier alone can be used:</p><pre class="verbatim"># Full id
&gt;&gt;&gt; residue=chain[(' ', 100, ' ')]
# Shortcut id
&gt;&gt;&gt; residue=chain[100]
</pre><p>The reason for the hetero-flag is that many, many PDB files use the
same sequence identifier for an amino acid and a hetero-residue or
a water, which would create obvious problems if the hetero-flag was
not used. </p><p>Unsurprisingly, a Residue object stores a set of Atom children. It also contains a string that specifies the residue name (e.g. &#X201C;ASN&#X201D;)
and the segment identifier of the residue (well known to X-PLOR users, but not
used in the construction of the SMCRA data structure).</p><p>Let&#X2019;s look at some examples. Asn 10 with a blank insertion code would have residue
id <span style="font-family:monospace">(&#X2019; &#X2019;, 10, &#X2019; &#X2019;)</span>. Water 10 would have residue id <span style="font-family:monospace">(&#X2019;W&#X2019;, 10, &#X2019; &#X2019;)</span>.
A glucose molecule (a hetero residue with residue name GLC) with sequence identifier
10 would have residue id <span style="font-family:monospace">(&#X2019;H_GLC&#X2019;, 10, &#X2019; &#X2019;)</span>. In this way, the three
residues (with the same insertion code and sequence identifier) can be part
of the same chain because their residue id&#X2019;s are distinct.</p><p>In most cases, the hetflag and insertion code fields will be blank, e.g. <span style="font-family:monospace">(&#X2019; &#X2019;, 10, &#X2019; &#X2019;)</span>.
In these cases, the sequence identifier can be used as a shortcut for the full
id:</p><pre class="verbatim"># use full id
&gt;&gt;&gt; res10 = chain[(' ', 10, ' ')]
# use shortcut
&gt;&gt;&gt; res10 = chain[10]
</pre><p>Each Residue object in a Chain object should have a unique id. However, disordered
residues are dealt with in a special way, as described in section <a href="#point%20mutations">11.3.3</a>.</p><p>A Residue object has a number of additional methods:</p><pre class="verbatim">&gt;&gt;&gt; residue.get_resname()       # returns the residue name, e.g. "ASN"
&gt;&gt;&gt; residue.is_disordered()     # returns 1 if the residue has disordered atoms
&gt;&gt;&gt; residue.get_segid()         # returns the SEGID, e.g. "CHN1"
&gt;&gt;&gt; residue.has_id(name)        # test if a residue has a certain atom
</pre><p>You can use <span style="font-family:monospace">is_aa(residue)</span> to test if a Residue object is an amino acid.</p>
<!--TOC subsection id="sec167" Atom-->
<h3 id="sec167" class="subsection">11.2.5&#XA0;&#XA0;Atom</h3><!--SEC END --><p>The Atom object stores the data associated with an atom, and has no children.
The id of an atom is its atom name (e.g. &#X201C;OG&#X201D; for the side chain oxygen
of a Ser residue). An Atom id needs to be unique in a Residue. Again, an exception is made for disordered atoms, as described in section <a href="#disordered%20atoms">11.3.2</a>.</p><p>The atom id is simply the atom name (eg. <span style="font-family:monospace">&#X2019;CA&#X2019;</span>). In practice,
the atom name is created by stripping all spaces from the atom name
in the PDB file. </p><p>However, in PDB files, a space can be part of an atom name. Often,
calcium atoms are called <span style="font-family:monospace">&#X2019;CA..&#X2019;</span> in order to distinguish them
from C&#X3B1; atoms (which are called <span style="font-family:monospace">&#X2019;.CA.&#X2019;</span>). In cases
were stripping the spaces would create problems (ie. two atoms called
<span style="font-family:monospace">&#X2019;CA&#X2019;</span> in the same residue) the spaces are kept.</p><p>In a PDB file, an atom name consists of 4 chars, typically with leading and
trailing spaces. Often these spaces can be removed for ease of use (e.g. an
amino acid C &#X3B1;  atom is labeled &#X201C;.CA.&#X201D; in a PDB file, where
the dots represent spaces). To generate an atom name (and thus an atom id) the
spaces are removed, unless this would result in a name collision in a Residue
(i.e. two Atom objects with the same atom name and id). In the latter case,
the atom name including spaces is tried. This situation can e.g. happen when
one residue contains atoms with names &#X201C;.CA.&#X201D; and &#X201C;CA..&#X201D;, although
this is not very likely.</p><p>The atomic data stored includes the atom name, the atomic coordinates (including
standard deviation if present), the B factor (including anisotropic B factors
and standard deviation if present), the altloc specifier and the full atom name
including spaces. Less used items like the atom element number or the atomic
charge sometimes specified in a PDB file are not stored.</p><p>To manipulate the atomic coordinates, use the <span style="font-family:monospace">transform</span> method of
the <span style="font-family:monospace">Atom</span> object. Use the <span style="font-family:monospace">set_coord</span> method to specify the
atomic coordinates directly.</p><p>An Atom object has the following additional methods:</p><pre class="verbatim">&gt;&gt;&gt; a.get_name()       # atom name (spaces stripped, e.g. "CA")
&gt;&gt;&gt; a.get_id()         # id (equals atom name)
&gt;&gt;&gt; a.get_coord()      # atomic coordinates
&gt;&gt;&gt; a.get_vector()     # atomic coordinates as Vector object
&gt;&gt;&gt; a.get_bfactor()    # isotropic B factor
&gt;&gt;&gt; a.get_occupancy()  # occupancy
&gt;&gt;&gt; a.get_altloc()     # alternative location specifier
&gt;&gt;&gt; a.get_sigatm()     # standard deviation of atomic parameters
&gt;&gt;&gt; a.get_siguij()     # standard deviation of anisotropic B factor
&gt;&gt;&gt; a.get_anisou()     # anisotropic B factor
&gt;&gt;&gt; a.get_fullname()   # atom name (with spaces, e.g. ".CA.")
</pre><p>To represent the atom coordinates, siguij, anisotropic B factor and sigatm Numpy
arrays are used.</p><p>The <span style="font-family:monospace">get_vector</span> method returns a <span style="font-family:monospace">Vector</span> object representation of the coordinates of the <span style="font-family:monospace">Atom</span> object, allowing you to do vector operations on atomic coordinates. <span style="font-family:monospace">Vector</span> implements the full set of 3D vector operations, matrix multiplication (left and right) and some advanced rotation-related operations as well.</p><p>As an example of the capabilities of Bio.PDB&#X2019;s <span style="font-family:monospace">Vector</span> module,
suppose that you would like to find the position of a Gly residue&#X2019;s C&#X3B2;
atom, if it had one. Rotating the N atom of
the Gly residue along the C&#X3B1;-C bond over -120 degrees roughly
puts it in the position of a virtual C&#X3B2; atom. Here&#X2019;s how to
do it, making use of the <span style="font-family:monospace">rotaxis</span> method (which can be used
to construct a rotation around a certain axis) of the <span style="font-family:monospace">Vector</span>
module:</p><pre class="verbatim"># get atom coordinates as vectors
&gt;&gt;&gt; n = residue['N'].get_vector() 
&gt;&gt;&gt; c = residue['C'].get_vector() 
&gt;&gt;&gt; ca = residue['CA'].get_vector()
# center at origin
&gt;&gt;&gt; n = n - ca 
&gt;&gt;&gt; c = c - ca 
# find rotation matrix that rotates n 
# -120 degrees along the ca-c vector
&gt;&gt;&gt; rot = rotaxis(-pi * 120.0/180.0, c)
# apply rotation to ca-n vector
&gt;&gt;&gt; cb_at_origin = n.left_multiply(rot)
# put on top of ca atom
&gt;&gt;&gt; cb = cb_at_origin+ca
</pre><p>This example shows that it&#X2019;s possible to do some quite nontrivial
vector operations on atomic data, which can be quite useful. In addition
to all the usual vector operations (cross (use <span style="font-family:monospace">*</span><span style="font-family:monospace">*</span>), and
dot (use <span style="font-family:monospace">*</span>) product, angle, norm, etc.) and the above mentioned
<span style="font-family:monospace">rotaxis</span> function, the <span style="font-family:monospace">Vector</span> module also has methods
to rotate (<span style="font-family:monospace">rotmat</span>) or reflect (<span style="font-family:monospace">refmat</span>) one vector
on top of another.</p>
<!--TOC subsection id="sec168" Extracting a specific <span style="font-family:monospace">Atom/Residue/Chain/Model</span>
from a Structure-->
<h3 id="sec168" class="subsection">11.2.6&#XA0;&#XA0;Extracting a specific <span style="font-family:monospace">Atom/Residue/Chain/Model</span>
from a Structure</h3><!--SEC END --><p>These are some examples:</p><pre class="verbatim">&gt;&gt;&gt; model = structure[0]
&gt;&gt;&gt; chain = model['A']
&gt;&gt;&gt; residue = chain[100]
&gt;&gt;&gt; atom = residue['CA']
</pre><p>Note that you can use a shortcut:</p><pre class="verbatim">&gt;&gt;&gt; atom = structure[0]['A'][100]['CA']
</pre>
<!--TOC section id="sec169" Disorder-->
<h2 id="sec169" class="section">11.3&#XA0;&#XA0;Disorder</h2><!--SEC END --><p>Bio.PDB can handle both disordered atoms and point mutations (i.e. a
Gly and an Ala residue in the same position). </p>
<!--TOC subsection id="disorder problems" General approach-->
<h3 id="disorder problems" class="subsection">11.3.1&#XA0;&#XA0;General approach</h3><!--SEC END --><p>Disorder should be dealt with from two points of view: the atom and the residue
points of view. In general, we have tried to encapsulate all the complexity that
arises from disorder. If you just want to loop over all C&#X3B1; atoms,
you do not care that some residues have a disordered side chain. On the other
hand it should also be possible to represent disorder completely in the data
structure. Therefore, disordered atoms or residues are stored in special objects
that behave as if there is no disorder. This is done by only representing a
subset of the disordered atoms or residues. Which subset is picked (e.g. which
of the two disordered OG side chain atom positions of a Ser residue is used)
can be specified by the user.</p>
<!--TOC subsection id="disordered atoms" Disordered atoms-->
<h3 id="disordered atoms" class="subsection">11.3.2&#XA0;&#XA0;Disordered atoms</h3><!--SEC END --><p>Disordered atoms are represented by ordinary <span style="font-family:monospace">Atom</span> objects, but
all <span style="font-family:monospace">Atom</span> objects that represent the same physical atom are stored
in a <span style="font-family:monospace">DisorderedAtom</span> object (see Fig. <a href="#fig%3Asmcra">11.1</a>).
Each <span style="font-family:monospace">Atom</span> object in a <span style="font-family:monospace">DisorderedAtom</span> object can
be uniquely indexed using its altloc specifier. The <span style="font-family:monospace">DisorderedAtom</span>
object forwards all uncaught method calls to the selected Atom object,
by default the one that represents the atom with the highest
occupancy. The user can of course change the selected <span style="font-family:monospace">Atom</span>
object, making use of its altloc specifier. In this way atom disorder
is represented correctly without much additional complexity. In other
words, if you are not interested in atom disorder, you will not be
bothered by it.</p><p>Each disordered atom has a characteristic altloc identifier. You can
specify that a <span style="font-family:monospace">DisorderedAtom</span> object should behave like
the <span style="font-family:monospace">Atom</span> object associated with a specific altloc identifier:</p><pre class="verbatim">&gt;&gt;&gt; atom.disordered_select('A') # select altloc A atom
&gt;&gt;&gt; print(atom.get_altloc())
"A"
&gt;&gt;&gt; atom.disordered_select('B') # select altloc B atom
&gt;&gt;&gt; print(atom.get_altloc())
"B"
</pre>
<!--TOC subsection id="sec172" Disordered residues-->
<h3 id="sec172" class="subsection">11.3.3&#XA0;&#XA0;Disordered residues</h3><!--SEC END --><!--TOC subsubsection id="sec173" Common case-->
<h4 id="sec173" class="subsubsection">Common case</h4><!--SEC END --><p>The most common case is a residue that contains one or more disordered atoms.
This is evidently solved by using DisorderedAtom objects to represent the disordered
atoms, and storing the DisorderedAtom object in a Residue object just like ordinary
Atom objects. The DisorderedAtom will behave exactly like an ordinary atom (in
fact the atom with the highest occupancy) by forwarding all uncaught method
calls to one of the Atom objects (the selected Atom object) it contains.</p><!--TOC subsubsection id="point mutations" Point mutations-->
<h4 id="point mutations" class="subsubsection">Point mutations</h4><!--SEC END --><p>A special case arises when disorder is due to a point mutation, i.e. when two
or more point mutants of a polypeptide are present in the crystal. An example
of this can be found in PDB structure 1EN2.</p><p>Since these residues belong to a different residue type (e.g. let&#X2019;s
say Ser 60 and Cys 60) they should not be stored in a single <span style="font-family:monospace">Residue</span>
object as in the common case. In this case, each residue is represented
by one <span style="font-family:monospace">Residue</span> object, and both <span style="font-family:monospace">Residue</span> objects
are stored in a single <span style="font-family:monospace">DisorderedResidue</span> object (see Fig.
<a href="#fig%3Asmcra">11.1</a>).</p><p>The <span style="font-family:monospace">DisorderedResidue</span> object forwards all uncaught methods to
the selected <span style="font-family:monospace">Residue</span> object (by default the last <span style="font-family:monospace">Residue</span>
object added), and thus behaves like an ordinary residue. Each
<span style="font-family:monospace">Residue</span> object in a <span style="font-family:monospace">DisorderedResidue</span> object can be
uniquely identified by its residue name. In the above example, residue Ser 60
would have id &#X201C;SER&#X201D; in the <span style="font-family:monospace">DisorderedResidue</span> object, while
residue Cys 60 would have id &#X201C;CYS&#X201D;. The user can select the active
<span style="font-family:monospace">Residue</span> object in a <span style="font-family:monospace">DisorderedResidue</span> object via this id.</p><p>Example: suppose that a chain has a point mutation at position 10,
consisting of a Ser and a Cys residue. Make sure that residue 10 of
this chain behaves as the Cys residue.
</p><pre class="verbatim">&gt;&gt;&gt; residue = chain[10]
&gt;&gt;&gt; residue.disordered_select('CYS')
</pre><p>In addition, you can get a list of all <span style="font-family:monospace">Atom</span> objects (ie.
all <span style="font-family:monospace">DisorderedAtom</span> objects are &#X2019;unpacked&#X2019; to their individual
<span style="font-family:monospace">Atom</span> objects) using the <span style="font-family:monospace">get_unpacked_list</span> method
of a <span style="font-family:monospace">(Disordered)Residue</span> object.</p>
<!--TOC section id="sec175" Hetero residues-->
<h2 id="sec175" class="section">11.4&#XA0;&#XA0;Hetero residues</h2><!--SEC END -->
<!--TOC subsection id="hetero problems" Associated problems-->
<h3 id="hetero problems" class="subsection">11.4.1&#XA0;&#XA0;Associated problems</h3><!--SEC END --><p>A common problem with hetero residues is that several hetero and non-hetero
residues present in the same chain share the same sequence identifier (and insertion
code). Therefore, to generate a unique id for each hetero residue, waters and
other hetero residues are treated in a different way.</p><p>Remember that Residue object have the tuple (hetfield, resseq, icode) as id.
The hetfield is blank (&#X201C; &#X201D;) for amino and nucleic acids, and a string
for waters and other hetero residues. The content of the hetfield is explained
below.</p>
<!--TOC subsection id="sec177" Water residues-->
<h3 id="sec177" class="subsection">11.4.2&#XA0;&#XA0;Water residues</h3><!--SEC END --><p>The hetfield string of a water residue consists of the letter &#X201C;W&#X201D;. So
a typical residue id for a water is (&#X201C;W&#X201D;, 1, &#X201C; &#X201D;).</p>
<!--TOC subsection id="sec178" Other hetero residues-->
<h3 id="sec178" class="subsection">11.4.3&#XA0;&#XA0;Other hetero residues</h3><!--SEC END --><p>The hetfield string for other hetero residues starts with &#X201C;H_&#X201D; followed
by the residue name. A glucose molecule e.g. with residue name &#X201C;GLC&#X201D;
would have hetfield &#X201C;H_GLC&#X201D;. Its residue id could e.g. be (&#X201C;H_GLC&#X201D;,
1, &#X201C; &#X201D;).</p>
<!--TOC section id="sec179" Navigating through a Structure object-->
<h2 id="sec179" class="section">11.5&#XA0;&#XA0;Navigating through a Structure object</h2><!--SEC END --><!--TOC subsubsection id="sec180" Parse a PDB file, and extract some Model, Chain, Residue and Atom objects-->
<h4 id="sec180" class="subsubsection">Parse a PDB file, and extract some Model, Chain, Residue and Atom objects</h4><!--SEC END --><pre class="verbatim">&gt;&gt;&gt; from Bio.PDB.PDBParser import PDBParser
&gt;&gt;&gt; parser = PDBParser()
&gt;&gt;&gt; structure = parser.get_structure("test", "1fat.pdb")
&gt;&gt;&gt; model = structure[0]
&gt;&gt;&gt; chain = model["A"]
&gt;&gt;&gt; residue = chain[1]
&gt;&gt;&gt; atom = residue["CA"]
</pre><!--TOC subsubsection id="sec181" Iterating through all atoms of a structure-->
<h4 id="sec181" class="subsubsection">Iterating through all atoms of a structure</h4><!--SEC END --><pre class="verbatim">&gt;&gt;&gt; p = PDBParser()
&gt;&gt;&gt; structure = p.get_structure('X', 'pdb1fat.ent')
&gt;&gt;&gt; for model in structure:
...     for chain in model:
...         for residue in chain:
...             for atom in residue:
...                 print(atom)
...
</pre><p>There is a shortcut if you want to iterate over all atoms in a structure:
</p><pre class="verbatim">&gt;&gt;&gt; atoms = structure.get_atoms()
&gt;&gt;&gt; for atom in atoms:
...     print(atom)
...
</pre><p>Similarly, to iterate over all atoms in a chain, use
</p><pre class="verbatim">&gt;&gt;&gt; atoms = chain.get_atoms()
&gt;&gt;&gt; for atom in atoms:
...     print(atom)
...
</pre><!--TOC subsubsection id="sec182" Iterating over all residues of a model-->
<h4 id="sec182" class="subsubsection">Iterating over all residues of a model</h4><!--SEC END --><p>or if you want to iterate over all residues in a model:
</p><pre class="verbatim">&gt;&gt;&gt; residues = model.get_residues()
&gt;&gt;&gt; for residue in residues:
...     print(residue)
...
</pre><p>You can also use the <code>Selection.unfold_entities</code> function to get all residues from a structure:
</p><pre class="verbatim">&gt;&gt;&gt; res_list = Selection.unfold_entities(structure, 'R')
</pre><p>or to get all atoms from a chain:
</p><pre class="verbatim">&gt;&gt;&gt; atom_list = Selection.unfold_entities(chain, 'A')
</pre><p>Obviously, <code>A=atom, R=residue, C=chain, M=model, S=structure</code>.
You can use this to go up in the hierarchy, e.g. to get a list of
(unique) <code>Residue</code> or <code>Chain</code> parents from a list of
<code>Atoms</code>:</p><pre class="verbatim">&gt;&gt;&gt; residue_list = Selection.unfold_entities(atom_list, 'R')
&gt;&gt;&gt; chain_list = Selection.unfold_entities(atom_list, 'C')
</pre><p>For more info, see the API documentation.</p><!--TOC subsubsection id="sec183" Extract a hetero residue from a chain (e.g. a glucose (GLC) moiety with resseq 10)-->
<h4 id="sec183" class="subsubsection">Extract a hetero residue from a chain (e.g. a glucose (GLC) moiety with resseq 10)</h4><!--SEC END --><pre class="verbatim">&gt;&gt;&gt; residue_id = ("H_GLC", 10, " ")
&gt;&gt;&gt; residue = chain[residue_id]
</pre><!--TOC subsubsection id="sec184" Print all hetero residues in chain-->
<h4 id="sec184" class="subsubsection">Print all hetero residues in chain</h4><!--SEC END --><pre class="verbatim">&gt;&gt;&gt; for residue in chain.get_list():
...    residue_id = residue.get_id()
...    hetfield = residue_id[0]
...    if hetfield[0]=="H":
...        print(residue_id)
...
</pre><!--TOC subsubsection id="sec185" Print out the coordinates of all CA atoms in a structure with B factor greater than 50-->
<h4 id="sec185" class="subsubsection">Print out the coordinates of all CA atoms in a structure with B factor greater than 50</h4><!--SEC END --><pre class="verbatim">&gt;&gt;&gt; for model in structure.get_list():
...     for chain in model.get_list():
...         for residue in chain.get_list():
...             if residue.has_id("CA"):
...                 ca = residue["CA"]
...                 if ca.get_bfactor() &gt; 50.0:
...                     print(ca.get_coord())
...
</pre><!--TOC subsubsection id="sec186" Print out all the residues that contain disordered atoms-->
<h4 id="sec186" class="subsubsection">Print out all the residues that contain disordered atoms</h4><!--SEC END --><pre class="verbatim">&gt;&gt;&gt; for model in structure.get_list():
...     for chain in model.get_list():
...         for residue in chain.get_list():
...             if residue.is_disordered():
...                 resseq = residue.get_id()[1]
...                 resname = residue.get_resname()
...                 model_id = model.get_id()
...                 chain_id = chain.get_id()
...                 print(model_id, chain_id, resname, resseq)
... 
</pre><!--TOC subsubsection id="sec187" Loop over all disordered atoms, and select all atoms with altloc A (if present)-->
<h4 id="sec187" class="subsubsection">Loop over all disordered atoms, and select all atoms with altloc A (if present)</h4><!--SEC END --><p>
This will make sure that the SMCRA data structure will behave as if only the
atoms with altloc A are present.</p><pre class="verbatim">&gt;&gt;&gt; for model in structure.get_list():
...     for chain in model.get_list():
...         for residue in chain.get_list():
...             if residue.is_disordered():
...                 for atom in residue.get_list():
...                     if atom.is_disordered():
...                         if atom.disordered_has_id("A"):
...                             atom.disordered_select("A")
...
</pre><!--TOC subsubsection id="subsubsec:extracting_polypeptides" Extracting polypeptides from a <span style="font-family:monospace">Structure</span> object-->
<h4 id="subsubsec:extracting_polypeptides" class="subsubsection">Extracting polypeptides from a <span style="font-family:monospace">Structure</span> object</h4><!--SEC END --><p>To extract polypeptides from a structure, construct a list of <span style="font-family:monospace">Polypeptide</span> objects from a <span style="font-family:monospace">Structure</span> object using <span style="font-family:monospace">PolypeptideBuilder</span> as follows:</p><pre class="verbatim">&gt;&gt;&gt; model_nr = 1
&gt;&gt;&gt; polypeptide_list = build_peptides(structure, model_nr)
&gt;&gt;&gt; for polypeptide in polypeptide_list:
...     print(polypeptide)
...
</pre><p>A Polypeptide object is simply a UserList of Residue objects, and is always created from a single Model (in this case model 1).
You can use the resulting <span style="font-family:monospace">Polypeptide</span> object to get the sequence as a <span style="font-family:monospace">Seq</span> object or to get a list of C&#X3B1; atoms as well. Polypeptides can be built using a C-N or a C&#X3B1;-C&#X3B1; distance criterion.</p><p>Example:</p><pre class="verbatim"># Using C-N 
&gt;&gt;&gt; ppb=PPBuilder()
&gt;&gt;&gt; for pp in ppb.build_peptides(structure): 
...     print(pp.get_sequence())
...
# Using CA-CA
&gt;&gt;&gt; ppb=CaPPBuilder()
&gt;&gt;&gt; for pp in ppb.build_peptides(structure): 
...     print(pp.get_sequence())
...
</pre><p>Note that in the above case only model 0 of the structure is considered
by <span style="font-family:monospace">PolypeptideBuilder</span>. However, it is possible to use <span style="font-family:monospace">PolypeptideBuilder</span>
to build <span style="font-family:monospace">Polypeptide</span> objects from <span style="font-family:monospace">Model</span> and <span style="font-family:monospace">Chain</span>
objects as well.</p><!--TOC subsubsection id="sec189" Obtaining the sequence of a structure-->
<h4 id="sec189" class="subsubsection">Obtaining the sequence of a structure</h4><!--SEC END --><p>The first thing to do is to extract all polypeptides from the structure
(as above). The sequence of each polypeptide can then easily
be obtained from the <span style="font-family:monospace">Polypeptide</span> objects. The sequence is
represented as a Biopython <span style="font-family:monospace">Seq</span> object, and its alphabet is
defined by a <span style="font-family:monospace">ProteinAlphabet</span> object.</p><p>Example:</p><pre class="verbatim">&gt;&gt;&gt; seq = polypeptide.get_sequence()
&gt;&gt;&gt; print(seq)
Seq('SNVVE...', &lt;class Bio.Alphabet.ProteinAlphabet&gt;)
</pre>
<!--TOC section id="sec190" Analyzing structures-->
<h2 id="sec190" class="section">11.6&#XA0;&#XA0;Analyzing structures</h2><!--SEC END -->
<!--TOC subsection id="sec191" Measuring distances-->
<h3 id="sec191" class="subsection">11.6.1&#XA0;&#XA0;Measuring distances</h3><!--SEC END --><p>
The minus operator for atoms has been overloaded to return the distance between two atoms. 
</p><pre class="verbatim"># Get some atoms
&gt;&gt;&gt; ca1 = residue1['CA']
&gt;&gt;&gt; ca2 = residue2['CA']
# Simply subtract the atoms to get their distance
&gt;&gt;&gt; distance = ca1-ca2
</pre>
<!--TOC subsection id="sec192" Measuring angles-->
<h3 id="sec192" class="subsection">11.6.2&#XA0;&#XA0;Measuring angles</h3><!--SEC END --><p>
Use the vector representation of the atomic coordinates, and
the <span style="font-family:monospace">calc_angle</span> function from the <span style="font-family:monospace">Vector</span> module:
</p><pre class="verbatim">&gt;&gt;&gt; vector1 = atom1.get_vector()
&gt;&gt;&gt; vector2 = atom2.get_vector()
&gt;&gt;&gt; vector3 = atom3.get_vector()
&gt;&gt;&gt; angle = calc_angle(vector1, vector2, vector3)
</pre>
<!--TOC subsection id="sec193" Measuring torsion angles-->
<h3 id="sec193" class="subsection">11.6.3&#XA0;&#XA0;Measuring torsion angles</h3><!--SEC END --><p>
Use the vector representation of the atomic coordinates, and
the <span style="font-family:monospace">calc_dihedral</span> function from the <span style="font-family:monospace">Vector</span> module:
</p><pre class="verbatim">&gt;&gt;&gt; vector1 = atom1.get_vector()
&gt;&gt;&gt; vector2 = atom2.get_vector()
&gt;&gt;&gt; vector3 = atom3.get_vector()
&gt;&gt;&gt; vector4 = atom4.get_vector()
&gt;&gt;&gt; angle = calc_dihedral(vector1, vector2, vector3, vector4)
</pre>
<!--TOC subsection id="sec194" Determining atom-atom contacts-->
<h3 id="sec194" class="subsection">11.6.4&#XA0;&#XA0;Determining atom-atom contacts</h3><!--SEC END --><p>Use <span style="font-family:monospace">NeighborSearch</span> to perform neighbor lookup.
The neighbor lookup is done using a KD tree module written in C (see <span style="font-family:monospace">Bio.KDTree</span>), making it very fast.
It also includes a fast method to find all point pairs within a certain distance of each other.</p>
<!--TOC subsection id="sec195" Superimposing two structures-->
<h3 id="sec195" class="subsection">11.6.5&#XA0;&#XA0;Superimposing two structures</h3><!--SEC END --><p>Use a <span style="font-family:monospace">Superimposer</span> object to superimpose two coordinate sets.
This object calculates the rotation and translation matrix that rotates
two lists of atoms on top of each other in such a way that their RMSD
is minimized. Of course, the two lists need to contain the same number
of atoms. The <span style="font-family:monospace">Superimposer</span> object can also apply the rotation/translation
to a list of atoms. The rotation and translation are stored as a tuple
in the <span style="font-family:monospace">rotran</span> attribute of the <span style="font-family:monospace">Superimposer</span> object
(note that the rotation is right multiplying!). The RMSD is stored
in the <span style="font-family:monospace">rmsd</span> attribute.</p><p>The algorithm used by <span style="font-family:monospace">Superimposer</span> comes from [<a href="#golub1989">17</a>, Golub &amp; Van Loan] and makes use of singular value decomposition (this is implemented in the general <span style="font-family:monospace">Bio.SVDSuperimposer</span> module).</p><p>Example:</p><pre class="verbatim">&gt;&gt;&gt; sup = Superimposer()
# Specify the atom lists
# 'fixed' and 'moving' are lists of Atom objects
# The moving atoms will be put on the fixed atoms
&gt;&gt;&gt; sup.set_atoms(fixed, moving)
# Print rotation/translation/rmsd
&gt;&gt;&gt; print(sup.rotran)
&gt;&gt;&gt; print(sup.rms) 
# Apply rotation/translation to the moving atoms
&gt;&gt;&gt; sup.apply(moving)
</pre><p>To superimpose two structures based on their active sites, use the active site atoms to calculate the rotation/translation matrices (as above), and apply these to the whole molecule.</p>
<!--TOC subsection id="sec196" Mapping the residues of two related structures onto each other-->
<h3 id="sec196" class="subsection">11.6.6&#XA0;&#XA0;Mapping the residues of two related structures onto each other</h3><!--SEC END --><p>First, create an alignment file in FASTA format, then use the <span style="font-family:monospace">StructureAlignment</span>
class. This class can also be used for alignments with more than two
structures.</p>
<!--TOC subsection id="sec197" Calculating the Half Sphere Exposure-->
<h3 id="sec197" class="subsection">11.6.7&#XA0;&#XA0;Calculating the Half Sphere Exposure</h3><!--SEC END --><p>Half Sphere Exposure (HSE) is a new, 2D measure of solvent exposure
[<a href="#hamelryck2005">20</a>].
Basically, it counts the number of C&#X3B1; atoms around a residue
in the direction of its side chain, and in the opposite direction
(within a radius of 13 &#XC5;). Despite its simplicity, it outperforms
many other measures of solvent exposure.</p><p>HSE comes in two flavors: HSE&#X3B1; and HSE&#X3B2;. The former
only uses the C&#X3B1; atom positions, while the latter uses the
C&#X3B1; and C&#X3B2; atom positions. The HSE measure is calculated
by the <span style="font-family:monospace">HSExposure</span> class, which can also calculate the contact
number. The latter class has methods which return dictionaries that
map a <span style="font-family:monospace">Residue</span> object to its corresponding HSE&#X3B1;, HSE&#X3B2;
and contact number values.</p><p>Example:</p><pre class="verbatim">&gt;&gt;&gt; model = structure[0]
&gt;&gt;&gt; hse = HSExposure()
# Calculate HSEalpha
&gt;&gt;&gt; exp_ca = hse.calc_hs_exposure(model, option='CA3')
# Calculate HSEbeta
&gt;&gt;&gt; exp_cb=hse.calc_hs_exposure(model, option='CB')
# Calculate classical coordination number
&gt;&gt;&gt; exp_fs = hse.calc_fs_exposure(model)
# Print HSEalpha for a residue
&gt;&gt;&gt; print(exp_ca[some_residue])
</pre>
<!--TOC subsection id="sec198" Determining the secondary structure-->
<h3 id="sec198" class="subsection">11.6.8&#XA0;&#XA0;Determining the secondary structure</h3><!--SEC END --><p>For this functionality, you need to install DSSP (and obtain a license
for it &#X2014; free for academic use, see <a href="http://www.cmbi.kun.nl/gv/dssp/"><span style="font-family:monospace">http://www.cmbi.kun.nl/gv/dssp/</span></a>).
Then use the <span style="font-family:monospace">DSSP</span> class, which maps <span style="font-family:monospace">Residue</span> objects
to their secondary structure (and accessible surface area). The DSSP
codes are listed in Table <a href="#cap%3ADSSP-codes">11.1</a>. Note that DSSP (the
program, and thus by consequence the class) cannot handle multiple
models!</p><blockquote class="table"><div class="center"><hr style="width:80%;height:2"></div>
<table border=1  style="border-spacing:0;" class="cellpadding1"><tr><td style="text-align:center;border:solid 1px;white-space:nowrap" >Code</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >Secondary structure </td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" >H</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >&#X3B1;-helix </td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" >B</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >Isolated &#X3B2;-bridge residue </td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" >E</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >Strand </td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" >G</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >3-10 helix </td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" >I</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >&#X3A0;-helix </td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" >T</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >Turn</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" >S</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >Bend </td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" >-</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >Other</td></tr>
</table>
<div class="caption"><table style="border-spacing:6px;border-collapse:separate;" class="cellpading0"><tr><td style="vertical-align:top;text-align:left;" >Table 11.1: <a id="cap:DSSP-codes"></a>DSSP codes in Bio.PDB.</td></tr>
</table></div>
<div class="center"><hr style="width:80%;height:2"></div></blockquote><p>The <span style="font-family:monospace">DSSP</span> class can also be used to calculate the accessible surface area of a residue. But see also section <a href="#subsec%3Aresidue_depth">11.6.9</a>.</p>
<!--TOC subsection id="subsec:residue_depth" Calculating the residue depth-->
<h3 id="subsec:residue_depth" class="subsection">11.6.9&#XA0;&#XA0;Calculating the residue depth</h3><!--SEC END --><p>Residue depth is the average distance of a residue&#X2019;s atoms from the
solvent accessible surface. It&#X2019;s a fairly new and very powerful parameterization
of solvent accessibility. For this functionality, you need to install
Michel Sanner&#X2019;s MSMS program (<a href="http://www.scripps.edu/pub/olson-web/people/sanner/html/msms_home.html"><span style="font-family:monospace">http://www.scripps.edu/pub/olson-web/people/sanner/html/msms_home.html</span></a>).
Then use the <span style="font-family:monospace">ResidueDepth</span> class. This class behaves as a
dictionary which maps <span style="font-family:monospace">Residue</span> objects to corresponding (residue
depth, C&#X3B1; depth) tuples. The C&#X3B1; depth is the distance
of a residue&#X2019;s C&#X3B1; atom to the solvent accessible surface. </p><p>Example:</p><pre class="verbatim">&gt;&gt;&gt; model = structure[0]
&gt;&gt;&gt; rd = ResidueDepth(model, pdb_file)
&gt;&gt;&gt; residue_depth, ca_depth=rd[some_residue]
</pre><p>You can also get access to the molecular surface itself (via the <span style="font-family:monospace">get_surface</span>
function), in the form of a Numeric Python array with the surface points.</p>
<!--TOC section id="sec200" Common problems in PDB files-->
<h2 id="sec200" class="section">11.7&#XA0;&#XA0;Common problems in PDB files</h2><!--SEC END --><p>It is well known that many PDB files contain semantic errors (not the
structures themselves, but their representation in PDB files).
Bio.PDB tries to handle this in two ways. The PDBParser
object can behave in two ways: a restrictive way and a permissive
way, which is the default.</p><p>Example:</p><pre class="verbatim"># Permissive parser
&gt;&gt;&gt; parser = PDBParser(PERMISSIVE=1)
&gt;&gt;&gt; parser = PDBParser() # The same (default)
# Strict parser
&gt;&gt;&gt; strict_parser = PDBParser(PERMISSIVE=0)
</pre><p>In the permissive state (DEFAULT), PDB files that obviously contain
errors are &#X201C;corrected&#X201D; (i.e. some residues or atoms are left out).
These errors include:</p><ul class="itemize"><li class="li-itemize">
Multiple residues with the same identifier
</li><li class="li-itemize">Multiple atoms with the same identifier (taking into account the altloc
identifier)
</li></ul><p>
These errors indicate real problems in the PDB file (for details see
[<a href="#hamelryck2003a">18</a>, Hamelryck and Manderick, 2003]). In the restrictive state, PDB files with errors cause an exception to occur. This is useful to find errors in PDB files.</p><p>Some errors however are automatically corrected. Normally each disordered
atom should have a non-blank altloc identifier. However, there are
many structures that do not follow this convention, and have a blank
and a non-blank identifier for two disordered positions of the same
atom. This is automatically interpreted in the right way.</p><p>Sometimes a structure contains a list of residues belonging to chain
A, followed by residues belonging to chain B, and again followed by
residues belonging to chain A, i.e. the chains are &#X2019;broken&#X2019;. This
is also correctly interpreted.</p>
<!--TOC subsection id="problem structures" Examples-->
<h3 id="problem structures" class="subsection">11.7.1&#XA0;&#XA0;Examples</h3><!--SEC END --><p>The PDBParser/Structure class was tested on about 800 structures (each belonging
to a unique SCOP superfamily). This takes about 20 minutes, or on average 1.5
seconds per structure. Parsing the structure of the large ribosomal subunit
(1FKK), which contains about 64000 atoms, takes 10 seconds on a 1000 MHz PC.</p><p>Three exceptions were generated in cases where an unambiguous data structure
could not be built. In all three cases, the likely cause is an error in the
PDB file that should be corrected. Generating an exception in these cases
is much better than running the chance of incorrectly describing
the structure in a data structure.</p>
<!--TOC subsubsection id="sec202" Duplicate residues-->
<h4 id="sec202" class="subsubsection">11.7.1.1&#XA0;&#XA0;Duplicate residues</h4><!--SEC END --><p>One structure contains two amino acid residues in one chain with the same sequence
identifier (resseq 3) and icode. Upon inspection it was found that this chain
contains the residues Thr A3, &#X2026;, Gly A202, Leu A3, Glu A204. Clearly,
Leu A3 should be Leu A203. A couple of similar situations exist for structure
1FFK (which e.g. contains Gly B64, Met B65, Glu B65, Thr B67, i.e. residue Glu
B65 should be Glu B66).</p>
<!--TOC subsubsection id="sec203" Duplicate atoms-->
<h4 id="sec203" class="subsubsection">11.7.1.2&#XA0;&#XA0;Duplicate atoms</h4><!--SEC END --><p>Structure 1EJG contains a Ser/Pro point mutation in chain A at position 22.
In turn, Ser 22 contains some disordered atoms. As expected, all atoms belonging
to Ser 22 have a non-blank altloc specifier (B or C). All atoms of Pro 22 have
altloc A, except the N atom which has a blank altloc. This generates an exception,
because all atoms belonging to two residues at a point mutation should have
non-blank altloc. It turns out that this atom is probably shared by Ser and
Pro 22, as Ser 22 misses the N atom. Again, this points to a problem in the
file: the N atom should be present in both the Ser and the Pro residue, in both
cases associated with a suitable altloc identifier.</p>
<!--TOC subsection id="sec204" Automatic correction-->
<h3 id="sec204" class="subsection">11.7.2&#XA0;&#XA0;Automatic correction</h3><!--SEC END --><p>Some errors are quite common and can be easily corrected without much risk of
making a wrong interpretation. These cases are listed below.</p>
<!--TOC subsubsection id="sec205" A blank altloc for a disordered atom-->
<h4 id="sec205" class="subsubsection">11.7.2.1&#XA0;&#XA0;A blank altloc for a disordered atom</h4><!--SEC END --><p>Normally each disordered atom should have a non-blank altloc identifier. However,
there are many structures that do not follow this convention, and have a blank
and a non-blank identifier for two disordered positions of the same atom. This
is automatically interpreted in the right way.</p>
<!--TOC subsubsection id="sec206" Broken chains-->
<h4 id="sec206" class="subsubsection">11.7.2.2&#XA0;&#XA0;Broken chains</h4><!--SEC END --><p>Sometimes a structure contains a list of residues belonging to chain A, followed
by residues belonging to chain B, and again followed by residues belonging to
chain A, i.e. the chains are &#X201C;broken&#X201D;. This is correctly interpreted.</p>
<!--TOC subsection id="sec207" Fatal errors-->
<h3 id="sec207" class="subsection">11.7.3&#XA0;&#XA0;Fatal errors</h3><!--SEC END --><p>Sometimes a PDB file cannot be unambiguously interpreted. Rather than guessing
and risking a mistake, an exception is generated, and the user is expected to
correct the PDB file. These cases are listed below.</p>
<!--TOC subsubsection id="sec208" Duplicate residues-->
<h4 id="sec208" class="subsubsection">11.7.3.1&#XA0;&#XA0;Duplicate residues</h4><!--SEC END --><p>All residues in a chain should have a unique id. This id is generated based
on:</p><ul class="itemize"><li class="li-itemize">
The sequence identifier (resseq).
</li><li class="li-itemize">The insertion code (icode).
</li><li class="li-itemize">The hetfield string (&#X201C;W&#X201D; for waters and &#X201C;H_&#X201D; followed by the
residue name for other hetero residues)
</li><li class="li-itemize">The residue names of the residues in the case of point mutations (to store the
Residue objects in a DisorderedResidue object).
</li></ul><p>
If this does not lead to a unique id something is quite likely wrong, and an
exception is generated.</p>
<!--TOC subsubsection id="sec209" Duplicate atoms-->
<h4 id="sec209" class="subsubsection">11.7.3.2&#XA0;&#XA0;Duplicate atoms</h4><!--SEC END --><p>All atoms in a residue should have a unique id. This id is generated based on:</p><ul class="itemize"><li class="li-itemize">
The atom name (without spaces, or with spaces if a problem arises).
</li><li class="li-itemize">The altloc specifier.
</li></ul><p>
If this does not lead to a unique id something is quite likely wrong, and an
exception is generated.</p>
<!--TOC section id="sec210" Accessing the Protein Data Bank-->
<h2 id="sec210" class="section">11.8&#XA0;&#XA0;Accessing the Protein Data Bank</h2><!--SEC END -->
<!--TOC subsection id="sec211" Downloading structures from the Protein Data Bank-->
<h3 id="sec211" class="subsection">11.8.1&#XA0;&#XA0;Downloading structures from the Protein Data Bank</h3><!--SEC END --><p>Structures can be downloaded from the PDB (Protein Data Bank)
by using the <span style="font-family:monospace">retrieve_pdb_file</span> method on a <span style="font-family:monospace">PDBList</span> object.
The argument for this method is the PDB identifier of the structure.</p><pre class="verbatim">&gt;&gt;&gt; pdbl = PDBList()
&gt;&gt;&gt; pdbl.retrieve_pdb_file('1FAT')
</pre><p>The <span style="font-family:monospace">PDBList</span> class can also be used as a command-line tool: 
</p><pre class="verbatim">python PDBList.py 1fat
</pre><p>The downloaded file will be called <span style="font-family:monospace">pdb1fat.ent</span> and stored
in the current working directory. Note that the <span style="font-family:monospace">retrieve_pdb_file</span>
method also has an optional argument <span style="font-family:monospace">pdir</span> that specifies
a specific directory in which to store the downloaded PDB files. </p><p>The <span style="font-family:monospace">retrieve_pdb_file</span> method also has some options to specify
the compression format used for the download, and the program used
for local decompression (default <span style="font-family:monospace">.Z</span> format and <span style="font-family:monospace">gunzip</span>).
In addition, the PDB ftp site can be specified upon creation of the
<span style="font-family:monospace">PDBList</span> object. By default, the server of the Worldwide Protein Data Bank (<a href="ftp://ftp.wwpdb.org/pub/pdb/data/structures/divided/pdb/"><span style="font-family:monospace">ftp://ftp.wwpdb.org/pub/pdb/data/structures/divided/pdb/</span></a>)
is used. See the API documentation for more details. Thanks again
to Kristian Rother for donating this module.</p>
<!--TOC subsection id="sec212" Downloading the entire PDB-->
<h3 id="sec212" class="subsection">11.8.2&#XA0;&#XA0;Downloading the entire PDB</h3><!--SEC END --><p>The following commands will store all PDB files in the <span style="font-family:monospace">/data/pdb</span>
directory: </p><pre class="verbatim">python PDBList.py all /data/pdb

python PDBList.py all /data/pdb -d
</pre><p>The API method for this is called <span style="font-family:monospace">download_entire_pdb</span>.
Adding the <span style="font-family:monospace">-d</span> option will store all files in the same directory.
Otherwise, they are sorted into PDB-style subdirectories according
to their PDB ID&#X2019;s. Depending on the traffic, a complete download will
take 2-4 days. </p>
<!--TOC subsection id="sec213" Keeping a local copy of the PDB up to date-->
<h3 id="sec213" class="subsection">11.8.3&#XA0;&#XA0;Keeping a local copy of the PDB up to date</h3><!--SEC END --><p>This can also be done using the <span style="font-family:monospace">PDBList</span> object. One simply
creates a <span style="font-family:monospace">PDBList</span> object (specifying the directory where
the local copy of the PDB is present) and calls the <span style="font-family:monospace">update_pdb</span>
method:</p><pre class="verbatim">&gt;&gt;&gt; pl = PDBList(pdb='/data/pdb')
&gt;&gt;&gt; pl.update_pdb()
</pre><p>One can of course make a weekly <span style="font-family:monospace">cronjob</span> out of this to keep
the local copy automatically up-to-date. The PDB ftp site can also
be specified (see API documentation).</p><p><span style="font-family:monospace">PDBList</span> has some additional methods that can be of use. The
<span style="font-family:monospace">get_all_obsolete</span> method can be used to get a list of all
obsolete PDB entries. The <span style="font-family:monospace">changed_this_week</span> method can
be used to obtain the entries that were added, modified or obsoleted
during the current week. For more info on the possibilities of <span style="font-family:monospace">PDBList</span>,
see the API documentation.</p>
<!--TOC section id="sec214" General questions-->
<h2 id="sec214" class="section">11.9&#XA0;&#XA0;General questions</h2><!--SEC END -->
<!--TOC subsection id="sec215" How well tested is Bio.PDB?-->
<h3 id="sec215" class="subsection">11.9.1&#XA0;&#XA0;How well tested is Bio.PDB?</h3><!--SEC END --><p>Pretty well, actually. Bio.PDB has been extensively tested on nearly
5500 structures from the PDB - all structures seemed to be parsed
correctly. More details can be found in the Bio.PDB Bioinformatics
article. Bio.PDB has been used/is being used in many research projects
as a reliable tool. In fact, I&#X2019;m using Bio.PDB almost daily for research
purposes and continue working on improving it and adding new features.</p>
<!--TOC subsection id="sec216" How fast is it?-->
<h3 id="sec216" class="subsection">11.9.2&#XA0;&#XA0;How fast is it?</h3><!--SEC END --><p>The <span style="font-family:monospace">PDBParser</span> performance was tested on about 800 structures
(each belonging to a unique SCOP superfamily). This takes about 20
minutes, or on average 1.5 seconds per structure. Parsing the structure
of the large ribosomal subunit (1FKK), which contains about 64000
atoms, takes 10 seconds on a 1000 MHz PC. In short: it&#X2019;s more than
fast enough for many applications.</p>
<!--TOC subsection id="sec217" Is there support for molecular graphics?-->
<h3 id="sec217" class="subsection">11.9.3&#XA0;&#XA0;Is there support for molecular graphics?</h3><!--SEC END --><p>Not directly, mostly since there are quite a few Python based/Python
aware solutions already, that can potentially be used with Bio.PDB.
My choice is Pymol, BTW (I&#X2019;ve used this successfully with Bio.PDB,
and there will probably be specific PyMol modules in Bio.PDB soon/some
day). Python based/aware molecular graphics solutions include:</p><ul class="itemize"><li class="li-itemize">
PyMol: <a href="http://pymol.sourceforge.net/"><span style="font-family:monospace">http://pymol.sourceforge.net/</span></a>
</li><li class="li-itemize">Chimera: <a href="http://www.cgl.ucsf.edu/chimera/"><span style="font-family:monospace">http://www.cgl.ucsf.edu/chimera/</span></a>
</li><li class="li-itemize">PMV: <a href="http://www.scripps.edu/~sanner/python/"><span style="font-family:monospace">http://www.scripps.edu/~sanner/python/</span></a>
</li><li class="li-itemize">Coot: <a href="http://www.ysbl.york.ac.uk/~emsley/coot/"><span style="font-family:monospace">http://www.ysbl.york.ac.uk/~emsley/coot/</span></a>
</li><li class="li-itemize">CCP4mg: <a href="http://www.ysbl.york.ac.uk/~lizp/molgraphics.html"><span style="font-family:monospace">http://www.ysbl.york.ac.uk/~lizp/molgraphics.html</span></a>
</li><li class="li-itemize">mmLib: <a href="http://pymmlib.sourceforge.net/"><span style="font-family:monospace">http://pymmlib.sourceforge.net/</span></a> 
</li><li class="li-itemize">VMD: <a href="http://www.ks.uiuc.edu/Research/vmd/"><span style="font-family:monospace">http://www.ks.uiuc.edu/Research/vmd/</span></a>
</li><li class="li-itemize">MMTK: <a href="http://starship.python.net/crew/hinsen/MMTK/"><span style="font-family:monospace">http://starship.python.net/crew/hinsen/MMTK/</span></a>
</li></ul>
<!--TOC subsection id="sec218" Who&#X2019;s using Bio.PDB?-->
<h3 id="sec218" class="subsection">11.9.4&#XA0;&#XA0;Who&#X2019;s using Bio.PDB?</h3><!--SEC END --><p>Bio.PDB was used in the construction of DISEMBL, a web server that
predicts disordered regions in proteins (<a href="http://dis.embl.de/"><span style="font-family:monospace">http://dis.embl.de/</span></a>),
and COLUMBA, a website that provides annotated protein structures
(<a href="http://www.columba-db.de/"><span style="font-family:monospace">http://www.columba-db.de/</span></a>). Bio.PDB has also been used to
perform a large scale search for active sites similarities between
protein structures in the PDB [<a href="#hamelryck2003b">19</a>, Hamelryck, 2003], and to develop a new algorithm
that identifies linear secondary structure elements [<a href="#majumdar2005">26</a>, Majumdar <span style="font-style:italic">et al.</span>, 2005].</p><p>Judging from requests for features and information, Bio.PDB is also
used by several LPCs (Large Pharmaceutical Companies :-).</p>
<!--TOC chapter id="sec219" Bio.PopGen: Population genetics-->
<h1 id="sec219" class="chapter">Chapter&#XA0;12&#XA0;&#XA0;Bio.PopGen: Population genetics</h1><!--SEC END --><p>Bio.PopGen is a Biopython module supporting population genetics,
available in Biopython 1.44 onwards.</p><p>The medium term objective for the module is to support widely used data
formats, applications and databases. This module is currently under intense
development and support for new features should appear at a rather fast pace.
Unfortunately this might also entail some instability on the API, especially
if you are using a development version. APIs that are made available on
our official public releases should be much more stable.</p>
<!--TOC section id="sec220" GenePop-->
<h2 id="sec220" class="section">12.1&#XA0;&#XA0;GenePop</h2><!--SEC END --><p>GenePop (<a href="http://genepop.curtin.edu.au/"><span style="font-family:monospace">http://genepop.curtin.edu.au/</span></a>) is a popular population
genetics software package supporting Hardy-Weinberg tests, linkage
desiquilibrium, population diferentiation, basic statistics, <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub> and
migration estimates, among others. GenePop does not supply sequence
based statistics as it doesn&#X2019;t handle sequence data.
The GenePop file format is supported by a wide range of other population
genetic software applications, thus making it a relevant format in the
population genetics field.</p><p>Bio.PopGen provides a parser and generator of GenePop file format.
Utilities to manipulate the content of a record are also provided.
Here is an example on how to read a GenePop file (you can find
example GenePop data files in the Test/PopGen directory of Biopython):</p><pre class="verbatim">from Bio.PopGen import GenePop

handle = open("example.gen")
rec = GenePop.read(handle)
handle.close()
</pre><p>This will read a file called example.gen and parse it. If you
do print rec, the record will be output again, in GenePop format.</p><p>The most important information in rec will be the loci names and
population information (but there is more &#X2013; use help(GenePop.Record)
to check the API documentation). Loci names can be found on rec.loci_list.
Population information can be found on rec.populations.
Populations is a list with one element per population. Each element is itself
a list of individuals, each individual is a pair composed by individual
name and a list of alleles (2 per marker), here is an example for
rec.populations:</p><pre class="verbatim">[
    [
        ('Ind1', [(1, 2),    (3, 3), (200, 201)],
        ('Ind2', [(2, None), (3, 3), (None, None)],
    ],
    [
        ('Other1', [(1, 1),  (4, 3), (200, 200)],
    ]
]
</pre><p>So we have two populations, the first with two individuals, the
second with only one. The first individual of the first
population is called Ind1, allelic information for each of
the 3 loci follows. Please note that for any locus, information
might be missing (see as an example, Ind2 above).</p><p>A few utility functions to manipulate GenePop records are made
available, here is an example:</p><pre class="verbatim">from Bio.PopGen import GenePop

#Imagine that you have loaded rec, as per the code snippet above...

rec.remove_population(pos)
#Removes a population from a record, pos is the population position in
#  rec.populations, remember that it starts on position 0.
#  rec is altered.

rec.remove_locus_by_position(pos)
#Removes a locus by its position, pos is the locus position in
#  rec.loci_list, remember that it starts on position 0.
#  rec is altered.

rec.remove_locus_by_name(name)
#Removes a locus by its name, name is the locus name as in
#  rec.loci_list. If the name doesn't exist the function fails
#  silently.
#  rec is altered.

rec_loci = rec.split_in_loci()
#Splits a record in loci, that is, for each loci, it creates a new
#  record, with a single loci and all populations.
#  The result is returned in a dictionary, being each key the locus name.
#  The value is the GenePop record.
#  rec is not altered.

rec_pops =  rec.split_in_pops(pop_names)
#Splits a record in populations, that is, for each population, it creates
#  a new record, with a single population and all loci.
#  The result is returned in a dictionary, being each key
#  the population name. As population names are not available in GenePop,
#  they are passed in array (pop_names).
#  The value of each dictionary entry is the GenePop record.
#  rec is not altered.
</pre><p>GenePop does not support population names, a limitation which can be
cumbersome at times. Functionality to enable population names is currently
being planned for Biopython. These extensions won&#X2019;t break compatibility in
any way with the standard format. In the medium term, we would also like to
support the GenePop web service.</p>
<!--TOC section id="sec221" Coalescent simulation-->
<h2 id="sec221" class="section">12.2&#XA0;&#XA0;Coalescent simulation</h2><!--SEC END --><p>A coalescent simulation is a backward model of population genetics with relation to
time. A simulation of ancestry is done until the Most Recent Common Ancestor (MRCA) is found.
This ancestry relationship starting on the MRCA and ending on the current generation
sample is sometimes called a genealogy. Simple cases assume a population of constant
size in time, haploidy, no population structure, and simulate the alleles of a single
locus under no selection pressure.</p><p>Coalescent theory is used in many fields like selection detection, estimation of
demographic parameters of real populations or disease gene mapping.</p><p>The strategy followed in the Biopython implementation of the coalescent was not
to create a new, built-in, simulator from scratch but to use an existing one,
Fastsimcoal2 (<a href="http://cmpg.unibe.ch/software/fastsimcoal2/"><span style="font-family:monospace">http://cmpg.unibe.ch/software/fastsimcoal2/</span></a>). Fastsimcoal2 allows for,
among others, population structure, multiple demographic events, simulation
of multiple types of loci (SNPs, sequences, STRs/microsatellites and RFLPs)
with recombination, diploidy multiple chromosomes or ascertainment bias. Notably
Fastsimcoal2 doesn&#X2019;t support any selection model. We recommend reading
Fastsimcoal2&#X2019;s
documentation, available in the link above.</p><p>The input for Fastsimcoal2 is a file specifying the desired demography and genome,
the output is a set of files (typically around 1000) with the simulated genomes
of a sample of individuals per subpopulation. This set of files can be used
in many ways, like to compute confidence intervals where which certain
statistics (e.g., <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub> or Tajima D) are expected to lie. Real population
genetics datasets statistics can then be compared to those confidence intervals.</p><p>Biopython coalescent code allows to create demographic scenarios and genomes and
to run Fastsimcoal2.</p>
<!--TOC subsection id="sec222" Creating scenarios-->
<h3 id="sec222" class="subsection">12.2.1&#XA0;&#XA0;Creating scenarios</h3><!--SEC END --><p>Creating a scenario involves both creating a demography and a chromosome structure.
In many cases (e.g. when doing Approximate Bayesian Computations &#X2013; ABC) it is
important to test many parameter variations (e.g. vary the effective population size,
<span style="font-style:italic">N</span><sub><span style="font-style:italic">e</span></sub>, between 10, 50, 500 and 1000 individuals). The code provided allows for
the simulation of scenarios with different demographic parameters very easily.</p><p>Below we see how we can create scenarios and then how simulate them.</p>
<!--TOC subsubsection id="sec223" Demography-->
<h4 id="sec223" class="subsubsection">12.2.1.1&#XA0;&#XA0;Demography</h4><!--SEC END --><p>A few predefined demographies are built-in, all have two shared parameters: sample size
(called sample_size on the template, see below for its use) per deme and deme size, i.e.
subpopulation size (pop_size). All demographies are available as templates where all
parameters can be varied, each template has a system name. The prefedined
demographies/templates are:</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold">Single population, constant size</span></dt><dd class="dd-description"> The standard parameters are enough to specify
it. Template name: simple.
</dd><dt class="dt-description"><span style="font-weight:bold">Single population, bottleneck</span></dt><dd class="dd-description"> As seen on figure <a href="#fig%3Abottle">12.2.1.1</a>. The parameters
are current population size (pop_size on template ne3 on figure), time of expansion,
given as the generation in the past when it occurred (expand_gen), 
effective population size during bottleneck (ne2), time of contraction
(contract_gen) and original size in the remote past (ne3). Template name: bottle.
</dd><dt class="dt-description"><span style="font-weight:bold">Island model</span></dt><dd class="dd-description"> The typical island model. The total number of demes is specified
by total_demes and the migration rate by mig. Template name island.
</dd><dt class="dt-description"><span style="font-weight:bold">Stepping stone model - 1 dimension</span></dt><dd class="dd-description"> The stepping stone model in 1 dimension,
extremes disconnected. The total number of demes is total_demes, migration rate
is mig. Template name is ssm_1d.
</dd><dt class="dt-description"><span style="font-weight:bold">Stepping stone model - 2 dimensions</span></dt><dd class="dd-description"> The stepping stone model in 2 dimensions,
extremes disconnected. The parameters are x for the horizontal dimension and y
for the vertical (being the total number of demes x times y), migration rate is mig.
Template name is ssm_2d.
</dd></dl><p>
<a id="fig:bottle"></a>
<img src="images/bottle.png">
</p><p>In our first example, we will generate a template for a single population, constant size
model with a sample size of 30 and a deme size of 500. The code for this is:</p><pre class="verbatim">from Bio.PopGen.SimCoal.Template import generate_simcoal_from_template

generate_simcoal_from_template('simple',
    [(1, [('SNP', [24, 0.0005, 0.0])])],
    [('sample_size', [30]),
    ('pop_size', [100])])
</pre><p>Executing this code snippet will generate a file on the current directory called
simple_100_300.par this file can be given as input to Fastsimcoal2 to simulate the
demography (below we will see how Biopython can take care of calling
Fastsimcoal2).</p><p>This code consists of a single function call, let&#X2019;s discuss it parameter by parameter.</p><p>The first parameter is the template id (from the list above). We are using the id
&#X2019;simple&#X2019; which is the template for a single population of constant size along time.</p><p>The second parameter is the chromosome structure. Please ignore it for now, it will be
explained in the next section.</p><p>The third parameter is a list of all required parameters (recall that the simple model
only needs sample_size and pop_size) and possible values (in this case each
parameter only has a possible value).</p><p>Now, let&#X2019;s consider an example where we want to generate several island models, and we
are interested in varying the number of demes: 10, 50 and 100 with a migration
rate of 1%. Sample size and deme
size will be the same as before. Here is the code:</p><pre class="verbatim">from Bio.PopGen.SimCoal.Template import generate_simcoal_from_template

generate_simcoal_from_template('island',
    [(1, [('SNP', [24, 0.0005, 0.0])])],
    [('sample_size', [30]),
    ('pop_size', [100]),
    ('mig', [0.01]),
    ('total_demes', [10, 50, 100])])
</pre><p>In this case, 3 files will be generated: island_100_0.01_100_30.par,
island_10_0.01_100_30.par and island_50_0.01_100_30.par. Notice the
rule to make file names: template name, followed by parameter values in
reverse order.</p><p>A few, arguably more esoteric template demographies exist (please check the
Bio/PopGen/SimCoal/data directory on Biopython source tree). Furthermore it is possible
for the user to create new templates. That functionality will be discussed in a future
version of this document.</p>
<!--TOC subsubsection id="sec224" Chromosome structure-->
<h4 id="sec224" class="subsubsection">12.2.1.2&#XA0;&#XA0;Chromosome structure</h4><!--SEC END --><p>We strongly recommend reading Fastsimcoal2 documentation to understand the full potential
available in modeling chromosome structures. In this subsection we only discuss how
to implement chromosome structures using the Biopython interface, not the underlying
Fastsimcoal2 capabilities.</p><p>We will start by implementing a single chromosome, with 24 SNPs with
a recombination rate immediately on the right of each locus of 0.0005 and a
minimum frequency of the minor allele of 0. This will be specified by the
following list (to be passed as second parameter to the function
generate_simcoal_from_template):</p><pre class="verbatim">[(1, [('SNP', [24, 0.0005, 0.0])])]
</pre><p>This is actually the chromosome structure used in the above examples.</p><p>The chromosome structure is represented by a list of chromosomes,
each chromosome (i.e., each element in the list)
is composed by a tuple (a pair): the first element
is the number of times the chromosome is to be repeated (as there
might be interest in repeating the same chromosome many times).
The second element is a list of the actual components of the chromosome.
Each element is again a pair, the first member is the locus type and
the second element the parameters for that locus type. Confused?
Before showing more examples let&#X2019;s review the example above: We have
a list with one element (thus one chromosome), the chromosome is
a single instance (therefore not to be repeated), it is composed
of 24 SNPs, with a recombination rate of 0.0005 between each
consecutive SNP, the minimum frequency of the minor allele is
0.0 (i.e, it can be absent from a certain population).</p><p>Let&#X2019;s see a more complicated example:</p><pre class="verbatim">[
  (5, [
       ('SNP', [24, 0.0005, 0.0])
      ]
  ),
  (2, [
       ('DNA', [10, 0.0, 0.00005, 0.33]),
       ('RFLP', [1, 0.0, 0.0001]),
       ('MICROSAT', [1, 0.0, 0.001, 0.0, 0.0])
      ]
  )
]
</pre><p>We start by having 5 chromosomes with the same structure as
above (i.e., 24 SNPs). We then have 2 chromosomes which
have a DNA sequence with 10 nucleotides, 0.0 recombination rate,
0.0005 mutation rate, and a transition rate of 0.33. Then we
have an RFLP with 0.0 recombination rate to the next locus and
a 0.0001 mutation rate. Finally we have a microsatellite (or STR),
with 0.0 recombination rate to the next locus (note, that as this
is a single microsatellite which has no loci following, this
recombination rate here is irrelevant), with a mutation rate
of 0.001, geometric parameter of 0.0 and a range constraint
of 0.0 (for information about this parameters please consult
the Fastsimcoal2 documentation, you can use them to simulate
various mutation models, including the typical &#X2013; for microsatellites &#X2013;
stepwise mutation model among others).</p>
<!--TOC subsection id="sec225" Running Fastsimcoal2-->
<h3 id="sec225" class="subsection">12.2.2&#XA0;&#XA0;Running Fastsimcoal2</h3><!--SEC END --><p>We now discuss how to run Fastsimcoal2 from inside Biopython. It is required
that the binary for Fastsimcoal2 is called fastsimcoal21 (or fastsimcoal21.exe on Windows
based platforms), please note that the typical name when downloading the
program is in the format fastsimcoal2_x_y. As such, when installing
Fastsimcoal2
you will need to rename of the downloaded executable so that Biopython can
find it.</p><p>It is possible to run Fastsimcoal2 on files that were not generated using the method
above (e.g., writing a parameter file by hand), but we will show an
example by creating a model using the framework presented above.</p><pre class="verbatim">from Bio.PopGen.SimCoal.Template import generate_simcoal_from_template
from Bio.PopGen.SimCoal.Controller import FastSimCoalController


generate_simcoal_from_template('simple',
    [
      (5, [
           ('SNP', [24, 0.0005, 0.0])
          ]
      ),
      (2, [
           ('DNA', [10, 0.0, 0.00005, 0.33]),
           ('RFLP', [1, 0.0, 0.0001]),
           ('MICROSAT', [1, 0.0, 0.001, 0.0, 0.0])
          ]
      )
    ],
    [('sample_size', [30]),
    ('pop_size', [100])])

ctrl = FastSimCoalController()
ctrl.run_fastsimcoal('simple_100_30.par', 50)
</pre><p>The lines of interest are the last two (plus the new import).
Firstly a controller for the
application is created. The directory where the binary is located has
to be specified.</p><p>The simulator is then run on the last line: we know, from the rules explained
above, that the input file name is simple_100_30.par for the
simulation parameter file created. We then specify
that we want to run 50 independent simulations, by default Biopython
requests a simulation of diploid data, but a third parameter can
be added to simulate haploid data (adding as a parameter the
string &#X2019;0&#X2019;). Fastsimcoal2 will now run (please
note that this can take quite a lot of time) and will create a directory
with the simulation results. The results can now be analysed (typically
studying the data with Arlequin3). In the future Biopython might support
reading the Arlequin3 format and thus allowing for the analysis of
Fastsimcoal2
data inside Biopython.</p>
<!--TOC section id="sec226" Other applications-->
<h2 id="sec226" class="section">12.3&#XA0;&#XA0;Other applications</h2><!--SEC END --><p>Here we discuss interfaces and utilities to deal with population genetics&#X2019;
applications which arguably have a smaller user base.</p>
<!--TOC subsection id="sec227" FDist: Detecting selection and molecular adaptation-->
<h3 id="sec227" class="subsection">12.3.1&#XA0;&#XA0;FDist: Detecting selection and molecular adaptation</h3><!--SEC END --><p>FDist is a selection detection application suite based on computing
(i.e. simulating) a &#X201C;neutral&#X201D; confidence interval based on <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub> and
heterozygosity. Markers (which can be SNPs, microsatellites, AFLPs
among others) which lie outside the &#X201C;neutral&#X201D; interval are to be
considered as possible candidates for being under selection.</p><p>FDist is mainly used when the number of markers is considered enough
to estimate an average <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub>, but not enough to either have outliers
calculated from the dataset directly or, with even more markers for
which the relative positions in the genome are known, to use
approaches based on, e.g., Extended Haplotype Heterozygosity (EHH).</p><p>The typical usage pattern for FDist is as follows:</p><ol class="enumerate" type=1><li class="li-enumerate">
Import a dataset from an external format into FDist format.
</li><li class="li-enumerate">Compute average <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub>. This is done by datacal inside FDist.
</li><li class="li-enumerate">Simulate &#X201C;neutral&#X201D; markers based on the
average <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub> and expected number of total populations.
This is the core operation, done by fdist inside FDist.
</li><li class="li-enumerate">Calculate the confidence interval, based on the desired
confidence boundaries (typically 95% or 99%). This is done by
cplot and is mainly used to plot the interval.
</li><li class="li-enumerate">Assess each marker status against the simulation &#X201C;neutral&#X201D;
confidence interval. Done
by pv. This is used to detect the outlier status of each marker
against the simulation.
</li></ol><p>We will now discuss each step with illustrating example code
(for this example to work FDist binaries have to be on the
executable PATH).</p><p>The FDist data format is application specific and is not used at
all by other applications, as such you will probably have to convert
your data for use with FDist. Biopython can help you do this.
Here is an example converting from GenePop format to FDist format
(along with imports that will be needed on examples further below):</p><pre class="verbatim">from Bio.PopGen import GenePop
from Bio.PopGen import FDist
from Bio.PopGen.FDist import Controller
from Bio.PopGen.FDist.Utils import convert_genepop_to_fdist

gp_rec = GenePop.read(open("example.gen"))
fd_rec = convert_genepop_to_fdist(gp_rec)
in_file = open("infile", "w")
in_file.write(str(fd_rec))
in_file.close()
</pre><p>In this code we simply parse a GenePop file and convert it to a FDist
record.</p><p>Printing an FDist record will generate
a string that can be directly saved to a file and supplied to FDist. FDist
requires the input file to be called infile, therefore we save the record on
a file with that name.</p><p>The most important fields on a FDist record are: num_pops, the number of
populations; num_loci, the number of loci and loci_data with the marker
data itself. Most probably the details of the record are of no interest
to the user, as the record only purpose is to be passed to FDist.</p><p>The next step is to calculate the average <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub> of the dataset (along
with the sample size):</p><pre class="verbatim">ctrl = Controller.FDistController()
fst, samp_size = ctrl.run_datacal()
</pre><p>On the first line we create an object to control the call of FDist
suite, this object will be used further on in order to call other
suite applications.</p><p>On the second line we call the datacal application which computes the
average <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub>
and the sample size. It is worth noting that the <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub> computed by
datacal is a <em>variation</em> of Weir and Cockerham&#X2019;s &#X3B8;.</p><p>We can now call the main fdist application in order to simulate neutral
markers.</p><pre class="verbatim">sim_fst = ctrl.run_fdist(npops = 15, nsamples = fd_rec.num_pops, fst = fst,
    sample_size = samp_size, mut = 0, num_sims = 40000)
</pre><dl class="description"><dt class="dt-description">
<span style="font-weight:bold">npops</span></dt><dd class="dd-description"> Number of populations existing in nature. This is really a
&#X201C;guestimate&#X201D;. Has to be lower than 100.
</dd><dt class="dt-description"><span style="font-weight:bold">nsamples</span></dt><dd class="dd-description"> Number of populations sampled, has to be lower than npops.
</dd><dt class="dt-description"><span style="font-weight:bold">fst</span></dt><dd class="dd-description"> Average <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub>.
</dd><dt class="dt-description"><span style="font-weight:bold">sample_size</span></dt><dd class="dd-description"> Average number of individuals sampled on each population.
</dd><dt class="dt-description"><span style="font-weight:bold">mut</span></dt><dd class="dd-description"> Mutation model: 0 - Infinite alleles; 1 - Stepwise mutations
</dd><dt class="dt-description"><span style="font-weight:bold">num_sims</span></dt><dd class="dd-description"> Number of simulations to perform. Typically a number around
40000 will be OK, but if you get a confidence interval that looks sharp
(this can be detected when plotting the confidence interval computed
below) the value can be increased (a suggestion would be steps of 10000
simulations).
</dd></dl><p>The confusion in wording between number of samples and sample size
stems from the original application.</p><p>A file named out.dat will be created with the simulated heterozygosities
and <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub>s, it will have as many lines as the number of simulations
requested.</p><p>Note that fdist returns the average <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub> that it was <em>capable</em> of
simulating, for more details about this issue please read below the paragraph
on approximating the desired average <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub>.</p><p>The next (optional) step is to calculate the confidence interval:</p><pre class="verbatim">cpl_interval = ctrl.run_cplot(ci=0.99)
</pre><p>You can only call cplot after having run fdist.</p><p>This will calculate the confidence intervals (99% in this case)
for a previous fdist run. A list of quadruples is returned. The
first element represents the heterozygosity, the second the lower
bound of <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub> confidence interval for that heterozygosity,
the third the average and the fourth the upper bound. This can
be used to trace the confidence interval contour. This list
is also written to a file, out.cpl.</p><p>The main purpose of this step is return a set of points which can
be easily used to plot a confidence interval. It can be skipped
if the objective is only to assess the status of each marker against
the simulation, which is the next step...</p><pre class="verbatim">pv_data = ctrl.run_pv()
</pre><p>You can only call cplot after having run datacal and fdist.</p><p>This will use the simulated markers to assess the status of each
individual real marker. A list, in the same order than the loci_list
that is on the FDist record (which is in the same order that the GenePop
record) is returned. Each element in the list is a quadruple, the
fundamental member of each quadruple is the last element (regarding the
other elements, please refer to the pv documentation &#X2013; for the
sake of simplicity we will not discuss them here) which returns the
probability of the simulated <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub> being lower than the marker <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub>.
Higher values would indicate a stronger candidate for positive selection,
lower values a candidate for balancing selection, and intermediate values
a possible neutral marker. What is &#X201C;higher&#X201D;, &#X201C;lower&#X201D; or &#X201C;intermediate&#X201D;
is really a subjective issue, but taking a &#X201C;confidence interval&#X201D; approach
and considering a 95% confidence interval, &#X201C;higher&#X201D; would be between 0.95
and 1.0, &#X201C;lower&#X201D; between 0.0 and 0.05 and &#X201C;intermediate&#X201D; between 0.05 and
0.95.</p>
<!--TOC subsubsection id="sec228" Approximating the desired average <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub>-->
<h4 id="sec228" class="subsubsection">12.3.1.1&#XA0;&#XA0;Approximating the desired average <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub></h4><!--SEC END --><p>Fdist tries to approximate the desired average <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub> by doing a
coalescent simulation using migration rates based on the formula</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">N</span><sub><span style="font-style:italic">m</span></sub>&#XA0;=&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center">1&#XA0;&#X2212;&#XA0;<span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub></td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center">4<span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub></td></tr>
</table></td><td class="dcell">&#XA0;</td></tr>
</table><p>This formula assumes a few premises like an infinite number of populations.</p><p>In practice, when the number of populations is low, the mutation model
is stepwise and the sample size increases, fdist will not be able to
simulate an acceptable approximate average <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub>.</p><p>To address that, a function is provided to iteratively approach the desired
value by running several fdists in sequence. This approach is computationally
more intensive than running a single fdist run, but yields good results.
The following code runs fdist approximating the desired <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub>:</p><pre class="verbatim">sim_fst = ctrl.run_fdist_force_fst(npops = 15, nsamples = fd_rec.num_pops,
    fst = fst, sample_size = samp_size, mut = 0, num_sims = 40000,
    limit = 0.05)
</pre><p>The only new optional parameter, when comparing with run_fdist, is limit
which is the desired maximum error. run_fdist can (and probably should)
be safely replaced with run_fdist_force_fst.</p>
<!--TOC subsubsection id="sec229" Final notes-->
<h4 id="sec229" class="subsubsection">12.3.1.2&#XA0;&#XA0;Final notes</h4><!--SEC END --><p>The process to determine the average <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub> can be more sophisticated than
the one presented here. For more information we refer you to the FDist
README file. Biopython&#X2019;s code can be used to implement more sophisticated
approaches.</p>
<!--TOC section id="sec230" Future Developments-->
<h2 id="sec230" class="section">12.4&#XA0;&#XA0;Future Developments</h2><!--SEC END --><p>The most desired future developments would be the ones you add yourself ;) .</p><p>That being said, already existing fully functional code is currently being
incorporated in Bio.PopGen, that code covers the applications FDist and
SimCoal2, the HapMap and UCSC Table Browser databases and some simple statistics
like <span style="font-style:italic">F</span><sub><span style="font-style:italic">st</span></sub>, or allele counts.</p>
<!--TOC chapter id="sec231" Phylogenetics with Bio.Phylo-->
<h1 id="sec231" class="chapter">Chapter&#XA0;13&#XA0;&#XA0;Phylogenetics with Bio.Phylo</h1><!--SEC END --><p>
<a id="sec:Phylo"></a></p><p>The Bio.Phylo module was introduced in Biopython 1.54. Following the lead of SeqIO and AlignIO,
it aims to provide a common way to work with phylogenetic trees independently of the source data
format, as well as a consistent API for I/O operations.</p><p>Bio.Phylo is described in an open-access journal article [<a href="#talevich2012">9</a>, Talevich
<span style="font-style:italic">et al.</span>, 2012], which you might also find helpful.</p>
<!--TOC section id="sec232" Demo: What&#X2019;s in a Tree?-->
<h2 id="sec232" class="section">13.1&#XA0;&#XA0;Demo: What&#X2019;s in a Tree?</h2><!--SEC END --><p>To get acquainted with the module, let&#X2019;s start with a tree that we&#X2019;ve already constructed, and
inspect it a few different ways. Then we&#X2019;ll colorize the branches, to use a special phyloXML
feature, and finally save it.</p><p>Create a simple Newick file named <span style="font-family:monospace">simple.dnd</span> using your favorite text editor,
or use <a href="http://biopython.org/SRC/biopython/Doc/examples/simple.dnd"><span style="font-family:monospace">simple.dnd</span></a>
provided with the Biopython source code:</p><pre class="verbatim">(((A,B),(C,D)),(E,F,G));
</pre><p>This tree has no branch lengths, only a topology and labelled terminals. (If you have a real
tree file available, you can follow this demo using that instead.)</p><p>Launch the Python interpreter of your choice:</p><pre class="verbatim">% ipython -pylab
</pre><p>For interactive work, launching the IPython interpreter with the <code>-pylab</code> flag enables
<span style="font-weight:bold">matplotlib</span> integration, so graphics will pop up automatically. We&#X2019;ll use that during
this demo.</p><p>Now, within Python, read the tree file, giving the file name and the name of the format.</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Phylo
&gt;&gt;&gt; tree = Phylo.read("simple.dnd", "newick")
</pre><p>Printing the tree object as a string gives us a look at the entire object hierarchy.</p><pre class="verbatim">&gt;&gt;&gt; print(tree)
Tree(rooted=False, weight=1.0)
    Clade(branch_length=1.0)
        Clade(branch_length=1.0)
            Clade(branch_length=1.0)
                Clade(branch_length=1.0, name='A')
                Clade(branch_length=1.0, name='B')
            Clade(branch_length=1.0)
                Clade(branch_length=1.0, name='C')
                Clade(branch_length=1.0, name='D')
        Clade(branch_length=1.0)
            Clade(branch_length=1.0, name='E')
            Clade(branch_length=1.0, name='F')
            Clade(branch_length=1.0, name='G')
</pre><p>The <span style="font-family:monospace">Tree</span> object contains global information about the tree, such as whether it&#X2019;s
rooted or unrooted. It has one root clade, and under that, it&#X2019;s nested lists of clades all the
way down to the tips.</p><p>The function <code>draw_ascii</code> creates a simple ASCII-art (plain text) dendrogram. This is a
convenient visualization for interactive exploration, in case better graphical tools aren&#X2019;t
available.</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Phylo
&gt;&gt;&gt; tree = Phylo.read("simple.dnd", "newick")
&gt;&gt;&gt; Phylo.draw_ascii(tree)
                                                          __________________ A
                                       __________________|
                                      |                  |__________________ B
                    __________________|
                   |                  |                   __________________ C
                   |                  |__________________|
___________________|                                     |__________________ D
                   |
                   |                   __________________ E
                   |                  |
                   |__________________|__________________ F
                                      |
                                      |__________________ G
&lt;BLANKLINE&gt;
</pre><p>If you have <span style="font-weight:bold">matplotlib</span> or <span style="font-weight:bold">pylab</span> installed, you can create a graphic
using the <code>draw</code> function (see Fig. <a href="#fig%3Aphylo-simple-draw">13.1</a>):</p><pre class="verbatim">&gt;&gt;&gt; tree.rooted = True
&gt;&gt;&gt; Phylo.draw(tree)
</pre><p>
<img src="images/phylo-simple-draw.png" width=666, height=530>
<a id="fig:phylo-simple-draw"></a>
</p>
<!--TOC subsection id="sec233" Coloring branches within a tree-->
<h3 id="sec233" class="subsection">13.1.1&#XA0;&#XA0;Coloring branches within a tree</h3><!--SEC END --><p>The functions <code>draw</code> and <code>draw_graphviz</code> support the display of different
colors and branch widths in a tree.
As of Biopython 1.59, the <code>color</code> and <code>width</code> attributes are available on the
basic Clade object and there&#X2019;s nothing extra required to use them.
Both attributes refer to the branch leading the given clade, and apply recursively, so
all descendent branches will also inherit the assigned width and color values during
display.</p><p>In earlier versions of Biopython, these were special features of PhyloXML trees, and
using the attributes required first converting the tree to a subclass of the basic tree
object called Phylogeny, from the Bio.Phylo.PhyloXML module.</p><p>In Biopython 1.55 and later, this is a convenient tree method:</p><pre class="verbatim">&gt;&gt;&gt; tree = tree.as_phyloxml()
</pre><p>In Biopython 1.54, you can accomplish the same thing with one extra import:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Phylo.PhyloXML import Phylogeny
&gt;&gt;&gt; tree = Phylogeny.from_tree(tree)
</pre><p>Note that the file formats Newick and Nexus don&#X2019;t support branch colors or widths, so
if you use these attributes in Bio.Phylo, you will only be able to save the values in
PhyloXML format. (You can still save a tree as Newick or Nexus, but the color and width
values will be skipped in the output file.)</p><p>Now we can begin assigning colors.
First, we&#X2019;ll color the root clade gray. We can do that by assigning the 24-bit color
value as an RGB triple, an HTML-style hex string, or the name of one of the predefined
colors.</p><pre class="verbatim">&gt;&gt;&gt; tree.root.color = (128, 128, 128)
</pre><p>Or:</p><pre class="verbatim">&gt;&gt;&gt; tree.root.color = "#808080"
</pre><p>Or:</p><pre class="verbatim">&gt;&gt;&gt; tree.root.color = "gray"
</pre><p>Colors for a clade are treated as cascading down through the entire clade, so when we colorize
the root here, it turns the whole tree gray. We can override that by assigning a different
color lower down on the tree.</p><p>Let&#X2019;s target the most recent common ancestor (MRCA) of the nodes named &#X201C;E&#X201D; and &#X201C;F&#X201D;. The
<code>common_ancestor</code> method returns a reference to that clade in the original tree, so when
we color that clade &#X201C;salmon&#X201D;, the color will show up in the original tree.</p><pre class="verbatim">&gt;&gt;&gt; mrca = tree.common_ancestor({"name": "E"}, {"name": "F"})
&gt;&gt;&gt; mrca.color = "salmon"
</pre><p>If we happened to know exactly where a certain clade is in the tree, in terms of nested list
entries, we can jump directly to that position in the tree by indexing it. Here, the index
<code>[0,1]</code> refers to the second child of the first child of the root.</p><pre class="verbatim">&gt;&gt;&gt; tree.clade[0, 1].color = "blue"
</pre><p>Finally, show our work (see Fig. <a href="#fig%3Aphylo-color-draw">13.1.1</a>):</p><pre class="verbatim">&gt;&gt;&gt; Phylo.draw(tree)
</pre><p>
<img src="images/phylo-color-draw.png" width=666, height=530>
<a id="fig:phylo-color-draw"></a>
</p><p>Note that a clade&#X2019;s color includes the branch leading to that clade, as well as its
descendents. The common ancestor of E and F turns out to be just under the root, and with this
coloring we can see exactly where the root of the tree is.</p><p>My, we&#X2019;ve accomplished a lot! Let&#X2019;s take a break here and save our work. Call the
<span style="font-family:monospace">write</span> function with a file name or handle &#X2014; here we use standard output, to see what
would be written &#X2014; and the format <span style="font-family:monospace">phyloxml</span>. PhyloXML saves the colors we assigned,
so you can open this phyloXML file in another tree viewer like Archaeopteryx, and the colors
will show up there, too.</p><pre class="verbatim">&gt;&gt;&gt; import sys
&gt;&gt;&gt; Phylo.write(tree, sys.stdout, "phyloxml")

&lt;phy:phyloxml xmlns:phy="http://www.phyloxml.org"&gt;
  &lt;phy:phylogeny rooted="true"&gt;
    &lt;phy:clade&gt;
      &lt;phy:branch_length&gt;1.0&lt;/phy:branch_length&gt;
      &lt;phy:color&gt;
        &lt;phy:red&gt;128&lt;/phy:red&gt;
        &lt;phy:green&gt;128&lt;/phy:green&gt;
        &lt;phy:blue&gt;128&lt;/phy:blue&gt;
      &lt;/phy:color&gt;
      &lt;phy:clade&gt;
        &lt;phy:branch_length&gt;1.0&lt;/phy:branch_length&gt;
        &lt;phy:clade&gt;
          &lt;phy:branch_length&gt;1.0&lt;/phy:branch_length&gt;
          &lt;phy:clade&gt;
            &lt;phy:name&gt;A&lt;/phy:name&gt;
            ...
</pre><p>The rest of this chapter covers the core functionality of Bio.Phylo in greater detail. For more
examples of using Bio.Phylo, see the cookbook page on Biopython.org:</p><p><a href="http://biopython.org/wiki/Phylo_cookbook"><span style="font-family:monospace">http://biopython.org/wiki/Phylo_cookbook</span></a></p>
<!--TOC section id="sec234" I/O functions-->
<h2 id="sec234" class="section">13.2&#XA0;&#XA0;I/O functions</h2><!--SEC END --><p>Like SeqIO and AlignIO, Phylo handles file input and output through four functions:
<code>parse</code>, <code>read</code>, <code>write</code> and <code>convert</code>,
all of which support the tree file formats Newick, NEXUS, phyloXML and NeXML, as
well as the Comparative Data Analysis Ontology (CDAO).</p><p>The <code>read</code> function parses a single tree in the given file and returns it. Careful; it
will raise an error if the file contains more than one tree, or no trees.</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Phylo
&gt;&gt;&gt; tree = Phylo.read("Tests/Nexus/int_node_labels.nwk", "newick")
&gt;&gt;&gt; print(tree)
</pre><p>(Example files are available in the <span style="font-family:monospace">Tests/Nexus/</span> and <span style="font-family:monospace">Tests/PhyloXML/</span>
directories of the Biopython distribution.)</p><p>To handle multiple (or an unknown number of) trees, use the <code>parse</code> function iterates
through each of the trees in the given file: </p><pre class="verbatim">&gt;&gt;&gt; trees = Phylo.parse("Tests/PhyloXML/phyloxml_examples.xml", "phyloxml")
&gt;&gt;&gt; for tree in trees:
...     print(tree)
</pre><p>Write a tree or iterable of trees back to file with the <code>write</code> function:</p><pre class="verbatim">&gt;&gt;&gt; trees = list(Phylo.parse("phyloxml_examples.xml", "phyloxml"))
&gt;&gt;&gt; tree1 = trees[0]
&gt;&gt;&gt; others = trees[1:]
&gt;&gt;&gt; Phylo.write(tree1, "tree1.xml", "phyloxml")
1
&gt;&gt;&gt; Phylo.write(others, "other_trees.xml", "phyloxml")
12
</pre><p>Convert files between any of the supported formats with the <code>convert</code> function:</p><pre class="verbatim">&gt;&gt;&gt; Phylo.convert("tree1.dnd", "newick", "tree1.xml", "nexml")
1
&gt;&gt;&gt; Phylo.convert("other_trees.xml", "phyloxml", "other_trees.nex", "nexus")
12
</pre><p>To use strings as input or output instead of actual files, use <code>StringIO</code> as you would
with SeqIO and AlignIO:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Phylo
&gt;&gt;&gt; from StringIO import StringIO
&gt;&gt;&gt; handle = StringIO("(((A,B),(C,D)),(E,F,G));")
&gt;&gt;&gt; tree = Phylo.read(handle, "newick")
</pre>
<!--TOC section id="sec235" View and export trees-->
<h2 id="sec235" class="section">13.3&#XA0;&#XA0;View and export trees</h2><!--SEC END --><p>The simplest way to get an overview of a <code>Tree</code> object is to <code>print</code> it:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Phylo
&gt;&gt;&gt; tree = Phylo.read("PhyloXML/example.xml", "phyloxml")
&gt;&gt;&gt; print(tree)
Phylogeny(description='phyloXML allows to use either a "branch_length" attribute...', name='example from Prof. Joe Felsenstein's book "Inferring Phyl...', rooted=True)
    Clade()
        Clade(branch_length=0.06)
            Clade(branch_length=0.102, name='A')
            Clade(branch_length=0.23, name='B')
        Clade(branch_length=0.4, name='C')
</pre><p>This is essentially an outline of the object hierarchy Biopython uses to represent a tree. But
more likely, you&#X2019;d want to see a drawing of the tree. There are three functions to do this.</p><p>As we saw in the demo, <code>draw_ascii</code> prints an ascii-art drawing of the tree (a
rooted phylogram) to standard output, or an open file handle if given. Not all of the
available information about the tree is shown, but it provides a way to quickly view the
tree without relying on any external dependencies.</p><pre class="verbatim">&gt;&gt;&gt; tree = Phylo.read("example.xml", "phyloxml")
&gt;&gt;&gt; Phylo.draw_ascii(tree)
             __________________ A
  __________|
_|          |___________________________________________ B
 |
 |___________________________________________________________________________ C

</pre><p>The <code>draw</code> function draws a more attractive image using the matplotlib
library. See the API documentation for details on the arguments it accepts to
customize the output.</p><pre class="verbatim">&gt;&gt;&gt; tree = Phylo.read("example.xml", "phyloxml")
&gt;&gt;&gt; Phylo.draw(tree, branch_labels=lambda c: c.branch_length)
</pre><p>
<img src="images/phylo-draw-example.png" width=701, height=465>
<a id="fig:phylo-draw-example"></a>
</p><p><code>draw_graphviz</code> draws an unrooted cladogram, but requires that you have Graphviz,
PyDot or PyGraphviz, NetworkX, and matplotlib (or pylab) installed. Using the same example as
above, and the <code>dot</code> program included with Graphviz, let&#X2019;s draw a rooted tree (see
Fig.&#XA0;<a href="#fig%3Aphylo-dot">13.3</a>):</p><pre class="verbatim">&gt;&gt;&gt; tree = Phylo.read("example.xml", "phyloxml")
&gt;&gt;&gt; Phylo.draw_graphviz(tree, prog='dot')
&gt;&gt;&gt; import pylab
&gt;&gt;&gt; pylab.show()                    # Displays the tree in an interactive viewer
&gt;&gt;&gt; pylab.savefig('phylo-dot.png')  # Creates a PNG file of the same graphic
</pre><p>
<img src="images/phylo-dot.png" width=400, height=312>
<a id="fig:phylo-dot"></a>
</p><p>(Tip: If you execute IPython with the <code>-pylab</code> option, calling <code>draw_graphviz</code> causes
the matplotlib viewer to launch automatically without manually calling <code>show()</code>.)</p><p>This exports the tree object to a NetworkX graph, uses Graphviz to lay out the nodes, and
displays it using matplotlib. 
There are a number of keyword arguments that can modify the resulting diagram, including
most of those accepted by the NetworkX functions <code>networkx.draw</code> and
<code>networkx.draw_graphviz</code>.</p><p>The display is also affected by the <code>rooted</code> attribute of the given tree object.
Rooted trees are shown with a &#X201C;head&#X201D; on each branch indicating direction (see
Fig.&#XA0;<a href="#fig%3Aphylo-rooted">13.3</a>):</p><pre class="verbatim">&gt;&gt;&gt; tree = Phylo.read("simple.dnd", "newick")
&gt;&gt;&gt; tree.rooted = True
&gt;&gt;&gt; Phylo.draw_graphviz(tree)
</pre><p>
<img src="images/phylo-rooted.png" width=432, height=368>
<a id="fig:phylo-rooted"></a>
</p><p>The &#X201C;prog&#X201D; argument specifies the Graphviz engine used for layout. The default,
<code>twopi</code>, behaves well for any size tree, reliably avoiding crossed branches. The
<code>neato</code> program may draw more attractive moderately-sized trees, but sometimes will
cross branches (see Fig.&#XA0;<a href="#fig%3Aphylo-color">13.3</a>). The <code>dot</code> program may be useful
with small trees, but tends to do surprising things with the layout of larger trees.</p><pre class="verbatim">&gt;&gt;&gt; Phylo.draw_graphviz(tree, prog="neato")
</pre><p>
<img src="images/phylo-color.png" width=499, height=348>
<a id="fig:phylo-color"></a>
</p><p>This viewing mode is particularly handy for exploring larger trees, because the matplotlib
viewer can zoom in on a selected region, thinning out a cluttered graphic.
</p><pre class="verbatim">&gt;&gt;&gt; tree = Phylo.read("apaf.xml", "phyloxml")
&gt;&gt;&gt; Phylo.draw_graphviz(tree, prog="neato", node_size=0)
</pre><p>
<img src="images/phylo-apaf.png" width=519, height=400>
<a id="fig:phylo-apaf"></a>
<img src="images/phylo-apaf-zoom.png" width=506, height=400>
<a id="fig:phylo-apaf-zoom"></a>
</p><p>Note that branch lengths are not displayed accurately, because Graphviz ignores them when
creating the node layouts. The branch lengths are retained when exporting a tree as a NetworkX
graph object (<code>to_networkx</code>), however.</p><p>See the Phylo page on the Biopython wiki (<a href="http://biopython.org/wiki/Phylo"><span style="font-family:monospace">http://biopython.org/wiki/Phylo</span></a>) for
descriptions and examples of the more advanced functionality in <code>draw_ascii</code>,
<code>draw_graphviz</code> and <code>to_networkx</code>.</p>
<!--TOC section id="sec236" Using Tree and Clade objects-->
<h2 id="sec236" class="section">13.4&#XA0;&#XA0;Using Tree and Clade objects</h2><!--SEC END --><p>The <code>Tree</code> objects produced by <code>parse</code> and <code>read</code> are containers for recursive
sub-trees, attached to the <code>Tree</code> object at the <code>root</code> attribute (whether or not the
phylogenic tree is actually considered rooted). A <code>Tree</code> has globally applied information
for the phylogeny, such as rootedness, and a reference to a single <code>Clade</code>; a
<code>Clade</code> has node- and clade-specific information, such as branch length, and a list of
its own descendent <code>Clade</code> instances, attached at the <code>clades</code> attribute.</p><p>So there is a distinction between <code>tree</code> and <code>tree.root</code>. In practice, though, you
rarely need to worry about it. To smooth over the difference, both <code>Tree</code> and
<code>Clade</code> inherit from <code>TreeMixin</code>, which contains the implementations for methods
that would be commonly used to search, inspect or modify a tree or any of its clades. This
means that almost all of the methods supported by <code>tree</code> are also available on
<code>tree.root</code> and any clade below it. (<code>Clade</code> also has a <code>root</code> property, which
returns the clade object itself.)</p>
<!--TOC subsection id="sec237" Search and traversal methods-->
<h3 id="sec237" class="subsection">13.4.1&#XA0;&#XA0;Search and traversal methods</h3><!--SEC END --><p>For convenience, we provide a couple of simplified methods that return all external or internal
nodes directly as a list:</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold"><span style="font-family:monospace">get_terminals</span></span></dt><dd class="dd-description"> makes a list of all of this tree&#X2019;s terminal (leaf) nodes.
</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">get_nonterminals</span></span></dt><dd class="dd-description"> makes a list of all of this tree&#X2019;s nonterminal (internal)
nodes.
</dd></dl><p>These both wrap a method with full control over tree traversal, <code>find_clades</code>. Two more
traversal methods, <code>find_elements</code> and <code>find_any</code>, rely on the same core
functionality and accept the same arguments, which we&#X2019;ll call a &#X201C;target specification&#X201D; for
lack of a better description. These specify which objects in the tree will be matched and
returned during iteration. The first argument can be any of the following types:</p><ul class="itemize"><li class="li-itemize">
A <span style="font-weight:bold">TreeElement instance</span>, which tree elements will match by identity &#X2014; so
searching with a Clade instance as the target will find that clade in the tree;</li><li class="li-itemize">A <span style="font-weight:bold">string</span>, which matches tree elements&#X2019; string representation &#X2014; in
particular, a clade&#X2019;s <code>name</code> <span style="font-style:italic">(added in Biopython 1.56)</span>;</li><li class="li-itemize">A <span style="font-weight:bold">class</span> or <span style="font-weight:bold">type</span>, where every tree element of the same type (or
sub-type) will be matched;</li><li class="li-itemize">A <span style="font-weight:bold">dictionary</span> where keys are tree element attributes and values are matched to the
corresponding attribute of each tree element. This one gets even more elaborate:<ul class="itemize"><li class="li-itemize">
If an <span style="font-family:monospace">int</span> is given, it matches numerically equal attributes, e.g. 1 will
match 1 or 1.0</li><li class="li-itemize">If a boolean is given (True or False), the corresponding attribute value is
evaluated as a boolean and checked for the same</li><li class="li-itemize"><span style="font-family:monospace">None</span> matches <span style="font-family:monospace">None</span></li><li class="li-itemize">If a string is given, the value is treated as a regular expression (which must
match the whole string in the corresponding element attribute, not just a prefix). A
given string without special regex characters will match string attributes exactly, so
if you don&#X2019;t use regexes, don&#X2019;t worry about it. For example, in a tree with clade
names Foo1, Foo2 and Foo3, <code>tree.find_clades({"name": "Foo1"})</code> matches Foo1,
<code>{"name": "Foo.*"}</code> matches all three clades, and <code>{"name": "Foo"}</code> doesn&#X2019;t
match anything.</li></ul><p>Since floating-point arithmetic can produce some strange behavior, we don&#X2019;t support
matching <span style="font-family:monospace">float</span>s directly. Instead, use the boolean <span style="font-family:monospace">True</span> to match every
element with a nonzero value in the specified attribute, then filter on that attribute
manually with an inequality (or exact number, if you like living dangerously).</p><p>If the dictionary contains multiple entries, a matching element must match each of the
given attribute values &#X2014; think &#X201C;and&#X201D;, not &#X201C;or&#X201D;.</p></li><li class="li-itemize">A <span style="font-weight:bold">function</span> taking a single argument (it will be applied to each element in the
tree), returning True or False. For convenience, LookupError, AttributeError and ValueError
are silenced, so this provides another safe way to search for floating-point values in the
tree, or some more complex characteristic.</li></ul><p>After the target, there are two optional keyword arguments:</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold">terminal</span></dt><dd class="dd-description"> &#X2014; A boolean value to select for or against terminal clades (a.k.a. leaf
nodes): True searches for only terminal clades, False for non-terminal (internal) clades,
and the default, None, searches both terminal and non-terminal clades, as well as any tree
elements lacking the <code>is_terminal</code> method.</dd><dt class="dt-description"><span style="font-weight:bold">order</span></dt><dd class="dd-description"> &#X2014; Tree traversal order: <span style="font-family:monospace">"preorder"</span> (default) is depth-first search,
<span style="font-family:monospace">"postorder"</span> is DFS with child nodes preceding parents, and <span style="font-family:monospace">"level"</span> is
breadth-first search.</dd></dl><p>Finally, the methods accept arbitrary keyword arguments which are treated the same way as a
dictionary target specification: keys indicate the name of the element attribute to search for,
and the argument value (string, integer, None or boolean) is compared to the value of each
attribute found. If no keyword arguments are given, then any TreeElement types are matched.
The code for this is generally shorter than passing a dictionary as the target specification:
<code>tree.find_clades({"name": "Foo1"})</code> can be shortened to
<code>tree.find_clades(name="Foo1")</code>.</p><p>(In Biopython 1.56 or later, this can be even shorter: <code>tree.find_clades("Foo1")</code>)</p><p>Now that we&#X2019;ve mastered target specifications, here are the methods used to traverse a tree:</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold"><span style="font-family:monospace">find_clades</span></span></dt><dd class="dd-description">
Find each clade containing a matching element. That is, find each element as with
<code>find_elements</code>, but return the corresponding clade object. (This is usually what you
want.)<p>The result is an iterable through all matching objects, searching depth-first by default.
This is not necessarily the same order as the elements appear in the Newick, Nexus or XML
source file!</p></dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">find_elements</span></span></dt><dd class="dd-description">
Find all tree elements matching the given attributes, and return the matching elements
themselves. Simple Newick trees don&#X2019;t have complex sub-elements, so this behaves the same
as <code>find_clades</code> on them. PhyloXML trees often do have complex objects attached to
clades, so this method is useful for extracting those.</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">find_any</span></span></dt><dd class="dd-description">
Return the first element found by <code>find_elements()</code>, or None. This is also useful for
checking whether any matching element exists in the tree, and can be used in a conditional.</dd></dl><p>Two more methods help navigating between nodes in the tree:</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold"><span style="font-family:monospace">get_path</span></span></dt><dd class="dd-description">
List the clades directly between the tree root (or current clade) and the given target.
Returns a list of all clade objects along this path, ending with the given target, but
excluding the root clade.</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">trace</span></span></dt><dd class="dd-description">
List of all clade object between two targets in this tree. Excluding start, including
finish.</dd></dl>
<!--TOC subsection id="sec238" Information methods-->
<h3 id="sec238" class="subsection">13.4.2&#XA0;&#XA0;Information methods</h3><!--SEC END --><p>These methods provide information about the whole tree (or any clade).</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold"><span style="font-family:monospace">common_ancestor</span></span></dt><dd class="dd-description">
Find the most recent common ancestor of all the given targets. (This will be a Clade object). 
If no target is given, returns the root of the current clade (the one this method is called
from); if 1 target is given, this returns the target itself. However, if any of the
specified targets are not found in the current tree (or clade), an exception is raised.</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">count_terminals</span></span></dt><dd class="dd-description">
Counts the number of terminal (leaf) nodes within the tree.</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">depths</span></span></dt><dd class="dd-description">
Create a mapping of tree clades to depths. The result is a dictionary where the keys are
all of the Clade instances in the tree, and the values are the distance from the root to
each clade (including terminals). By default the distance is the cumulative branch length
leading to the clade, but with the <code>unit_branch_lengths=True</code> option, only the number
of branches (levels in the tree) is counted.</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">distance</span></span></dt><dd class="dd-description">
Calculate the sum of the branch lengths between two targets. If only one target is
specified, the other is the root of this tree.</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">total_branch_length</span></span></dt><dd class="dd-description">
Calculate the sum of all the branch lengths in this tree. This is usually just called the
&#X201C;length&#X201D; of the tree in phylogenetics, but we use a more explicit name to avoid confusion
with Python terminology.</dd></dl><p>The rest of these methods are boolean checks:</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold"><span style="font-family:monospace">is_bifurcating</span></span></dt><dd class="dd-description">
True if the tree is strictly bifurcating; i.e. all nodes have either 2 or 0 children
(internal or external, respectively). The root may have 3 descendents and still be
considered part of a bifurcating tree.</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">is_monophyletic</span></span></dt><dd class="dd-description">
Test if all of the given targets comprise a complete subclade &#X2014; i.e., there
exists a clade such that its terminals are the same set as the given targets. The targets
should be terminals of the tree. For convenience, this method returns the common ancestor
(MCRA) of the targets if they are monophyletic (instead of the value <code>True</code>), and
<code>False</code> otherwise.</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">is_parent_of</span></span></dt><dd class="dd-description"> True if target is a descendent of this tree &#X2014; not required
to be a direct descendent. To check direct descendents of a clade, simply use list
membership testing: <code>if subclade in clade: ...</code></dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">is_preterminal</span></span></dt><dd class="dd-description"> True if all direct descendents are terminal; False if any
direct descendent is not terminal.</dd></dl>
<!--TOC subsection id="sec239" Modification methods-->
<h3 id="sec239" class="subsection">13.4.3&#XA0;&#XA0;Modification methods</h3><!--SEC END --><p>These methods modify the tree in-place. If you want to keep the original tree intact, make a
complete copy of the tree first, using Python&#X2019;s <span style="font-family:monospace">copy</span> module:</p><pre class="verbatim">tree = Phylo.read('example.xml', 'phyloxml')
import copy
newtree = copy.deepcopy(tree)
</pre><dl class="description"><dt class="dt-description">
<span style="font-weight:bold"><span style="font-family:monospace">collapse</span></span></dt><dd class="dd-description">
Deletes the target from the tree, relinking its children to its parent.</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">collapse_all</span></span></dt><dd class="dd-description">
Collapse all the descendents of this tree, leaving only terminals. Branch lengths are
preserved, i.e. the distance to each terminal stays the same. With a target specification
(see above), collapses only the internal nodes matching the specification.</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">ladderize</span></span></dt><dd class="dd-description">
Sort clades in-place according to the number of terminal nodes. Deepest clades are placed
last by default. Use <code>reverse=True</code> to sort clades deepest-to-shallowest.</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">prune</span></span></dt><dd class="dd-description">
Prunes a terminal clade from the tree. If taxon is from a bifurcation, the connecting node
will be collapsed and its branch length added to remaining terminal node. This might no
longer be a meaningful value.</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">root_with_outgroup</span></span></dt><dd class="dd-description">
Reroot this tree with the outgroup clade containing the given targets, i.e. the common
ancestor of the outgroup. This method is only available on Tree objects, not Clades.<p>If the outgroup is identical to self.root, no change occurs. If the outgroup clade is
terminal (e.g. a single terminal node is given as the outgroup), a new bifurcating root
clade is created with a 0-length branch to the given outgroup. Otherwise, the internal node
at the base of the outgroup becomes a trifurcating root for the whole tree. If the original
root was bifurcating, it is dropped from the tree.</p><p>In all cases, the total branch length of the tree stays the same.</p></dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">root_at_midpoint</span></span></dt><dd class="dd-description">
Reroot this tree at the calculated midpoint between the two most distant
tips of the tree. (This uses <code>root_with_outgroup</code> under the hood.)</dd><dt class="dt-description"><span style="font-weight:bold"><span style="font-family:monospace">split</span></span></dt><dd class="dd-description">
Generate <span style="font-style:italic">n</span> (default 2) new descendants. In a species tree, this is a speciation
event. New clades have the given <code>branch_length</code> and the same name as this clade&#X2019;s
root plus an integer suffix (counting from 0) &#X2014; for example, splitting a clade named
&#X201C;A&#X201D; produces the sub-clades &#X201C;A0&#X201D; and &#X201C;A1&#X201D;.</dd></dl><p>See the Phylo page on the Biopython wiki (<a href="http://biopython.org/wiki/Phylo"><span style="font-family:monospace">http://biopython.org/wiki/Phylo</span></a>) for
more examples of using the available methods.</p>
<!--TOC subsection id="sec240" Features of PhyloXML trees-->
<h3 id="sec240" class="subsection">13.4.4&#XA0;&#XA0;Features of PhyloXML trees</h3><!--SEC END --><p>
<a id="sec:PhyloXML"></a></p><p>The phyloXML file format includes fields for annotating trees with additional data types and
visual cues.</p><p>See the PhyloXML page on the Biopython wiki (<a href="http://biopython.org/wiki/PhyloXML"><span style="font-family:monospace">http://biopython.org/wiki/PhyloXML</span></a>) for
descriptions and examples of using the additional annotation features provided by PhyloXML.</p>
<!--TOC section id="sec241" Running external applications-->
<h2 id="sec241" class="section">13.5&#XA0;&#XA0;Running external applications</h2><!--SEC END --><p>
<a id="sec:PhyloApps"></a></p><p>While Bio.Phylo doesn&#X2019;t infer trees from alignments itself, there are third-party
programs available that do. These are supported through the module
<span style="font-family:monospace">Bio.Phylo.Applications</span>, using the same general framework as
<span style="font-family:monospace">Bio.Emboss.Applications</span>, <span style="font-family:monospace">Bio.Align.Applications</span> and others.</p><p>Biopython 1.58 introduced a wrapper for PhyML
(<a href="http://www.atgc-montpellier.fr/phyml/"><span style="font-family:monospace">http://www.atgc-montpellier.fr/phyml/</span></a>). The program accepts an input alignment in
<span style="font-family:monospace">phylip-relaxed</span> format (that&#X2019;s Phylip format, but without the 10-character limit
on taxon names) and a variety of options. A quick example:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Phylo
&gt;&gt;&gt; from Bio.Phylo.Applications import PhymlCommandline
&gt;&gt;&gt; cmd = PhymlCommandline(input='Tests/Phylip/random.phy')
&gt;&gt;&gt; out_log, err_log = cmd()
</pre><p>This generates a tree file and a stats file with the names
[<span style="font-style:italic">input&#XA0;filename</span>]<code>_phyml_tree.txt</code> and
[<span style="font-style:italic">input&#XA0;filename</span>]<code>_phyml_stats.txt</code>. The tree file is in Newick format:</p><pre class="verbatim">&gt;&gt;&gt; tree = Phylo.read('Tests/Phylip/random.phy_phyml_tree.txt', 'newick')
&gt;&gt;&gt; Phylo.draw_ascii(tree)
</pre><p>A similar wrapper for RAxML (<a href="http://sco.h-its.org/exelixis/software.html"><span style="font-family:monospace">http://sco.h-its.org/exelixis/software.html</span></a>)
was added in Biopython 1.60, and FastTree
(<a href="http://www.microbesonline.org/fasttree/"><span style="font-family:monospace">http://www.microbesonline.org/fasttree/</span></a>) in Biopython 1.62.</p><p>Note that some popular Phylip programs, including <span style="font-family:monospace">dnaml</span> and <span style="font-family:monospace">protml</span>,
are already available through the EMBOSS wrappers in <span style="font-family:monospace">Bio.Emboss.Applications</span> if
you have the Phylip extensions to EMBOSS installed on your system.
See Section&#XA0;<a href="#sec%3Aalignment-tools">6.4</a> for some examples and clues on how to use
programs like these.</p>
<!--TOC section id="sec242" PAML integration-->
<h2 id="sec242" class="section">13.6&#XA0;&#XA0;PAML integration</h2><!--SEC END --><p>
<a id="sec:PhyloPAML"></a></p><p>Biopython 1.58 brought support for PAML
(<a href="http://abacus.gene.ucl.ac.uk/software/paml.html"><span style="font-family:monospace">http://abacus.gene.ucl.ac.uk/software/paml.html</span></a>), a suite of programs for
phylogenetic analysis by maximum likelihood. Currently the programs codeml, baseml and 
yn00 are implemented. Due to PAML&#X2019;s usage of control files rather than command line 
arguments to control runtime options, usage of this wrapper strays from the format of 
other application wrappers in Biopython. </p><p>A typical workflow would be to initialize a PAML object, specifying an alignment file, a
tree file, an output file and a working directory. Next, runtime options are set via the
<span style="font-family:monospace">set_options()</span> method or by reading an existing control file. Finally, the
program is run via the <span style="font-family:monospace">run()</span> method and the output file is automatically parsed
to a results dictionary.</p><p>Here is an example of typical usage of codeml:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Phylo.PAML import codeml
&gt;&gt;&gt; cml = codeml.Codeml()
&gt;&gt;&gt; cml.alignment = "Tests/PAML/alignment.phylip"
&gt;&gt;&gt; cml.tree = "Tests/PAML/species.tree"
&gt;&gt;&gt; cml.out_file = "results.out"
&gt;&gt;&gt; cml.working_dir = "./scratch"
&gt;&gt;&gt; cml.set_options(seqtype=1,
...         verbose=0,
...         noisy=0,
...         RateAncestor=0,
...         model=0,
...         NSsites=[0, 1, 2],
...         CodonFreq=2,
...         cleandata=1,
...         fix_alpha=1,
...         kappa=4.54006)
&gt;&gt;&gt; results = cml.run()
&gt;&gt;&gt; ns_sites = results.get("NSsites")
&gt;&gt;&gt; m0 = ns_sites.get(0)
&gt;&gt;&gt; m0_params = m0.get("parameters")
&gt;&gt;&gt; print(m0_params.get("omega"))
</pre><p>Existing output files may be parsed as well using a module&#X2019;s <span style="font-family:monospace">read()</span> function:
</p><pre class="verbatim">&gt;&gt;&gt; results = codeml.read("Tests/PAML/Results/codeml/codeml_NSsites_all.out")
&gt;&gt;&gt; print(results.get("lnL max"))
</pre><p>Detailed documentation for this new module currently lives on the Biopython wiki:
<a href="http://biopython.org/wiki/PAML"><span style="font-family:monospace">http://biopython.org/wiki/PAML</span></a></p>
<!--TOC section id="sec243" Future plans-->
<h2 id="sec243" class="section">13.7&#XA0;&#XA0;Future plans</h2><!--SEC END --><p>
<a id="sec:PhyloFuture"></a></p><p>Bio.Phylo is under active development. Here are some features we might add in future
releases:</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold">New methods</span></dt><dd class="dd-description"> 
Generally useful functions for operating on Tree or Clade objects appear on the Biopython
wiki first, so that casual users can test them and decide if they&#X2019;re useful before we add
them to Bio.Phylo:<p><a href="http://biopython.org/wiki/Phylo_cookbook"><span style="font-family:monospace">http://biopython.org/wiki/Phylo_cookbook</span></a></p></dd><dt class="dt-description"><span style="font-weight:bold">Bio.Nexus port</span></dt><dd class="dd-description">
Much of this module was written during Google Summer of Code 2009, under the auspices of
NESCent, as a project to implement Python support for the phyloXML data format (see
<a href="#sec%3APhyloXML">13.4.4</a>). Support for Newick and Nexus formats was added by porting part of the
existing Bio.Nexus module to the new classes used by Bio.Phylo.<p>Currently, Bio.Nexus contains some useful features that have not yet been ported to
Bio.Phylo classes &#X2014; notably, calculating a consensus tree. If you find some functionality
lacking in Bio.Phylo, try poking throught Bio.Nexus to see if it&#X2019;s there instead.</p></dd></dl><p>We&#X2019;re open to any suggestions for improving the functionality and usability of this module;
just let us know on the mailing list or our bug database.</p><p>Finally, if you need additional functionality not yet included in the Phylo
module, check if it&#X2019;s available in another of the high-quality Python libraries
for phylogenetics such as DendroPy (<a href="http://pythonhosted.org/DendroPy/"><span style="font-family:monospace">http://pythonhosted.org/DendroPy/</span></a>) or
PyCogent (<a href="http://pycogent.org/"><span style="font-family:monospace">http://pycogent.org/</span></a>). Since these libraries also support
standard file formats for phylogenetic trees, you can easily transfer data
between libraries by writing to a temporary file or StringIO object.</p>
<!--TOC chapter id="sec244" Sequence motif analysis using Bio.motifs-->
<h1 id="sec244" class="chapter">Chapter&#XA0;14&#XA0;&#XA0;Sequence motif analysis using Bio.motifs</h1><!--SEC END --><p>This chapter gives an overview of the functionality of the
<code>Bio.motifs</code> package included in Biopython. It is intended
for people who are involved in the analysis of sequence motifs, so I&#X2019;ll
assume that you are familiar with basic notions of motif analysis. In
case something is unclear, please look at Section&#XA0;<a href="#sec%3Alinks">14.10</a>
for some relevant links.</p><p>Most of this chapter describes the new <code>Bio.motifs</code> package included
in Biopython 1.61 onwards, which is replacing the older <code>Bio.Motif</code> package
introduced with Biopython 1.50, which was in turn based on two older former
Biopython modules, <code>Bio.AlignAce</code> and <code>Bio.MEME</code>. It provides
most of their functionality with a unified motif object implementation.</p><p>Speaking of other libraries, if you are reading this you might be
interested in <a href="http://fraenkel.mit.edu/TAMO/">TAMO</a>, another python library
designed to deal with sequence motifs. It supports more <em>de-novo</em>
motif finders, but it is not a part of Biopython and has some restrictions
on commercial use.</p>
<!--TOC section id="sec245" Motif objects-->
<h2 id="sec245" class="section">14.1&#XA0;&#XA0;Motif objects</h2><!--SEC END --><p>
<a id="sec:object"></a></p><p>Since we are interested in motif analysis, we need to take a look at
<code>Motif</code> objects in the first place. For that we need to import 
the Bio.motifs library:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import motifs
</pre><p>and we can start creating our first motif objects. We can either create
a <code>Motif</code> object from a list of instances of the motif, or we can
obtain a <code>Motif</code> object by parsing a file from a motif database
or motif finding software.</p>
<!--TOC subsection id="sec246" Creating a motif from instances-->
<h3 id="sec246" class="subsection">14.1.1&#XA0;&#XA0;Creating a motif from instances</h3><!--SEC END --><p>Suppose we have these instances of a DNA motif:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; instances = [Seq("TACAA"),
...              Seq("TACGC"),
...              Seq("TACAC"),
...              Seq("TACCC"),
...              Seq("AACCC"),
...              Seq("AATGC"),
...              Seq("AATGC"),
...             ]
</pre><p>then we can create a Motif object as follows:
</p><pre class="verbatim">&gt;&gt;&gt; m = motifs.create(instances)
</pre><p>The instances are saved in an attribute <code>m.instances</code>, which is essentially a Python list with some added functionality, as described below.
Printing out the Motif object shows the instances from which it was constructed:
</p><pre class="verbatim">&gt;&gt;&gt; print(m)
TACAA
TACGC
TACAC
TACCC
AACCC
AATGC
AATGC
&lt;BLANKLINE&gt;
</pre><p>The length of the motif defined as the sequence length, which should be the same for all instances:
</p><pre class="verbatim">&gt;&gt;&gt; len(m)
5
</pre><p>The Motif object has an attribute <code>.counts</code> containing the counts of each
nucleotide at each position. Printing this counts matrix shows it in an easily readable format:
</p><pre class="verbatim">&gt;&gt;&gt; print(m.counts)
        0      1      2      3      4
A:   3.00   7.00   0.00   2.00   1.00
C:   0.00   0.00   5.00   2.00   6.00
G:   0.00   0.00   0.00   3.00   0.00
T:   4.00   0.00   2.00   0.00   0.00
&lt;BLANKLINE&gt;
</pre><p>You can access these counts as a dictionary:
</p><pre class="verbatim">&gt;&gt;&gt; m.counts['A']
[3, 7, 0, 2, 1]
</pre><p>but you can also think of it as a 2D array with the nucleotide as the first
dimension and the position as the second dimension:
</p><pre class="verbatim">&gt;&gt;&gt; m.counts['T', 0]
4
&gt;&gt;&gt; m.counts['T', 2]
2
&gt;&gt;&gt; m.counts['T', 3]
0
</pre><p>You can also directly access columns of the counts matrix
</p><pre class="verbatim">&gt;&gt;&gt; m.counts[:, 3]
{'A': 2, 'C': 2, 'T': 0, 'G': 3}
</pre><p>Instead of the nucleotide itself, you can also use the index of the nucleotide
in the sorted letters in the alphabet of the motif:
</p><pre class="verbatim">&gt;&gt;&gt; m.alphabet
IUPACUnambiguousDNA()
&gt;&gt;&gt; m.alphabet.letters
'GATC'
&gt;&gt;&gt; sorted(m.alphabet.letters)
['A', 'C', 'G', 'T']
&gt;&gt;&gt; m.counts['A',:]
(3, 7, 0, 2, 1)
&gt;&gt;&gt; m.counts[0,:]
(3, 7, 0, 2, 1)
</pre><p>The motif has an associated consensus sequence, defined as the sequence of
letters along the positions of the motif for which the largest value in the 
corresponding columns of the <code>.counts</code> matrix is obtained:
</p><pre class="verbatim">&gt;&gt;&gt; m.consensus
Seq('TACGC', IUPACUnambiguousDNA())
</pre><p>as well as an anticonsensus sequence, corresponding to the smallest values in
the columns of the <code>.counts</code> matrix:
</p><pre class="verbatim">&gt;&gt;&gt; m.anticonsensus
Seq('GGGTG', IUPACUnambiguousDNA())
</pre><p>You can also ask for a degenerate consensus sequence, in which ambiguous
nucleotides are used for positions where there are multiple nucleotides with
high counts:
</p><pre class="verbatim">&gt;&gt;&gt; m.degenerate_consensus
Seq('WACVC', IUPACAmbiguousDNA())
</pre><p>Here, W and R follow the IUPAC nucleotide ambiguity codes: W is either A or T,
and V is A, C, or G [<a href="#cornish1985">10</a>]. The degenerate consensus sequence is
constructed following the rules specified by Cavener [<a href="#cavener1987">11</a>].</p><p>We can also get the reverse complement of a motif:
</p><pre class="verbatim">&gt;&gt;&gt; r = m.reverse_complement()
&gt;&gt;&gt; r.consensus
Seq('GCGTA', IUPACUnambiguousDNA())
&gt;&gt;&gt; r.degenerate_consensus
Seq('GBGTW', IUPACAmbiguousDNA())
&gt;&gt;&gt; print(r)
TTGTA
GCGTA
GTGTA
GGGTA
GGGTT
GCATT
GCATT
&lt;BLANKLINE&gt;
</pre><p>The reverse complement and the degenerate consensus sequence are
only defined for DNA motifs.</p>
<!--TOC subsection id="sec247" Creating a sequence logo-->
<h3 id="sec247" class="subsection">14.1.2&#XA0;&#XA0;Creating a sequence logo</h3><!--SEC END --><p>
If we have internet access, we can create a <a href="http://weblogo.berkeley.edu">weblogo</a>:
</p><pre class="verbatim">&gt;&gt;&gt; m.weblogo("mymotif.png")
</pre><p>We should get our logo saved as a PNG in the specified file.</p>
<!--TOC section id="sec248" Reading motifs-->
<h2 id="sec248" class="section">14.2&#XA0;&#XA0;Reading motifs</h2><!--SEC END --><p>
<a id="sec:io"></a></p><p>Creating motifs from instances by hand is a bit boring, so it&#X2019;s
useful to have some I/O functions for reading and writing
motifs. There are not any really well established standards for storing
motifs, but there are a couple of formats that are more used than
others.</p>
<!--TOC subsection id="sec249" JASPAR-->
<h3 id="sec249" class="subsection">14.2.1&#XA0;&#XA0;JASPAR</h3><!--SEC END --><p>
One of the most popular motif databases is <a href="http://jaspar.genereg.net">JASPAR</a>. In addition to the motif sequence information, the JASPAR database stores a lot of meta-information for each motif. The module <code>Bio.motifs</code> contains a specialized class <code>jaspar.Motif</code> in which this meta-information is represented as attributes:
</p><ul class="itemize"><li class="li-itemize">
<code>matrix_id</code> - the unique JASPAR motif ID, e.g. &#X2019;MA0004.1&#X2019;
</li><li class="li-itemize"><code>name</code> - the name of the TF, e.g. &#X2019;Arnt&#X2019;
</li><li class="li-itemize"><code>collection</code> - the JASPAR collection to which the motif belongs, e.g. &#X2019;CORE&#X2019;
</li><li class="li-itemize"><code>tf_class</code> - the structual class of this TF, e.g. &#X2019;Zipper-Type&#X2019;
</li><li class="li-itemize"><code>tf_family</code> - the family to which this TF belongs, e.g. &#X2019;Helix-Loop-Helix&#X2019;
</li><li class="li-itemize"><code>species</code> - the species to which this TF belongs, may have multiple values, these are specified as taxonomy IDs, e.g. 10090
</li><li class="li-itemize"><code>tax_group</code> - the taxonomic supergroup to which this motif belongs, e.g. &#X2019;vertebrates&#X2019;
</li><li class="li-itemize"><code>acc</code> - the accesion number of the TF protein, e.g. &#X2019;P53762&#X2019;
</li><li class="li-itemize"><code>data_type</code> - the type of data used to construct this motif, e.g. &#X2019;SELEX&#X2019;
</li><li class="li-itemize"><code>medline</code> - the Pubmed ID of literature supporting this motif, may be multiple values, e.g. 7592839 
</li><li class="li-itemize"><code>pazar_id</code> - external reference to the TF in the <a href="http://pazar.info">PAZAR</a> database, e.g. &#X2019;TF0000003&#X2019;
</li><li class="li-itemize"><code>comment</code> - free form text containing notes about the construction of the motif
</li></ul><p>The <code>jaspar.Motif</code> class inherits from the generic <code>Motif</code> class and therefore provides all the facilities of any of the motif formats &#X2014; reading motifs, writing motifs, scanning sequences for motif instances etc.</p><p>JASPAR stores motifs in several different ways including three different flat file formats and as an SQL database. All of these formats facilitate the construction of a counts matrix. However, the amount of meta information described above that is available varies with the format.</p><!--TOC subsubsection id="sec250" The JASPAR <span style="font-family:monospace">sites</span> format-->
<h4 id="sec250" class="subsubsection">The JASPAR <span style="font-family:monospace">sites</span> format</h4><!--SEC END --><p>The first of the three flat file formats contains a list of instances. As an example, these are the beginning and ending lines of the JASPAR <code>Arnt.sites</code> file showing known binding sites of the mouse helix-loop-helix transcription factor Arnt.
</p><pre class="verbatim">&gt;MA0004 ARNT 1
CACGTGatgtcctc
&gt;MA0004 ARNT 2
CACGTGggaggtac
&gt;MA0004 ARNT 3
CACGTGccgcgcgc
...
&gt;MA0004 ARNT 18
AACGTGacagccctcc
&gt;MA0004 ARNT 19
AACGTGcacatcgtcc
&gt;MA0004 ARNT 20
aggaatCGCGTGc
</pre><p>The parts of the sequence in capital letters are the motif instances that were found to align to each other.</p><p>We can create a <code>Motif</code> object from these instances as follows:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import motifs
&gt;&gt;&gt; with open("Arnt.sites") as handle:
...     arnt = motifs.read(handle, "sites")
...
</pre><p>The instances from which this motif was created is stored in the <code>.instances</code> property:
</p><pre class="verbatim">&gt;&gt;&gt; print(arnt.instances[:3])
[Seq('CACGTG', IUPACUnambiguousDNA()), Seq('CACGTG', IUPACUnambiguousDNA()), Seq('CACGTG', IUPACUnambiguousDNA())]
&gt;&gt;&gt; for instance in arnt.instances:
...     print(instance)
...
CACGTG
CACGTG
CACGTG
CACGTG
CACGTG
CACGTG
CACGTG
CACGTG
CACGTG
CACGTG
CACGTG
CACGTG
CACGTG
CACGTG
CACGTG
AACGTG
AACGTG
AACGTG
AACGTG
CGCGTG
</pre><p>The counts matrix of this motif is automatically calculated from the instances:
</p><pre class="verbatim">&gt;&gt;&gt; print(arnt.counts)
        0      1      2      3      4      5
A:   4.00  19.00   0.00   0.00   0.00   0.00
C:  16.00   0.00  20.00   0.00   0.00   0.00
G:   0.00   1.00   0.00  20.00   0.00  20.00
T:   0.00   0.00   0.00   0.00  20.00   0.00
&lt;BLANKLINE&gt;
</pre><p>This format does not store any meta information.</p><!--TOC subsubsection id="sec251" The JASPAR <span style="font-family:monospace">pfm</span> format-->
<h4 id="sec251" class="subsubsection">The JASPAR <span style="font-family:monospace">pfm</span> format</h4><!--SEC END --><p>JASPAR also makes motifs available directly as a count matrix,
without the instances from which it was created. This <code>pfm</code> format only
stores the counts matrix for a single motif.
For example, this is the JASPAR file <code>SRF.pfm</code> containing the counts matrix for the human SRF transcription factor:
</p><pre class="verbatim"> 2 9 0 1 32 3 46 1 43 15 2 2
 1 33 45 45 1 1 0 0 0 1 0 1
39 2 1 0 0 0 0 0 0 0 44 43
 4 2 0 0 13 42 0 45 3 30 0 0
</pre><p>We can create a motif for this count matrix as follows:
</p><pre class="verbatim">&gt;&gt;&gt; with open("SRF.pfm") as handle:
...     srf = motifs.read(handle, "pfm")
...
&gt;&gt;&gt; print(srf.counts)
        0      1      2      3      4      5      6      7      8      9     10     11
A:   2.00   9.00   0.00   1.00  32.00   3.00  46.00   1.00  43.00  15.00   2.00   2.00
C:   1.00  33.00  45.00  45.00   1.00   1.00   0.00   0.00   0.00   1.00   0.00   1.00
G:  39.00   2.00   1.00   0.00   0.00   0.00   0.00   0.00   0.00   0.00  44.00  43.00
T:   4.00   2.00   0.00   0.00  13.00  42.00   0.00  45.00   3.00  30.00   0.00   0.00
&lt;BLANKLINE&gt;
</pre><p>As this motif was created from the counts matrix directly, it has no instances associated with it:
</p><pre class="verbatim">&gt;&gt;&gt; print(srf.instances)
None
</pre><p>We can now ask for the consensus sequence of these two motifs:
</p><pre class="verbatim">&gt;&gt;&gt; print(arnt.counts.consensus)
CACGTG
&gt;&gt;&gt; print(srf.counts.consensus)
GCCCATATATGG
</pre><p>As with the instances file, not meta information is stored in this format.</p><!--TOC subsubsection id="sec252" The JASPAR format <span style="font-family:monospace">jaspar</span>-->
<h4 id="sec252" class="subsubsection">The JASPAR format <span style="font-family:monospace">jaspar</span></h4><!--SEC END --><p>The <code>jaspar</code> file format allows multiple motifs to be specified in a single file. In this format each of the motif records consist of a header line followed by four lines defining the counts matrix. The header line begins with a <code>&gt;</code> character (similar to the Fasta file format) and is followed by the unique JASPAR matrix ID and the TF name. The following example shows a <code>jaspar</code> formatted file containing the three motifs Arnt, RUNX1 and MEF2A:
</p><pre class="verbatim">&gt;MA0004.1 Arnt
A  [ 4 19  0  0  0  0 ]
C  [16  0 20  0  0  0 ]
G  [ 0  1  0 20  0 20 ]
T  [ 0  0  0  0 20  0 ]
&gt;MA0002.1 RUNX1
A  [10 12  4  1  2  2  0  0  0  8 13 ]
C  [ 2  2  7  1  0  8  0  0  1  2  2 ]
G  [ 3  1  1  0 23  0 26 26  0  0  4 ]
T  [11 11 14 24  1 16  0  0 25 16  7 ]
&gt;MA0052.1 MEF2A
A  [ 1  0 57  2  9  6 37  2 56  6 ]
C  [50  0  1  1  0  0  0  0  0  0 ]
G  [ 0  0  0  0  0  0  0  0  2 50 ]
T  [ 7 58  0 55 49 52 21 56  0  2 ]
</pre><p>The motifs are read as follows:
</p><pre class="verbatim">&gt;&gt;&gt; fh = open("jaspar_motifs.txt")
&gt;&gt;&gt; for m in motifs.parse(fh, "jaspar"))
...     print(m)
TF name  Arnt
Matrix ID MA0004.1
Matrix:
        0      1      2      3      4      5
A:   4.00  19.00   0.00   0.00   0.00   0.00
C:  16.00   0.00  20.00   0.00   0.00   0.00
G:   0.00   1.00   0.00  20.00   0.00  20.00
T:   0.00   0.00   0.00   0.00  20.00   0.00



TF name  RUNX1
Matrix ID MA0002.1
Matrix:
        0      1      2      3      4      5      6      7      8      9     10
A:  10.00  12.00   4.00   1.00   2.00   2.00   0.00   0.00   0.00   8.00  13.00
C:   2.00   2.00   7.00   1.00   0.00   8.00   0.00   0.00   1.00   2.00   2.00
G:   3.00   1.00   1.00   0.00  23.00   0.00  26.00  26.00   0.00   0.00   4.00
T:  11.00  11.00  14.00  24.00   1.00  16.00   0.00   0.00  25.00  16.00   7.00



TF name  MEF2A
Matrix ID MA0052.1
Matrix:
        0      1      2      3      4      5      6      7      8      9
A:   1.00   0.00  57.00   2.00   9.00   6.00  37.00   2.00  56.00   6.00
C:  50.00   0.00   1.00   1.00   0.00   0.00   0.00   0.00   0.00   0.00
G:   0.00   0.00   0.00   0.00   0.00   0.00   0.00   0.00   2.00  50.00
T:   7.00  58.00   0.00  55.00  49.00  52.00  21.00  56.00   0.00   2.00
</pre><p>Note that printing a JASPAR motif yields both the counts data and the available meta-information.</p><!--TOC subsubsection id="sec253" Accessing the JASPAR database-->
<h4 id="sec253" class="subsubsection">Accessing the JASPAR database</h4><!--SEC END --><p>In addition to parsing these flat file formats, we can also retrieve motifs from a JASPAR SQL database. Unlike the flat file formats, a JASPAR database allows storing of all possible meta information defined in the JASPAR <code>Motif</code> class. It is beyond the scope of this document to describe how to set up a JASPAR database (please see the main <a href="http://jaspar.genereg.net">JASPAR</a> website). Motifs are read from a JASPAR database using the <code>Bio.motifs.jaspar.db</code> module. First connect to the JASPAR database using the JASPAR5 class which models the the latest JASPAR schema:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.motifs.jaspar.db import JASPAR5
&gt;&gt;&gt; 
&gt;&gt;&gt; JASPAR_DB_HOST = &lt;hostname&gt;
&gt;&gt;&gt; JASPAR_DB_NAME = &lt;db_name&gt;
&gt;&gt;&gt; JASPAR_DB_USER = &lt;user&gt;
&gt;&gt;&gt; JASPAR_DB_PASS = &lt;passord&gt;
&gt;&gt;&gt; 
&gt;&gt;&gt; jdb = JASPAR5(
...     host=JASPAR_DB_HOST,
...     name=JASPAR_DB_NAME,
...     user=JASPAR_DB_USER,
...     password=JASPAR_DB_PASS
... )
</pre><p>Now we can fetch a single motif by its unique JASPAR ID with the <code>fetch_motif_by_id</code> method. Note that a JASPAR ID conists of a base ID and a version number seperated by a decimal point, e.g. &#X2019;MA0004.1&#X2019;. The <code>fetch_motif_by_id</code> method allows you to use either the fully specified ID or just the base ID. If only the base ID is provided, the latest version of the motif is returned.
</p><pre class="verbatim">&gt;&gt;&gt; arnt = jdb.fetch_motif_by_id("MA0004")
</pre><p>Printing the motif reveals that the JASPAR SQL database stores much more meeta-information than the flat files:
</p><pre class="verbatim">&gt;&gt;&gt; print(arnt)
TF name Arnt
Matrix ID MA0004.1
Collection CORE
TF class Zipper-Type
TF family Helix-Loop-Helix
Species 10090
Taxonomic group vertebrates
Accession ['P53762']
Data type used SELEX
Medline 7592839
PAZAR ID TF0000003
Comments -
Matrix:
 0      1      2      3      4      5
A:   4.00  19.00   0.00   0.00   0.00   0.00
C:  16.00   0.00  20.00   0.00   0.00   0.00
G:   0.00   1.00   0.00  20.00   0.00  20.00
T:   0.00   0.00   0.00   0.00  20.00   0.00


</pre><p>We can also fetch motifs by name. The name must be an exact match (partial matches or database wildcards are not currently supported). Note that as the name is not guaranteed to be unique, the <code>fetch_motifs_by_name</code> method actually returns a list.
</p><pre class="verbatim">&gt;&gt;&gt; motifs = jdb.fetch_motifs_by_name("Arnt")
&gt;&gt;&gt; print(motifs[0])
TF name Arnt
Matrix ID MA0004.1
Collection CORE
TF class Zipper-Type
TF family Helix-Loop-Helix
Species 10090
Taxonomic group vertebrates
Accession ['P53762']
Data type used SELEX
Medline 7592839
PAZAR ID TF0000003
Comments -
Matrix:
 0      1      2      3      4      5
A:   4.00  19.00   0.00   0.00   0.00   0.00
C:  16.00   0.00  20.00   0.00   0.00   0.00
G:   0.00   1.00   0.00  20.00   0.00  20.00
T:   0.00   0.00   0.00   0.00  20.00   0.00


</pre><p>The <code>fetch_motifs</code> method allows you to fetch motifs which match a specified set of criteria. These criteria include any of the above described meta information as well as certain matrix properties such as the minimum information content (<code>min_ic</code> in the example below), the minimum length of the matrix or the minimum number of sites used to construct the matrix. Only motifs which pass ALL the specified criteria are returned. Note that selection criteria which correspond to meta information which allow for multiple values may be specified as either a single value or a list of values, e.g. <code>tax_group</code> and <code>tf_family</code> in the example below.
</p><pre class="verbatim">&gt;&gt;&gt; motifs = jdb.fetch_motifs(
...     collection = 'CORE',
...     tax_group = ['vertebrates', 'insects'],
...     tf_class = 'Winged Helix-Turn-Helix',
...     tf_family = ['Forkhead', 'Ets'],
...     min_ic = 12
... )
&gt;&gt;&gt; for motif in motifs:
...     pass # do something with the motif
</pre><!--TOC subsubsection id="sec254" Compatibility with Perl TFBS modules-->
<h4 id="sec254" class="subsubsection">Compatibility with Perl TFBS modules</h4><!--SEC END --><p>An important thing to note is that the JASPAR <code>Motif</code> class was designed to be compatible with the popular <a href="http://tfbs.genereg.net/">Perl TFBS modules</a>. Therefore some specifics about the choice of defaults for background and pseudocounts as well as how information content is computed and sequences searched for instances is based on this compatibility criteria. These choices are noted in the specific subsections below.</p><ul class="itemize"><li class="li-itemize">
<span style="font-weight:bold">Choice of background:</span> <br>
The Perl <code>TFBS</code> modules appear to allow a choice of custom background probabilities (although the documentation states that uniform background is assumed). However the default is to use a uniform background. Therefore it is recommended that you use a uniform background for computing the position-specific scoring matrix (PSSM). This is the default when using the Biopython <code>motifs</code> module.
</li><li class="li-itemize"><span style="font-weight:bold">Choice of pseudocounts:</span> <br>
By default, the Perl <code>TFBS</code> modules use a pseudocount equal to &#X221A;<span style="text-decoration:overline"><span style="font-style:italic">N</span></span> * bg[nucleotide], where <span style="font-style:italic">N</span> represents the total number of sequences used to construct the matrix. To apply this same pseudocount formula, set the motif <code>pseudocounts</code> attribute using the <code>jaspar.calculate\_pseudcounts()</code> funtion:
<pre class="verbatim">&gt;&gt;&gt; motif.pseudocounts = motifs.jaspar.calculate_pseudocounts(motif)
</pre>Note that it is possible for the counts matrix to have an unequal number of sequences making up the columns. The pseudocount computation uses the average number of sequences making up the matrix. However, when <code>normalize</code> is called on the counts matrix, each count value in a column is divided by the total number of sequences making up that specific column, not by the average number of sequences. This differs from the Perl <code>TFBS</code> modules because the normalization is not done as a separate step and so the average number of sequences is used throughout the computation of the pssm. Therefore, for matrices with unequal column counts, the pssm computed by the <code>motifs</code> module will differ somewhat from the pssm computed by the Perl <code>TFBS</code> modules.
</li><li class="li-itemize"><span style="font-weight:bold">Computation of matrix information content:</span> <br>
The information content (IC) or specificity of a matrix is computed using the <code>mean</code> method of the <code>PositionSpecificScoringMatrix</code> class. However of note, in the Perl <code>TFBS</code> modules the default behaviour is to compute the IC without first applying pseudocounts, even though by default the PSSMs are computed using pseudocounts as described above.
</li><li class="li-itemize"><span style="font-weight:bold">Searching for instances:</span> <br>
Searching for instances with the Perl <code>TFBS</code> motifs was usually performed using a relative score threshold, i.e. a score in the range 0 to 1. In order to compute the absolute PSSM score corresponding to a relative score one can use the equation:
<pre class="verbatim">&gt;&gt;&gt; abs_score =  (pssm.max - pssm.min) * rel_score + pssm.min
</pre>To convert the absolute score of an instance back to a relative score, one can use the equation:
<pre class="verbatim">&gt;&gt;&gt; rel_score = (abs_score - pssm.min) / (pssm.max - pssm.min)
</pre>For example, using the Arnt motif before, let&#X2019;s search a sequence with a relative score threshold of 0.8.
<pre class="verbatim">&gt;&gt;&gt; test_seq=Seq("TAAGCGTGCACGCGCAACACGTGCATTA", unambiguous_dna)
&gt;&gt;&gt; arnt.pseudocounts = motifs.jaspar.calculate_pseudocounts(arnt) 
&gt;&gt;&gt; pssm = arnt.pssm()
&gt;&gt;&gt; max_score = pssm.max()
&gt;&gt;&gt; min_score = pssm.min()
&gt;&gt;&gt; abs_score_threshold = (max_score - min_score) * 0.8 + min_score
&gt;&gt;&gt; for position, score in pssm.search(test_seq,
                                       threshold=abs_score_threshold):
...     rel_score = (score - min_score) / (max_score - min_score)
...     print("Position %d: score = %5.3f, rel. score = %5.3f" % (
            position, score, rel_score))
... 
Position 2: score = 5.362, rel. score = 0.801
Position 8: score = 6.112, rel. score = 0.831
Position -20: score = 7.103, rel. score = 0.870
Position 17: score = 10.351, rel. score = 1.000
Position -11: score = 10.351, rel. score = 1.000
</pre></li></ul>
<!--TOC subsection id="sec255" MEME-->
<h3 id="sec255" class="subsection">14.2.2&#XA0;&#XA0;MEME</h3><!--SEC END --><p>MEME [<a href="#bailey1994">12</a>] is a tool for discovering motifs in a group of related
DNA or protein sequences. It takes as input a group of DNA or protein sequences 
and outputs as many motifs as requested. Therefore, in contrast to JASPAR
files, MEME output files typically contain multiple motifs. This is an example.</p><p>At the top of an output file generated by MEME shows some background information
about the MEME and the version of MEME used:
</p><pre class="verbatim">********************************************************************************
MEME - Motif discovery tool
********************************************************************************
MEME version 3.0 (Release date: 2004/08/18 09:07:01)
...
</pre><p>Further down, the input set of training sequences is recapitulated:
</p><pre class="verbatim">********************************************************************************
TRAINING SET
********************************************************************************
DATAFILE= INO_up800.s
ALPHABET= ACGT
Sequence name            Weight Length  Sequence name            Weight Length
-------------            ------ ------  -------------            ------ ------
CHO1                     1.0000    800  CHO2                     1.0000    800
FAS1                     1.0000    800  FAS2                     1.0000    800
ACC1                     1.0000    800  INO1                     1.0000    800
OPI3                     1.0000    800
********************************************************************************
</pre><p>and the exact command line that was used:
</p><pre class="verbatim">********************************************************************************
COMMAND LINE SUMMARY
********************************************************************************
This information can also be useful in the event you wish to report a
problem with the MEME software.

command: meme -mod oops -dna -revcomp -nmotifs 2 -bfile yeast.nc.6.freq INO_up800.s
...
</pre><p>Next is detailed information on each motif that was found:
</p><pre class="verbatim">********************************************************************************
MOTIF  1        width =   12   sites =   7   llr = 95   E-value = 2.0e-001
********************************************************************************
--------------------------------------------------------------------------------
        Motif 1 Description
--------------------------------------------------------------------------------
Simplified        A  :::9:a::::3:
pos.-specific     C  ::a:9:11691a
probability       G  ::::1::94:4:
matrix            T  aa:1::9::11:
</pre><p>To parse this file (stored as <code>meme.dna.oops.txt</code>), use
</p><pre class="verbatim">&gt;&gt;&gt; handle = open("meme.dna.oops.txt")
&gt;&gt;&gt; record = motifs.parse(handle, "meme")
&gt;&gt;&gt; handle.close()
</pre><p>The <code>motifs.parse</code> command reads the complete file directly, so you can
close the file after calling <code>motifs.parse</code>.
The header information is stored in attributes:
</p><pre class="verbatim">&gt;&gt;&gt; record.version
'3.0'
&gt;&gt;&gt; record.datafile
'INO_up800.s'
&gt;&gt;&gt; record.command
'meme -mod oops -dna -revcomp -nmotifs 2 -bfile yeast.nc.6.freq INO_up800.s'
&gt;&gt;&gt; record.alphabet
IUPACUnambiguousDNA()
&gt;&gt;&gt; record.sequences
['CHO1', 'CHO2', 'FAS1', 'FAS2', 'ACC1', 'INO1', 'OPI3']
</pre><p>The record is an object of the <code>Bio.motifs.meme.Record</code> class.
The class inherits from list, and you can think of <code>record</code> as a list of Motif objects:
</p><pre class="verbatim">&gt;&gt;&gt; len(record)
2
&gt;&gt;&gt; motif = record[0]
&gt;&gt;&gt; print(motif.consensus)
TTCACATGCCGC
&gt;&gt;&gt; print(motif.degenerate_consensus)
TTCACATGSCNC
</pre><p>In addition to these generic motif attributes, each motif also stores its
specific information as calculated by MEME. For example,
</p><pre class="verbatim">&gt;&gt;&gt; motif.num_occurrences
7
&gt;&gt;&gt; motif.length
12
&gt;&gt;&gt; evalue = motif.evalue
&gt;&gt;&gt; print("%3.1g" % evalue)
0.2
&gt;&gt;&gt; motif.name
'Motif 1'
</pre><p>In addition to using an index into the record, as we did above,
you can also find it by its name:
</p><pre class="verbatim">&gt;&gt;&gt; motif = record['Motif 1']
</pre><p>Each motif has an attribute <code>.instances</code> with the sequence instances
in which the motif was found, providing some information on each instance:
</p><pre class="verbatim">&gt;&gt;&gt; len(motif.instances)
7
&gt;&gt;&gt; motif.instances[0]
Instance('TTCACATGCCGC', IUPACUnambiguousDNA())
&gt;&gt;&gt; motif.instances[0].motif_name
'Motif 1'
&gt;&gt;&gt; motif.instances[0].sequence_name
'INO1'
&gt;&gt;&gt; motif.instances[0].start
620
&gt;&gt;&gt; motif.instances[0].strand
'-'
&gt;&gt;&gt; motif.instances[0].length
12
&gt;&gt;&gt; pvalue = motif.instances[0].pvalue
&gt;&gt;&gt; print("%5.3g" % pvalue)
1.85e-08
</pre><!--TOC subsubsection id="sec256" MAST-->
<h4 id="sec256" class="subsubsection">MAST</h4><!--SEC END -->
<!--TOC subsection id="sec257" TRANSFAC-->
<h3 id="sec257" class="subsection">14.2.3&#XA0;&#XA0;TRANSFAC</h3><!--SEC END --><p>TRANSFAC is a manually curated database of transcription factors, together
with their genomic binding sites and DNA binding profiles [<a href="#matys2003">27</a>].
While the file format used in the TRANSFAC database is nowadays also used
by others, we will refer to it as the TRANSFAC file format.</p><p>A minimal file in the TRANSFAC format looks as follows:
</p><pre class="verbatim">ID  motif1
P0      A      C      G      T
01      1      2      2      0      S
02      2      1      2      0      R
03      3      0      1      1      A
04      0      5      0      0      C
05      5      0      0      0      A
06      0      0      4      1      G
07      0      1      4      0      G
08      0      0      0      5      T
09      0      0      5      0      G
10      0      1      2      2      K
11      0      2      0      3      Y
12      1      0      3      1      G
//
</pre><p>This file shows the frequency matrix of motif <code>motif1</code> of 12 nucleotides.
In general, one file in the TRANSFAC format can contain multiple motifs. For
example, this is the contents of the example TRANSFAC file <code>transfac.dat</code>:
</p><pre class="verbatim">VV  EXAMPLE January 15, 2013
XX
//
ID  motif1
P0      A      C      G      T
01      1      2      2      0      S
02      2      1      2      0      R
03      3      0      1      1      A
...
11      0      2      0      3      Y
12      1      0      3      1      G
//
ID  motif2
P0      A      C      G      T
01      2      1      2      0      R
02      1      2      2      0      S
...
09      0      0      0      5      T
10      0      2      0      3      Y
//
</pre><p>To parse a TRANSFAC file, use
</p><pre class="verbatim">&gt;&gt;&gt; handle = open("transfac.dat")
&gt;&gt;&gt; record = motifs.parse(handle, "TRANSFAC")
&gt;&gt;&gt; handle.close()
</pre><p>The overall version number, if available, is stored as <code>record.version</code>:
</p><pre class="verbatim">&gt;&gt;&gt; record.version
'EXAMPLE January 15, 2013'
</pre><p>Each motif in <code>record</code> is in instance of the <code>Bio.motifs.transfac.Motif</code>
class, which inherits both from the <code>Bio.motifs.Motif</code> class and
from a Python dictionary. The dictionary uses the two-letter keys to
store any additional information about the motif:
</p><pre class="verbatim">&gt;&gt;&gt; motif = record[0]
&gt;&gt;&gt; motif.degenerate_consensus # Using the Bio.motifs.Motif method
Seq('SRACAGGTGKYG', IUPACAmbiguousDNA())
&gt;&gt;&gt; motif['ID'] # Using motif as a dictionary
'motif1'
</pre><p>TRANSFAC files are typically much more elaborate than this example, containing
lots of additional information about the motif. Table <a href="#table%3Atransfaccodes">14.2.3</a>
lists the two-letter field codes that are commonly found in TRANSFAC files:
</p><blockquote class="table"><div class="center"><hr style="width:80%;height:2"></div>
<a id="table:transfaccodes"></a>
<div class="center">
<div class="caption"><table style="border-spacing:6px;border-collapse:separate;" class="cellpading0"><tr><td style="vertical-align:top;text-align:left;" >Table 14.1: Fields commonly found in TRANSFAC files</td></tr>
</table></div>
<table border=1  style="border-spacing:0;" class="cellpadding1"><tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>AC</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Accession number </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>AS</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Accession numbers, secondary </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>BA</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Statistical basis </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>BF</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Binding factors </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>BS</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Factor binding sites underlying the matrix </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>CC</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Comments </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>CO</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Copyright notice </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>DE</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Short factor description </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>DR</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >External databases </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>DT</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Date created/updated </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>HC</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Subfamilies </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>HP</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Superfamilies </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>ID</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Identifier </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>NA</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Name of the binding factor </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>OC</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Taxonomic classification </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>OS</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Species/Taxon </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>OV</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Older version </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>PV</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Preferred version </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>TY</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Type </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>XX</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Empty line; these are not stored in the Record. </td></tr>
</table>
</div>
<div class="center"><hr style="width:80%;height:2"></div></blockquote><p>Each motif also has an attribute <code>.references</code> containing the
references associated with the motif, using these two-letter keys:</p><blockquote class="table"><div class="center"><hr style="width:80%;height:2"></div>
<div class="center">
<div class="caption"><table style="border-spacing:6px;border-collapse:separate;" class="cellpading0"><tr><td style="vertical-align:top;text-align:left;" >Table 14.2: Fields used to store references in TRANSFAC files</td></tr>
</table></div>
<table border=1  style="border-spacing:0;" class="cellpadding1"><tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>RN</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Reference number </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>RA</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Reference authors </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>RL</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Reference data </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>RT</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >Reference title </td></tr>
<tr><td style="text-align:left;border:solid 1px;white-space:nowrap" ><code>RX</code></td><td style="text-align:left;border:solid 1px;white-space:nowrap" >PubMed ID </td></tr>
</table>
</div>
<div class="center"><hr style="width:80%;height:2"></div></blockquote><p>Printing the motifs writes them out in their native TRANSFAC format:
</p><pre class="verbatim">&gt;&gt;&gt; print(record)
VV  EXAMPLE January 15, 2013
XX
//
ID  motif1
XX
P0      A      C      G      T
01      1      2      2      0      S
02      2      1      2      0      R
03      3      0      1      1      A
04      0      5      0      0      C
05      5      0      0      0      A
06      0      0      4      1      G
07      0      1      4      0      G
08      0      0      0      5      T
09      0      0      5      0      G
10      0      1      2      2      K
11      0      2      0      3      Y
12      1      0      3      1      G
XX
//
ID  motif2
XX
P0      A      C      G      T
01      2      1      2      0      R
02      1      2      2      0      S
03      0      5      0      0      C
04      3      0      1      1      A
05      0      0      4      1      G
06      5      0      0      0      A
07      0      1      4      0      G
08      0      0      5      0      G
09      0      0      0      5      T
10      0      2      0      3      Y
XX
//
&lt;BLANKLINE&gt;
</pre><p>You can export the motifs in the TRANSFAC format by capturing this output
in a string and saving it in a file:
</p><pre class="verbatim">&gt;&gt;&gt; text = str(record)
&gt;&gt;&gt; handle = open("mytransfacfile.dat", 'w')
&gt;&gt;&gt; handle.write(text)
&gt;&gt;&gt; handle.close()
</pre>
<!--TOC section id="sec258" Writing motifs-->
<h2 id="sec258" class="section">14.3&#XA0;&#XA0;Writing motifs</h2><!--SEC END --><p>Speaking of exporting, let&#X2019;s look at export functions in general.
We can use the <code>format</code> method to write the motif in the simple JASPAR <code>pfm</code> format:
</p><pre class="verbatim">&gt;&gt;&gt; print(arnt.format("pfm"))
  4.00  19.00   0.00   0.00   0.00   0.00
 16.00   0.00  20.00   0.00   0.00   0.00
  0.00   1.00   0.00  20.00   0.00  20.00
  0.00   0.00   0.00   0.00  20.00   0.00
</pre><p>Similarly, we can use <code>format</code> to write the motif in the JASPAR <code>jaspar</code> format:
</p><pre class="verbatim">&gt;&gt;&gt; print(arnt.format("jaspar"))
&gt;MA0004.1  Arnt
A [  4.00  19.00   0.00   0.00   0.00   0.00]
C [ 16.00   0.00  20.00   0.00   0.00   0.00]
G [  0.00   1.00   0.00  20.00   0.00  20.00]
T [  0.00   0.00   0.00   0.00  20.00   0.00]
</pre><p>To write the motif in a TRANSFAC-like matrix format, use
</p><pre class="verbatim">&gt;&gt;&gt; print(m.format("transfac"))
P0      A      C      G      T
01      3      0      0      4      W
02      7      0      0      0      A
03      0      5      0      2      C
04      2      2      3      0      V
05      1      6      0      0      C
XX
//
&lt;BLANKLINE&gt;
</pre><p>To write out multiple motifs, you can use <code>motifs.write</code>.
This function can be used regardless of whether the motifs originated from a TRANSFAC file. For example,
</p><pre class="verbatim">&gt;&gt;&gt; two_motifs = [arnt, srf]
&gt;&gt;&gt; print(motifs.write(two_motifs, 'transfac'))
P0      A      C      G      T
01      4     16      0      0      C
02     19      0      1      0      A
03      0     20      0      0      C
04      0      0     20      0      G
05      0      0      0     20      T
06      0      0     20      0      G
XX
//
P0      A      C      G      T
01      2      1     39      4      G
02      9     33      2      2      C
03      0     45      1      0      C
04      1     45      0      0      C
05     32      1      0     13      A
06      3      1      0     42      T
07     46      0      0      0      A
08      1      0      0     45      T
09     43      0      0      3      A
10     15      1      0     30      T
11      2      0     44      0      G
12      2      1     43      0      G
XX
//
&lt;BLANKLINE&gt;
</pre><p>Or, to write multiple motifs in the <code>jaspar</code> format:
</p><pre class="verbatim">&gt;&gt;&gt; two_motifs = [arnt, mef2a]
&gt;&gt;&gt; print(motifs.write(two_motifs, "jaspar"))
&gt;MA0004.1  Arnt
A [  4.00  19.00   0.00   0.00   0.00   0.00]
C [ 16.00   0.00  20.00   0.00   0.00   0.00]
G [  0.00   1.00   0.00  20.00   0.00  20.00]
T [  0.00   0.00   0.00   0.00  20.00   0.00]
&gt;MA0052.1  MEF2A
A [  1.00   0.00  57.00   2.00   9.00   6.00  37.00   2.00  56.00   6.00]
C [ 50.00   0.00   1.00   1.00   0.00   0.00   0.00   0.00   0.00   0.00]
G [  0.00   0.00   0.00   0.00   0.00   0.00   0.00   0.00   2.00  50.00]
T [  7.00  58.00   0.00  55.00  49.00  52.00  21.00  56.00   0.00   2.00]
</pre>
<!--TOC section id="sec259" Position-Weight Matrices-->
<h2 id="sec259" class="section">14.4&#XA0;&#XA0;Position-Weight Matrices</h2><!--SEC END --><p>The <code>.counts</code> attribute of a Motif object shows how often each
nucleotide appeared at each position along the alignment. We can
normalize this matrix by dividing by the number of instances in the
alignment, resulting in the probability of each nucleotide at each
position along the alignment. We refer to these probabilities as
the position-weight matrix. However, beware that in the literature
this term may also be used to refer to the position-specific scoring
matrix, which we discuss below.</p><p>Usually, pseudocounts are added to each position before normalizing.
This avoids overfitting of the position-weight matrix to the limited
number of motif instances in the alignment, and can also prevent
probabilities from becoming zero. To add a fixed pseudocount to all
nucleotides at all positions, specify a number for the
<code>pseudocounts</code> argument:
</p><pre class="verbatim">&gt;&gt;&gt; pwm = m.counts.normalize(pseudocounts=0.5)
&gt;&gt;&gt; print(pwm)
        0      1      2      3      4
A:   0.39   0.83   0.06   0.28   0.17
C:   0.06   0.06   0.61   0.28   0.72
G:   0.06   0.06   0.06   0.39   0.06
T:   0.50   0.06   0.28   0.06   0.06
&lt;BLANKLINE&gt;
</pre><p>Alternatively, <code>pseudocounts</code> can be a dictionary specifying the
pseudocounts for each nucleotide. For example, as the GC content of
the human genome is about 40%, you may want to choose the
pseudocounts accordingly:
</p><pre class="verbatim">&gt;&gt;&gt; pwm = m.counts.normalize(pseudocounts={'A':0.6, 'C': 0.4, 'G': 0.4, 'T': 0.6})
&gt;&gt;&gt; print(pwm)
        0      1      2      3      4
A:   0.40   0.84   0.07   0.29   0.18
C:   0.04   0.04   0.60   0.27   0.71
G:   0.04   0.04   0.04   0.38   0.04
T:   0.51   0.07   0.29   0.07   0.07
&lt;BLANKLINE&gt;
</pre><p>The position-weight matrix has its own methods to calculate the
consensus, anticonsensus, and degenerate consensus sequences:
</p><pre class="verbatim">&gt;&gt;&gt; pwm.consensus
Seq('TACGC', IUPACUnambiguousDNA())
&gt;&gt;&gt; pwm.anticonsensus
Seq('GGGTG', IUPACUnambiguousDNA())
&gt;&gt;&gt; pwm.degenerate_consensus
Seq('WACNC', IUPACAmbiguousDNA())
</pre><p>Note that due to the pseudocounts, the degenerate consensus sequence
calculated from the position-weight matrix is slightly different
from the degenerate consensus sequence calculated from the instances
in the motif:
</p><pre class="verbatim">&gt;&gt;&gt; m.degenerate_consensus
Seq('WACVC', IUPACAmbiguousDNA())
</pre><p>The reverse complement of the position-weight matrix can be calculated directly from the <code>pwm</code>:
</p><pre class="verbatim">&gt;&gt;&gt; rpwm = pwm.reverse_complement()
&gt;&gt;&gt; print(rpwm)
        0      1      2      3      4
A:   0.07   0.07   0.29   0.07   0.51
C:   0.04   0.38   0.04   0.04   0.04
G:   0.71   0.27   0.60   0.04   0.04
T:   0.18   0.29   0.07   0.84   0.40
&lt;BLANKLINE&gt;
</pre>
<!--TOC section id="sec260" Position-Specific Scoring Matrices-->
<h2 id="sec260" class="section">14.5&#XA0;&#XA0;Position-Specific Scoring Matrices</h2><!--SEC END --><p>Using the background distribution and PWM with pseudo-counts added,
it&#X2019;s easy to compute the log-odds ratios, telling us what are the log
odds of a particular symbol to be coming from a motif against the
background. We can use the <code>.log_odds()</code> method on the position-weight
matrix:
</p><pre class="verbatim">&gt;&gt;&gt; pssm = pwm.log_odds()
&gt;&gt;&gt; print(pssm)
        0      1      2      3      4
A:   0.68   1.76  -1.91   0.21  -0.49
C:  -2.49  -2.49   1.26   0.09   1.51
G:  -2.49  -2.49  -2.49   0.60  -2.49
T:   1.03  -1.91   0.21  -1.91  -1.91
&lt;BLANKLINE&gt;
</pre><p>Here we can see positive values for symbols more frequent in the motif
than in the background and negative for symbols more frequent in the
background. 0.0 means that it&#X2019;s equally likely to see a symbol in the
background and in the motif.</p><p>This assumes that A, C, G, and T are equally likely in the background. To
calculate the position-specific scoring matrix against a background with
unequal probabilities for A, C, G, T, use the <code>background</code> argument.
For example, against a background with a 40% GC content, use
</p><pre class="verbatim">&gt;&gt;&gt; background = {'A':0.3,'C':0.2,'G':0.2,'T':0.3}
&gt;&gt;&gt; pssm = pwm.log_odds(background)
&gt;&gt;&gt; print(pssm)
        0      1      2      3      4
A:   0.42   1.49  -2.17  -0.05  -0.75
C:  -2.17  -2.17   1.58   0.42   1.83
G:  -2.17  -2.17  -2.17   0.92  -2.17
T:   0.77  -2.17  -0.05  -2.17  -2.17
&lt;BLANKLINE&gt;
</pre><p>The maximum and minimum score obtainable from the PSSM are stored in the
<code>.max</code> and <code>.min</code> properties:
</p><pre class="verbatim">&gt;&gt;&gt; print("%4.2f" % pssm.max)
6.59
&gt;&gt;&gt; print("%4.2f" % pssm.min)
-10.85
</pre><p>The mean and standard deviation of the PSSM scores with respect to a specific
background are calculated by the <code>.mean</code> and <code>.std</code> methods.
</p><pre class="verbatim">&gt;&gt;&gt; mean = pssm.mean(background)
&gt;&gt;&gt; std = pssm.std(background)
&gt;&gt;&gt; print("mean = %0.2f, standard deviation = %0.2f" % (mean, std))
mean = 3.21, standard deviation = 2.59
</pre><p>A uniform background is used if <code>background</code> is not specified.
The mean is particularly important, as its value is equal to the 
Kullback-Leibler divergence or relative entropy, and is a measure for the
information content of the motif compared to the background. As in Biopython
the base-2 logarithm is used in the calculation of the log-odds scores, the
information content has units of bits.</p><p>The <code>.reverse_complement</code>, <code>.consensus</code>, <code>.anticonsensus</code>, and
<code>.degenerate_consensus</code> methods can be applied directly to PSSM objects.</p>
<!--TOC section id="sec261" Searching for instances-->
<h2 id="sec261" class="section">14.6&#XA0;&#XA0;Searching for instances</h2><!--SEC END --><p>
<a id="sec:search"></a></p><p>The most frequent use for a motif is to find its instances in some
sequence. For the sake of this section, we will use an artificial sequence like this:</p><pre class="verbatim">&gt;&gt;&gt; test_seq=Seq("TACACTGCATTACAACCCAAGCATTA", m.alphabet)
&gt;&gt;&gt; len(test_seq)
26
</pre>
<!--TOC subsection id="sec262" Searching for exact matches-->
<h3 id="sec262" class="subsection">14.6.1&#XA0;&#XA0;Searching for exact matches</h3><!--SEC END --><p>The simplest way to find instances, is to look for exact matches of
the true instances of the motif:
</p><pre class="verbatim">&gt;&gt;&gt; for pos, seq in m.instances.search(test_seq):
...     print("%i %s" % (pos, seq))
... 
0 TACAC
10 TACAA
13 AACCC
</pre><p>We can do the same with the reverse complement (to find instances on the complementary strand):
</p><pre class="verbatim">&gt;&gt;&gt; for pos, seq in r.instances.search(test_seq):
...     print("%i %s" % (pos, seq))
... 
6 GCATT
20 GCATT
</pre>
<!--TOC subsection id="sec263" Searching for matches using the PSSM score-->
<h3 id="sec263" class="subsection">14.6.2&#XA0;&#XA0;Searching for matches using the PSSM score</h3><!--SEC END --><p>It&#X2019;s just as easy to look for positions, giving rise to high log-odds scores against our motif:
</p><pre class="verbatim">&gt;&gt;&gt; for position, score in pssm.search(test_seq, threshold=3.0):
...     print("Position %d: score = %5.3f" % (position, score))
... 
Position 0: score = 5.622
Position -20: score = 4.601
Position 10: score = 3.037
Position 13: score = 5.738
Position -6: score = 4.601
</pre><p>The negative positions refer to instances of the motif found on the
reverse strand of the test sequence, and follow the Python convention
on negative indices. Therefore, the instance of the motif at <code>pos</code>
is located at <code>test_seq[pos:pos+len(m)]</code> both for positive and for
negative values of <code>pos</code>.</p><p>You may notice the threshold parameter, here set arbitrarily to
3.0. This is in <span style="font-style:italic">log</span><sub>2</sub>, so we are now looking only for words, which
are eight times more likely to occur under the motif model than in the
background. The default threshold is 0.0, which selects everything
that looks more like the motif than the background.</p><p>You can also calculate the scores at all positions along the sequence:
</p><pre class="verbatim">&gt;&gt;&gt; pssm.calculate(test_seq)
array([  5.62230396,  -5.6796999 ,  -3.43177247,   0.93827754,
        -6.84962511,  -2.04066086, -10.84962463,  -3.65614533,
        -0.03370807,  -3.91102552,   3.03734159,  -2.14918518,
        -0.6016975 ,   5.7381525 ,  -0.50977498,  -3.56422281,
        -8.73414803,  -0.09919716,  -0.6016975 ,  -2.39429784,
       -10.84962463,  -3.65614533], dtype=float32)
</pre><p>In general, this is the fastest way to calculate PSSM scores.
The scores returned by <code>pssm.calculate</code> are for the forward strand
only. To obtain the scores on the reverse strand, you can take the reverse
complement of the PSSM:
</p><pre class="verbatim">&gt;&gt;&gt; rpssm = pssm.reverse_complement()
&gt;&gt;&gt; rpssm.calculate(test_seq)
array([ -9.43458748,  -3.06172252,  -7.18665981,  -7.76216221,
        -2.04066086,  -4.26466274,   4.60124254,  -4.2480607 ,
        -8.73414803,  -2.26503372,  -6.49598789,  -5.64668512,
        -8.73414803, -10.84962463,  -4.82356262,  -4.82356262,
        -5.64668512,  -8.73414803,  -4.15613794,  -5.6796999 ,
         4.60124254,  -4.2480607 ], dtype=float32)
</pre>
<!--TOC subsection id="sec264" Selecting a score threshold-->
<h3 id="sec264" class="subsection">14.6.3&#XA0;&#XA0;Selecting a score threshold</h3><!--SEC END --><p>If you want to use a less arbitrary way of selecting thresholds, you
can explore the distribution of PSSM scores. Since the space for a score
distribution grows exponentially with motif length, we are using an
approximation with a given precision to keep computation cost manageable:
</p><pre class="verbatim">&gt;&gt;&gt; distribution = pssm.distribution(background=background, precision=10**4)
</pre><p>The <code>distribution</code> object can be used to determine a number of different thresholds.
We can specify the requested false-positive rate (probability of &#X201C;finding&#X201D; a motif instance in background generated sequence):
</p><pre class="verbatim">&gt;&gt;&gt; threshold = distribution.threshold_fpr(0.01)
&gt;&gt;&gt; print("%5.3f" % threshold)
4.009
</pre><p>or the false-negative rate (probability of &#X201C;not finding&#X201D; an instance generated from the motif):
</p><pre class="verbatim">&gt;&gt;&gt; threshold = distribution.threshold_fnr(0.1)
&gt;&gt;&gt; print("%5.3f" % threshold)
-0.510
</pre><p>or a threshold (approximately) satisfying some relation between the false-positive rate and the false-negative rate (fnr/fpr&#X2243; <span style="font-style:italic">t</span>):
</p><pre class="verbatim">&gt;&gt;&gt; threshold = distribution.threshold_balanced(1000)
&gt;&gt;&gt; print("%5.3f" % threshold)
6.241
</pre><p>or a threshold satisfying (roughly) the equality between the
false-positive rate and the &#X2212;<span style="font-style:italic">log</span> of the information content (as used
in patser software by Hertz and Stormo):
</p><pre class="verbatim">&gt;&gt;&gt; threshold = distribution.threshold_patser()
&gt;&gt;&gt; print("%5.3f" % threshold)
0.346
</pre><p>For example, in case of our motif, you can get the threshold giving
you exactly the same results (for this sequence) as searching for
instances with balanced threshold with rate of 1000.
</p><pre class="verbatim">&gt;&gt;&gt; threshold = distribution.threshold_fpr(0.01)
&gt;&gt;&gt; print("%5.3f" % threshold)
4.009
&gt;&gt;&gt; for position, score in pssm.search(test_seq, threshold=threshold):
...     print("Position %d: score = %5.3f" % (position, score))
... 
Position 0: score = 5.622
Position -20: score = 4.601
Position 13: score = 5.738
Position -6: score = 4.601
</pre>
<!--TOC section id="sec265" Each motif object has an associated Position-Specific Scoring Matrix-->
<h2 id="sec265" class="section">14.7&#XA0;&#XA0;Each motif object has an associated Position-Specific Scoring Matrix</h2><!--SEC END --><p>To facilitate searching for potential TFBSs using PSSMs, both the position-weight matrix and the position-specific scoring matrix are associated with each motif. Using the Arnt motif as an example:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio import motifs
&gt;&gt;&gt; with open("Arnt.sites") as handle:
...     motif = motifs.read(handle, 'sites')
...
&gt;&gt;&gt; print(motif.counts)
        0      1      2      3      4      5
A:   4.00  19.00   0.00   0.00   0.00   0.00
C:  16.00   0.00  20.00   0.00   0.00   0.00
G:   0.00   1.00   0.00  20.00   0.00  20.00
T:   0.00   0.00   0.00   0.00  20.00   0.00
&lt;BLANKLINE&gt;
&gt;&gt;&gt; print(motif.pwm)
        0      1      2      3      4      5
A:   0.20   0.95   0.00   0.00   0.00   0.00
C:   0.80   0.00   1.00   0.00   0.00   0.00
G:   0.00   0.05   0.00   1.00   0.00   1.00
T:   0.00   0.00   0.00   0.00   1.00   0.00
&lt;BLANKLINE&gt;
</pre><pre class="verbatim">&gt;&gt;&gt; print(motif.pssm)
        0      1      2      3      4      5
A:  -0.32   1.93   -inf   -inf   -inf   -inf
C:   1.68   -inf   2.00   -inf   -inf   -inf
G:   -inf  -2.32   -inf   2.00   -inf   2.00
T:   -inf   -inf   -inf   -inf   2.00   -inf
&lt;BLANKLINE&gt;
</pre><p>The negative infinities appear here because the corresponding entry in the frequency matrix is 0, and we are using zero pseudocounts by default:
</p><pre class="verbatim">&gt;&gt;&gt; for letter in "ACGT":
...     print("%s: %4.2f" % (letter, motif.pseudocounts[letter]))
... 
A: 0.00
C: 0.00
G: 0.00
T: 0.00
</pre><p>If you change the <code>.pseudocounts</code> attribute, the position-frequency matrix and the position-specific scoring matrix are recalculated automatically:
</p><pre class="verbatim">&gt;&gt;&gt; motif.pseudocounts = 3.0
&gt;&gt;&gt; for letter in "ACGT":
...     print("%s: %4.2f" % (letter, motif.pseudocounts[letter]))
... 
A: 3.00
C: 3.00
G: 3.00
T: 3.00
</pre><pre class="verbatim">&gt;&gt;&gt; print(motif.pwm)
        0      1      2      3      4      5
A:   0.22   0.69   0.09   0.09   0.09   0.09
C:   0.59   0.09   0.72   0.09   0.09   0.09
G:   0.09   0.12   0.09   0.72   0.09   0.72
T:   0.09   0.09   0.09   0.09   0.72   0.09
&lt;BLANKLINE&gt;
</pre><pre class="verbatim">&gt;&gt;&gt; print(motif.pssm)
        0      1      2      3      4      5
A:  -0.19   1.46  -1.42  -1.42  -1.42  -1.42
C:   1.25  -1.42   1.52  -1.42  -1.42  -1.42
G:  -1.42  -1.00  -1.42   1.52  -1.42   1.52
T:  -1.42  -1.42  -1.42  -1.42   1.52  -1.42
&lt;BLANKLINE&gt;
</pre><p>You can also set the <code>.pseudocounts</code> to a dictionary over the four nucleotides if you want to use different pseudocounts for them. Setting <code>motif.pseudocounts</code> to <code>None</code> resets it to its default value of zero.</p><p>The position-specific scoring matrix depends on the background distribution, which is uniform by default:
</p><pre class="verbatim">&gt;&gt;&gt; for letter in "ACGT":
...     print("%s: %4.2f" % (letter, motif.background[letter]))
... 
A: 0.25
C: 0.25
G: 0.25
T: 0.25
</pre><p>Again, if you modify the background distribution, the position-specific scoring matrix is recalculated:
</p><pre class="verbatim">&gt;&gt;&gt; motif.background = {'A': 0.2, 'C': 0.3, 'G': 0.3, 'T': 0.2}
&gt;&gt;&gt; print(motif.pssm)
        0      1      2      3      4      5
A:   0.13   1.78  -1.09  -1.09  -1.09  -1.09
C:   0.98  -1.68   1.26  -1.68  -1.68  -1.68
G:  -1.68  -1.26  -1.68   1.26  -1.68   1.26
T:  -1.09  -1.09  -1.09  -1.09   1.85  -1.09
&lt;BLANKLINE&gt;
</pre><p>Setting <code>motif.background</code> to <code>None</code> resets it to a uniform distribution:
</p><pre class="verbatim">&gt;&gt;&gt; motif.background = None
&gt;&gt;&gt; for letter in "ACGT":
...     print("%s: %4.2f" % (letter, motif.background[letter]))
... 
A: 0.25
C: 0.25
G: 0.25
T: 0.25
</pre><p>If you set <code>motif.background</code> equal to a single value, it will be interpreted as the GC content:
</p><pre class="verbatim">&gt;&gt;&gt; motif.background = 0.8
&gt;&gt;&gt; for letter in "ACGT":
...     print("%s: %4.2f" % (letter, motif.background[letter]))
... 
A: 0.10
C: 0.40
G: 0.40
T: 0.10
</pre><p>Note that you can now calculate the mean of the PSSM scores over the background against which it was computed:
</p><pre class="verbatim">&gt;&gt;&gt; print("%f" % motif.pssm.mean(motif.background))
4.703928
</pre><p>as well as its standard deviation:
</p><pre class="verbatim">&gt;&gt;&gt; print("%f" % motif.pssm.std(motif.background))
3.290900
</pre><p>and its distribution:
</p><pre class="verbatim">&gt;&gt;&gt; distribution = motif.pssm.distribution(background=motif.background)
&gt;&gt;&gt; threshold = distribution.threshold_fpr(0.01)
&gt;&gt;&gt; print("%f" % threshold)
3.854375
</pre><p>Note that the position-weight matrix and the position-specific scoring matrix are recalculated each time you call <code>motif.pwm</code> or <code>motif.pssm</code>, respectively. If speed is an issue and you want to use the PWM or PSSM repeatedly, you can save them as a variable, as in
</p><pre class="verbatim">&gt;&gt;&gt; pssm = motif.pssm
</pre>
<!--TOC section id="sec266" Comparing motifs-->
<h2 id="sec266" class="section">14.8&#XA0;&#XA0;Comparing motifs</h2><!--SEC END --><p>
<a id="sec:comp"></a>
Once we have more than one motif, we might want to compare them.</p><p>Before we start comparing motifs, I should point out that motif
boundaries are usually quite arbitrary. This means we often need
to compare motifs of different lengths, so comparison needs to involve
some kind of alignment. This means we have to take into account two things:
</p><ul class="itemize"><li class="li-itemize">
alignment of motifs
</li><li class="li-itemize">some function to compare aligned motifs
</li></ul><p>
To align the motifs, we use ungapped alignment of PSSMs and substitute zeros
for any missing columns at the beginning and end of the matrices. This means
that effectively we are using the background distribution for columns missing
from the PSSM.
The distance function then returns the minimal distance between motifs, as
well as the corresponding offset in their alignment.</p><p>To give an exmaple, let us first load another motif,
which is similar to our test motif <code>m</code>:
</p><pre class="verbatim">&gt;&gt;&gt; with open("REB1.pfm") as handle:
...    m_reb1 = motifs.read(handle, "pfm")
...
&gt;&gt;&gt; m_reb1.consensus
Seq('GTTACCCGG', IUPACUnambiguousDNA())
&gt;&gt;&gt; print(m_reb1.counts)
        0      1      2      3      4      5      6      7      8
A:  30.00   0.00   0.00 100.00   0.00   0.00   0.00   0.00  15.00
C:  10.00   0.00   0.00   0.00 100.00 100.00 100.00   0.00  15.00
G:  50.00   0.00   0.00   0.00   0.00   0.00   0.00  60.00  55.00
T:  10.00 100.00 100.00   0.00   0.00   0.00   0.00  40.00  15.00
&lt;BLANKLINE&gt;
</pre><p>To make the motifs comparable, we choose the same values for the pseudocounts and the background distribution as our motif <code>m</code>:
</p><pre class="verbatim">&gt;&gt;&gt; m_reb1.pseudocounts = {'A':0.6, 'C': 0.4, 'G': 0.4, 'T': 0.6}
&gt;&gt;&gt; m_reb1.background = {'A':0.3,'C':0.2,'G':0.2,'T':0.3}
&gt;&gt;&gt; pssm_reb1 = m_reb1.pssm
&gt;&gt;&gt; print(pssm_reb1)
        0      1      2      3      4      5      6      7      8
A:   0.00  -5.67  -5.67   1.72  -5.67  -5.67  -5.67  -5.67  -0.97
C:  -0.97  -5.67  -5.67  -5.67   2.30   2.30   2.30  -5.67  -0.41
G:   1.30  -5.67  -5.67  -5.67  -5.67  -5.67  -5.67   1.57   1.44
T:  -1.53   1.72   1.72  -5.67  -5.67  -5.67  -5.67   0.41  -0.97
&lt;BLANKLINE&gt;
</pre><p>We&#X2019;ll compare these motifs using the Pearson correlation.
Since we want it to resemble a distance measure, we actually take
1&#X2212;<span style="font-style:italic">r</span>, where <span style="font-style:italic">r</span> is the Pearson correlation coefficient (PCC):
</p><pre class="verbatim">&gt;&gt;&gt; distance, offset = pssm.dist_pearson(pssm_reb1)
&gt;&gt;&gt; print("distance = %5.3g" % distance)
distance = 0.239
&gt;&gt;&gt; print(offset)
-2
</pre><p>This means that the best PCC between motif <code>m</code> and <code>m_reb1</code> is obtained with the following alignment:
</p><pre class="verbatim">m:      bbTACGCbb
m_reb1: GTTACCCGG
</pre><p>where <code>b</code> stands for background distribution. The PCC itself is
roughly 1&#X2212;0.239=0.761.</p>
<!--TOC section id="sec267" <em>De novo</em> motif finding-->
<h2 id="sec267" class="section">14.9&#XA0;&#XA0;<em>De novo</em> motif finding</h2><!--SEC END --><p>
<a id="sec:find"></a></p><p>Currently, Biopython has only limited support for <em>de novo</em> motif
finding. Namely, we support running and parsing of AlignAce and
MEME. Since the number of motif finding tools is growing rapidly, 
contributions of new parsers are welcome. </p>
<!--TOC subsection id="sec268" MEME-->
<h3 id="sec268" class="subsection">14.9.1&#XA0;&#XA0;MEME</h3><!--SEC END --><p>
<a id="sec:meme"></a></p><p>Let&#X2019;s assume, you have run MEME on sequences of your choice with your
favorite parameters and saved the output in the file
<code>meme.out</code>. You can retrieve the motifs reported by MEME by
running the following piece of code:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import motifs
&gt;&gt;&gt; with open("meme.out") as handle:
...     motifsM = motifs.parse(handle, "meme")
...
</pre><pre class="verbatim">&gt;&gt;&gt; motifsM
[&lt;Bio.motifs.meme.Motif object at 0xc356b0&gt;]
</pre><p>Besides the most wanted list of motifs, the result object contains more useful information, accessible through properties with self-explanatory names:
</p><ul class="itemize"><li class="li-itemize">
<code>.alphabet</code>
</li><li class="li-itemize"><code>.datafile</code>
</li><li class="li-itemize"><code>.sequence_names</code>
</li><li class="li-itemize"><code>.version</code>
</li><li class="li-itemize"><code>.command</code>
</li></ul><p>The motifs returned by the MEME Parser can be treated exactly like regular
Motif objects (with instances), they also provide some extra
functionality, by adding additional information about the instances. </p><pre class="verbatim">&gt;&gt;&gt; motifsM[0].consensus
Seq('CTCAATCGTA', IUPACUnambiguousDNA())
&gt;&gt;&gt; motifsM[0].instances[0].sequence_name
'SEQ10;'
&gt;&gt;&gt; motifsM[0].instances[0].start
3
&gt;&gt;&gt; motifsM[0].instances[0].strand
'+'
</pre><pre class="verbatim">&gt;&gt;&gt; motifsM[0].instances[0].pvalue
8.71e-07
</pre>
<!--TOC subsection id="sec269" AlignAce-->
<h3 id="sec269" class="subsection">14.9.2&#XA0;&#XA0;AlignAce</h3><!--SEC END --><p>
<a id="sec:alignace"></a></p><p>We can do very similar things with the AlignACE program. Assume, you have
your output in the file <code>alignace.out</code>. You can parse your output
with the following code:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import motifs
&gt;&gt;&gt; with open("alignace.out") as handle:
...     motifsA = motifs.parse(handle, "alignace")
...
</pre><p>Again, your motifs behave as they should:
</p><pre class="verbatim">&gt;&gt;&gt; motifsA[0].consensus
Seq('TCTACGATTGAG', IUPACUnambiguousDNA())
</pre><p>In fact you can even see, that AlignAce found a very similar motif as
MEME. It is just a longer version of a reverse complement of the MEME
motif:
</p><pre class="verbatim">&gt;&gt;&gt; motifsM[0].reverse_complement().consensus
Seq('TACGATTGAG', IUPACUnambiguousDNA())
</pre><p>If you have AlignAce installed on the same machine, you can also run
it directly from Biopython. A short example of how this can be done is
shown below (other parameters can be specified as keyword parameters):</p><pre class="verbatim">&gt;&gt;&gt; command="/opt/bin/AlignACE"
&gt;&gt;&gt; input_file="test.fa"
&gt;&gt;&gt; from Bio.motifs.applications import AlignAceCommandline
&gt;&gt;&gt; cmd = AlignAceCommandline(cmd=command, input=input_file, gcback=0.6, numcols=10)
&gt;&gt;&gt; stdout, stderr= cmd()
</pre><p>Since AlignAce prints all of its output to standard output, you can get
to your motifs by parsing the first part of the result:
</p><pre class="verbatim">&gt;&gt;&gt; motifs = motifs.parse(stdout, "alignace")
</pre>
<!--TOC section id="sec270" Useful links -->
<h2 id="sec270" class="section">14.10&#XA0;&#XA0;Useful links </h2><!--SEC END --><p>
<a id="sec:links"></a></p><ul class="itemize"><li class="li-itemize">
<a href="http://en.wikipedia.org/wiki/Sequence_motif">Sequence motif</a> in wikipedia
</li><li class="li-itemize"><a href="http://en.wikipedia.org/wiki/Position_weight_matrix">PWM</a> in wikipedia
</li><li class="li-itemize"><a href="http://en.wikipedia.org/wiki/Consensus_sequence">Consensus sequence</a> in wikipedia
</li><li class="li-itemize"><a href="http://bio.cs.washington.edu/assessment/">Comparison of different motif finding programs</a> 
</li></ul>
<!--TOC chapter id="sec271" Cluster analysis-->
<h1 id="sec271" class="chapter">Chapter&#XA0;15&#XA0;&#XA0;Cluster analysis</h1><!--SEC END --><p>Cluster analysis is the grouping of items into clusters based on the similarity of the items to each other. In bioinformatics, clustering is widely used in gene expression data analysis to find groups of genes with similar gene expression profiles. This may identify functionally related genes, as well as suggest the function of presently unknown genes.</p><p>The Biopython module <code>Bio.Cluster</code> provides commonly used clustering algorithms and was designed with the application to gene expression data in mind. However, this module can also be used for cluster analysis of other types of data. <code>Bio.Cluster</code> and the underlying C Clustering Library is described by De Hoon <span style="font-style:italic">et al.</span> [<a href="#dehoon2004">14</a>].</p><p>The following four clustering approaches are implemented in <code>Bio.Cluster</code>:
</p><ul class="itemize"><li class="li-itemize">
Hierarchical clustering (pairwise centroid-, single-, complete-, and average-linkage);
</li><li class="li-itemize"><span style="font-style:italic">k</span>-means, <span style="font-style:italic">k</span>-medians, and <span style="font-style:italic">k</span>-medoids clustering;
</li><li class="li-itemize">Self-Organizing Maps;
</li><li class="li-itemize">Principal Component Analysis.
</li></ul><!--TOC subsection id="sec272" Data representation-->
<h3 id="sec272" class="subsection">Data representation</h3><!--SEC END --><p>The data to be clustered are represented by a <span style="font-style:italic">n</span> &#XD7; <span style="font-style:italic">m</span> Numerical Python array <code>data</code>. Within the context of gene expression data clustering, typically the rows correspond to different genes whereas the columns correspond to different experimental conditions. The clustering algorithms in <code>Bio.Cluster</code> can be applied both to rows (genes) and to columns (experiments).</p><!--TOC subsection id="sec273" Missing values-->
<h3 id="sec273" class="subsection">Missing values</h3><!--SEC END --><p>Often in microarray experiments, some of the data values are missing, which is indicated by an additional <span style="font-style:italic">n</span> &#XD7; <span style="font-style:italic">m</span> Numerical Python integer array <code>mask</code>. If <code>mask[i,j]==0</code>, then <code>data[i,j]</code> is missing and is ignored in the analysis.</p><!--TOC subsection id="sec274" Random number generator-->
<h3 id="sec274" class="subsection">Random number generator</h3><!--SEC END --><p>The <span style="font-style:italic">k</span>-means/medians/medoids clustering algorithms and Self-Organizing Maps (SOMs) include the use of a random number generator. The uniform random number generator in <code>Bio.Cluster</code> is based on the algorithm by L&#X2019;Ecuyer [<a href="#lecuyer1988">25</a>], while random numbers following the binomial distribution are generated using the BTPE algorithm by Kachitvichyanukul and Schmeiser [<a href="#kachitvichyanukul1988">23</a>]. The random number generator is initialized automatically during its first call. As this random number generator uses a combination of two multiplicative linear congruential generators, two (integer) seeds are needed for initialization, for which we use the system-supplied random number generator <code>rand</code> (in the C standard library). We initialize this generator by calling <code>srand</code> with the epoch time in seconds, and use the first two random numbers generated by <code>rand</code> as seeds for the uniform random number generator in <code>Bio.Cluster</code>.</p>
<!--TOC section id="sec275" Distance functions-->
<h2 id="sec275" class="section">15.1&#XA0;&#XA0;Distance functions</h2><!--SEC END --><p>
<a id="sec:distancefunctions"></a></p><p>In order to cluster items into groups based on their similarity, we should first define what exactly we mean by <em>similar</em>. <code>Bio.Cluster</code> provides eight distance functions, indicated by a single character, to measure similarity, or conversely, distance:
</p><ul class="itemize"><li class="li-itemize">
<code>'e'</code>:
Euclidean distance;
</li><li class="li-itemize"><code>'b'</code>:
City-block distance.
</li><li class="li-itemize"><code>'c'</code>:
Pearson correlation coefficient;
</li><li class="li-itemize"><code>'a'</code>:
Absolute value of the Pearson correlation coefficient;
</li><li class="li-itemize"><code>'u'</code>:
Uncentered Pearson correlation (equivalent to the cosine of the angle between two data vectors);
</li><li class="li-itemize"><code>'x'</code>:
Absolute uncentered Pearson correlation;
</li><li class="li-itemize"><code>'s'</code>:
Spearman&#X2019;s rank correlation;
</li><li class="li-itemize"><code>'k'</code>:
Kendall&#X2019;s &#X3C4;.
</li></ul><p>
The first two are true distance functions that satisfy the triangle inequality:
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">d</span></td><td class="dcell">&#X239B;<br>
&#X239C;<br>
&#X239D;</td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">u</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell">,</td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">v</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell">&#X239E;<br>
&#X239F;<br>
&#X23A0;</td><td class="dcell">&#X2264;&#XA0;<span style="font-style:italic">d</span></td><td class="dcell">&#X239B;<br>
&#X239C;<br>
&#X239D;</td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">u</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell">,</td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">w</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell">&#X239E;<br>
&#X239F;<br>
&#X23A0;</td><td class="dcell">+&#XA0;<span style="font-style:italic">d</span></td><td class="dcell">&#X239B;<br>
&#X239C;<br>
&#X239D;</td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">w</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell">,</td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">v</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell">&#X239E;<br>
&#X239F;<br>
&#X23A0;</td><td class="dcell"> for all &#XA0;</td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">u</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell">,&#XA0;</td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">v</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell">,&#XA0;</td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">w</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell">,</td></tr>
</table><p>
and are therefore refered to as <em>metrics</em>. In everyday language, this means that the shortest distance between two points is a straight line.</p><p>The remaining six distance measures are related to the correlation coefficient, where the distance <span style="font-style:italic">d</span> is defined in terms of the correlation <span style="font-style:italic">r</span> by <span style="font-style:italic">d</span>=1&#X2212;<span style="font-style:italic">r</span>. Note that these distance functions are <em>semi-metrics</em> that do not satisfy the triangle inequality. For example, for
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">u</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell">=</td><td class="dcell">&#X239B;<br>
&#X239D;</td><td class="dcell">1,0,&#X2212;1</td><td class="dcell">&#X239E;<br>
&#X23A0;</td><td class="dcell">;</td></tr>
</table><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">v</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell">=</td><td class="dcell">&#X239B;<br>
&#X239D;</td><td class="dcell">1,1,0</td><td class="dcell">&#X239E;<br>
&#X23A0;</td><td class="dcell">;</td></tr>
</table><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">w</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell">=</td><td class="dcell">&#X239B;<br>
&#X239D;</td><td class="dcell">0,1,1</td><td class="dcell">&#X239E;<br>
&#X23A0;</td><td class="dcell">;</td></tr>
</table><p>
we find a Pearson distance
<span style="font-style:italic">d</span>(<U><span style="font-style:italic">u</span></U>,<U><span style="font-style:italic">w</span></U>) = 1.8660, while
<span style="font-style:italic">d</span>(<U><span style="font-style:italic">u</span></U>,<U><span style="font-style:italic">v</span></U>)+<span style="font-style:italic">d</span>(<U><span style="font-style:italic">v</span></U>,<U><span style="font-style:italic">w</span></U>) = 1.6340.</p><!--TOC subsection id="sec276" Euclidean distance-->
<h3 id="sec276" class="subsection">Euclidean distance</h3><!--SEC END --><p>In <code>Bio.Cluster</code>, we define the Euclidean distance as
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">d</span>&#XA0;=&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center">1&#XA0;</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-size:xx-large">&#X2211;</span></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">i</span>=1</td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell">&#X239B;<br>
&#X239D;</td><td class="dcell"><span style="font-style:italic">x</span><sub><span style="font-style:italic">i</span></sub>&#X2212;<span style="font-style:italic">y</span><sub><span style="font-style:italic">i</span></sub></td><td class="dcell">&#X239E;<br>
&#X23A0;</td><td class="dcell"><sup>2</sup>.</td></tr>
</table><p>
Only those terms are included in the summation for which both
<span style="font-style:italic">x</span><sub><span style="font-style:italic">i</span></sub> and <span style="font-style:italic">y</span><sub><span style="font-style:italic">i</span></sub> are present, and the denominator <span style="font-style:italic">n</span> is chosen accordingly.
As the expression data <span style="font-style:italic">x</span><sub><span style="font-style:italic">i</span></sub> and <span style="font-style:italic">y</span><sub><span style="font-style:italic">i</span></sub> are subtracted directly from each other, we should make sure that the expression data are properly normalized when using the Euclidean distance.</p><!--TOC subsection id="sec277" City-block distance-->
<h3 id="sec277" class="subsection">City-block distance</h3><!--SEC END --><p>The city-block distance, alternatively known as the Manhattan distance, is related to the Euclidean distance. Whereas the Euclidean distance corresponds to the length of the shortest path between two points, the city-block distance is the sum of distances along each dimension. As gene expression data tend to have missing values, in <code>Bio.Cluster</code> we define the city-block distance as the sum of distances divided by the number of dimensions:
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">d</span>&#XA0;=&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center">1&#XA0;</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-size:xx-large">&#X2211;</span></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">i</span>=1</td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell">&#X23AA;<br>
&#X23AA;</td><td class="dcell"><span style="font-style:italic">x</span><sub><span style="font-style:italic">i</span></sub>&#X2212;<span style="font-style:italic">y</span><sub><span style="font-style:italic">i</span></sub></td><td class="dcell">&#X23AA;<br>
&#X23AA;</td><td class="dcell">.</td></tr>
</table><p>
This is equal to the distance you would have to walk between two points in a city, where you have to walk along city blocks. As for the Euclidean distance,
the expression data are subtracted directly from each other, and we should therefore make sure that they are properly normalized.</p><!--TOC subsection id="sec278" The Pearson correlation coefficient-->
<h3 id="sec278" class="subsection">The Pearson correlation coefficient</h3><!--SEC END --><p>The Pearson correlation coefficient is defined as
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">r</span>&#XA0;=&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-size:xx-large">&#X2211;</span></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">i</span>=1</td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell">&#X239B;<br>
&#X239C;<br>
&#X239C;<br>
&#X239D;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">x</span><sub><span style="font-style:italic">i</span></sub>&#XA0;&#X2212;<span style="text-decoration:overline">x</span></td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center">&#X3C3;<sub><span style="font-style:italic">x</span></sub></td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell">&#X239E;<br>
&#X239F;<br>
&#X239F;<br>
&#X23A0;</td><td class="dcell">&#X239B;<br>
&#X239C;<br>
&#X239C;<br>
&#X239D;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">y</span><sub><span style="font-style:italic">i</span></sub>&#XA0;&#X2212;&#X233;</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center">&#X3C3;<sub><span style="font-style:italic">y</span></sub></td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell">&#X239E;<br>
&#X239F;<br>
&#X239F;<br>
&#X23A0;</td><td class="dcell">,</td></tr>
</table><p>
in which
<span style="text-decoration:overline">x</span>, &#X233;
are the sample mean of <span style="font-style:italic">x</span> and <span style="font-style:italic">y</span> respectively, and
&#X3C3;<sub><span style="font-style:italic">x</span></sub>, &#X3C3;<sub><span style="font-style:italic">y</span></sub>
are the sample standard deviation of <span style="font-style:italic">x</span> and <span style="font-style:italic">y</span>.
The Pearson correlation coefficient is a measure for how well a straight line can be fitted to a scatterplot of <span style="font-style:italic">x</span> and <span style="font-style:italic">y</span>.
If all the points in the scatterplot lie on a straight line, the Pearson correlation coefficient is either +1 or -1, depending on whether the slope of line is positive or negative. If the Pearson correlation coefficient is equal to zero, there is no correlation between <span style="font-style:italic">x</span> and <span style="font-style:italic">y</span>.</p><p>The <em>Pearson distance</em> is then defined as 
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">d</span><sub>P</sub>&#XA0;&#X2261;&#XA0;1&#XA0;&#X2212;&#XA0;<span style="font-style:italic">r</span>.</td></tr>
</table><p>
As the Pearson correlation coefficient lies between -1 and 1, the Pearson distance lies between 0 and 2.</p><!--TOC subsection id="sec279" Absolute Pearson correlation-->
<h3 id="sec279" class="subsection">Absolute Pearson correlation</h3><!--SEC END --><p>By taking the absolute value of the Pearson correlation, we find a number between 0 and 1. If the absolute value is 1, all the points in the scatter plot lie on a straight line with either a positive or a negative slope. If the absolute value is equal to zero, there is no correlation between <span style="font-style:italic">x</span> and <span style="font-style:italic">y</span>.</p><p>The corresponding distance is defined as
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">d</span><sub>A</sub>&#XA0;&#X2261;&#XA0;1&#XA0;&#X2212;&#XA0;</td><td class="dcell">&#X23AA;<br>
&#X23AA;</td><td class="dcell"><span style="font-style:italic">r</span></td><td class="dcell">&#X23AA;<br>
&#X23AA;</td><td class="dcell">,</td></tr>
</table><p>
where <span style="font-style:italic">r</span> is the Pearson correlation coefficient. As the absolute value of the Pearson correlation coefficient lies between 0 and 1, the corresponding distance lies between 0 and 1 as well.</p><p>In the context of gene expression experiments, the absolute correlation is equal to 1 if the gene expression profiles of two genes are either exactly the same or exactly opposite. The absolute correlation coefficient should therefore be used with care.</p><!--TOC subsection id="sec280" Uncentered correlation (cosine of the angle)-->
<h3 id="sec280" class="subsection">Uncentered correlation (cosine of the angle)</h3><!--SEC END --><p>In some cases, it may be preferable to use the <em>uncentered correlation</em> instead of the regular Pearson correlation coefficient. The uncentered correlation is defined as
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">r</span><sub>U</sub>&#XA0;=&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-size:xx-large">&#X2211;</span></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">i</span>=1</td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell">&#X239B;<br>
&#X239C;<br>
&#X239C;<br>
&#X239D;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">x</span><sub><span style="font-style:italic">i</span></sub></td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center">&#X3C3;<sub><span style="font-style:italic">x</span></sub><sup>(0)</sup></td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell">&#X239E;<br>
&#X239F;<br>
&#X239F;<br>
&#X23A0;</td><td class="dcell">&#X239B;<br>
&#X239C;<br>
&#X239C;<br>
&#X239D;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">y</span><sub><span style="font-style:italic">i</span></sub></td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center">&#X3C3;<sub><span style="font-style:italic">y</span></sub><sup>(0)</sup></td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell">&#X239E;<br>
&#X239F;<br>
&#X239F;<br>
&#X23A0;</td><td class="dcell">,</td></tr>
</table><p>
where
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell">


&#XA0;&#XA0;&#XA0;&#XA0;&#XA0;

</td><td class="dcell"><table style="border-spacing:6px;border-collapse:separate;" class="cellpading0"><tr><td style="text-align:right;white-space:nowrap" >&#X3C3;<sub><span style="font-style:italic">x</span></sub><sup>(0)</sup></td><td style="text-align:center;white-space:nowrap" >&#XA0;=</td><td style="text-align:left;white-space:nowrap" ><table class="display"><tr style="vertical-align:middle"><td class="dcell">&#XA0;</td><td class="dcell"><table style="border-spacing:0" class="cellpadding0"><tr><td ALIGN="right"><div CLASS="vbar" STYLE="height:2em;"></div></td></tr>
<tr><td><span style="font-size:xx-large"><span style="font-size:150%">&#X221A;</span></span></td></tr>
</table></td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td class="hbar"></td></tr>
<tr><td style="text-align:center;white-space:nowrap" ><table class="display"><tr style="vertical-align:middle"><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-size:xx-large">&#X2211;</span></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">i</span>=1</td></tr>
</table></td><td class="dcell"><span style="font-style:italic">x</span><sub><span style="font-style:italic">i</span></sub><sup>2</sup></td></tr>
</table></td></tr>
</table></td><td class="dcell">;&#XA0;&#XA0;</td></tr>
</table></td><td style="text-align:right;white-space:nowrap" >&nbsp;</td></tr>
<tr><td style="text-align:right;white-space:nowrap" >&#X3C3;<sub><span style="font-style:italic">y</span></sub><sup>(0)</sup></td><td style="text-align:center;white-space:nowrap" >&#XA0;=</td><td style="text-align:left;white-space:nowrap" ><table class="display"><tr style="vertical-align:middle"><td class="dcell">&#XA0;</td><td class="dcell"><table style="border-spacing:0" class="cellpadding0"><tr><td ALIGN="right"><div CLASS="vbar" STYLE="height:2em;"></div></td></tr>
<tr><td><span style="font-size:xx-large"><span style="font-size:150%">&#X221A;</span></span></td></tr>
</table></td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td class="hbar"></td></tr>
<tr><td style="text-align:center;white-space:nowrap" ><table class="display"><tr style="vertical-align:middle"><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-size:xx-large">&#X2211;</span></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">i</span>=1</td></tr>
</table></td><td class="dcell"><span style="font-style:italic">y</span><sub><span style="font-style:italic">i</span></sub><sup>2</sup></td></tr>
</table></td></tr>
</table></td><td class="dcell">.&#XA0;&#XA0;
</td></tr>
</table></td><td style="text-align:right;white-space:nowrap" >&nbsp;</td></tr>
</table></td></tr>
</table><p>
This is the same expression as for the regular Pearson correlation coefficient, except that the sample means
<span style="text-decoration:overline">x</span>, &#X233;
are set equal to zero. The uncentered correlation may be appropriate if there is a zero reference state. For instance, in the case of gene expression data given in terms of log-ratios, a log-ratio equal to zero corresponds to the green and red signal being equal, which means that the experimental manipulation did not affect the gene expression.</p><p>The distance corresponding to the uncentered correlation coefficient is defined as 
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">d</span><sub>U</sub>&#XA0;&#X2261;&#XA0;1&#XA0;&#X2212;&#XA0;<span style="font-style:italic">r</span><sub>U</sub>,</td></tr>
</table><p>
where
<span style="font-style:italic">r</span><sub>U</sub>
is the uncentered correlation.
As the uncentered correlation coefficient lies between -1 and 1, the corresponding distance lies between 0 and 2.</p><p>The uncentered correlation is equal to the cosine of the angle of the two data vectors in <span style="font-style:italic">n</span>-dimensional space, and is often referred to as such.</p><!--TOC subsection id="sec281" Absolute uncentered correlation-->
<h3 id="sec281" class="subsection">Absolute uncentered correlation</h3><!--SEC END --><p>As for the regular Pearson correlation, we can define a distance measure using the absolute value of the uncentered correlation:
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">d</span><sub>AU</sub>&#XA0;&#X2261;&#XA0;1&#XA0;&#X2212;&#XA0;</td><td class="dcell">&#X23AA;<br>
&#X23AA;</td><td class="dcell"><span style="font-style:italic">r</span><sub>U</sub></td><td class="dcell">&#X23AA;<br>
&#X23AA;</td><td class="dcell">,</td></tr>
</table><p>
where
<span style="font-style:italic">r</span><sub>U</sub>
is the uncentered correlation coefficient. As the absolute value of the uncentered correlation coefficient lies between 0 and 1, the corresponding distance lies between 0 and 1 as well.</p><p>Geometrically, the absolute value of the uncentered correlation is equal to the cosine between the supporting lines of the two data vectors (i.e., the angle without taking the direction of the vectors into consideration).</p><!--TOC subsection id="sec282" Spearman rank correlation-->
<h3 id="sec282" class="subsection">Spearman rank correlation</h3><!--SEC END --><p>The Spearman rank correlation is an example of a non-parametric similarity measure, and tends to be more robust against outliers than the Pearson correlation.</p><p>To calculate the Spearman rank correlation, we replace each data value by their rank if we would order the data in each vector by their value. We then calculate the Pearson correlation between the two rank vectors instead of the data vectors.</p><p>As in the case of the Pearson correlation, we can define a distance measure corresponding to the Spearman rank correlation as
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">d</span><sub>S</sub>&#XA0;&#X2261;&#XA0;1&#XA0;&#X2212;&#XA0;<span style="font-style:italic">r</span><sub>S</sub>,</td></tr>
</table><p>
where
<span style="font-style:italic">r</span><sub>S</sub>
is the Spearman rank correlation.</p><!--TOC subsection id="sec283" Kendall&#X2019;s &#X3C4;-->
<h3 id="sec283" class="subsection">Kendall&#X2019;s &#X3C4;</h3><!--SEC END --><p>Kendall&#X2019;s &#X3C4;
is another example of a non-parametric similarity measure. It is similar to the Spearman rank correlation, but instead of the ranks themselves only the relative ranks are used to calculate &#X3C4; (see Snedecor &amp; Cochran [<a href="#snedecor1989">29</a>]).</p><p>We can define a distance measure corresponding to Kendall&#X2019;s &#X3C4;
as </p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">d</span><sub>K</sub>&#XA0;&#X2261;&#XA0;1&#XA0;&#X2212;&#XA0;&#X3C4;.</td></tr>
</table><p>
As Kendall&#X2019;s &#X3C4; is always between -1 and 1, the corresponding distance will be between 0 and 2.</p><!--TOC subsection id="sec284" Weighting-->
<h3 id="sec284" class="subsection">Weighting</h3><!--SEC END --><p>For most of the distance functions available in <code>Bio.Cluster</code>, a weight vector can be applied. The weight vector contains weights for the items in the data vector. If the weight for item <span style="font-style:italic">i</span> is <span style="font-style:italic">w</span><sub><span style="font-style:italic">i</span></sub>, then that item is treated as if it occurred <span style="font-style:italic">w</span><sub><span style="font-style:italic">i</span></sub> times in the data. The weight do not have to be integers.
For the Spearman rank correlation and Kendall&#X2019;s
&#X3C4;,
weights do not have a well-defined meaning and are therefore not implemented.</p><!--TOC subsection id="subsec:distancematrix" Calculating the distance matrix-->
<h3 id="subsec:distancematrix" class="subsection">Calculating the distance matrix</h3><!--SEC END --><p>The distance matrix is a square matrix with all pairwise distances between the items in <code>data</code>, and can be calculated by the function <code>distancematrix</code> in the <code>Bio.Cluster</code> module:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Cluster import distancematrix
&gt;&gt;&gt; matrix = distancematrix(data)
</pre><p>where the following arguments are defined:
</p><ul class="itemize"><li class="li-itemize">
<code>data</code> (required)<br>
Array containing the data for the items.
</li><li class="li-itemize"><code>mask</code> (default: <code>None</code>) <br>
Array of integers showing which data are missing. If <code>mask[i,j]==0</code>, then <code>data[i,j]</code> is missing. If <code>mask==None</code>, then all data are present.
</li><li class="li-itemize"><code>weight</code> (default: <code>None</code>) <br>
The weights to be used when calculating distances. If <code>weight==None</code>, then equal weights are assumed.
</li><li class="li-itemize"><code>transpose</code> (default: <code>0</code>) <br>
Determines if the distances between the rows of <code>data</code> are to be calculated (<code>transpose==0</code>), or between the columns of <code>data</code> (<code>transpose==1</code>).
</li><li class="li-itemize"><code>dist</code> (default: <code>'e'</code>, Euclidean distance) <br>
Defines the distance function to be used (see <a href="#sec%3Adistancefunctions">15.1</a>).
</li></ul><p>To save memory, the distance matrix is returned as a list of 1D arrays.
The number of columns in each row is equal to the row number. Hence, the first row has zero elements. An example of the return value is
</p><pre class="verbatim">[array([]),
 array([1.]),
 array([7., 3.]),
 array([4., 2., 6.])]
</pre><p>This corresponds to the distance matrix
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell">&#X239B;<br>
&#X239C;<br>
&#X239C;<br>
&#X239C;<br>
&#X239D;</td><td class="dcell">
</td><td class="dcell"><table style="border-spacing:6px;border-collapse:separate;" class="cellpading0"><tr><td style="text-align:center;white-space:nowrap" >0</td><td style="text-align:center;white-space:nowrap" >1</td><td style="text-align:center;white-space:nowrap" >7</td><td style="text-align:center;white-space:nowrap" >4&#XA0;&#XA0;</td></tr>
<tr><td style="text-align:center;white-space:nowrap" >1</td><td style="text-align:center;white-space:nowrap" >0</td><td style="text-align:center;white-space:nowrap" >3</td><td style="text-align:center;white-space:nowrap" >2&#XA0;&#XA0;</td></tr>
<tr><td style="text-align:center;white-space:nowrap" >7</td><td style="text-align:center;white-space:nowrap" >3</td><td style="text-align:center;white-space:nowrap" >0</td><td style="text-align:center;white-space:nowrap" >6&#XA0;&#XA0;</td></tr>
<tr><td style="text-align:center;white-space:nowrap" >4</td><td style="text-align:center;white-space:nowrap" >2</td><td style="text-align:center;white-space:nowrap" >6</td><td style="text-align:center;white-space:nowrap" >0
</td></tr>
</table></td><td class="dcell">
</td><td class="dcell">&#X239E;<br>
&#X239F;<br>
&#X239F;<br>
&#X239F;<br>
&#X23A0;</td><td class="dcell">.
</td></tr>
</table>
<!--TOC section id="sec286" Calculating cluster properties-->
<h2 id="sec286" class="section">15.2&#XA0;&#XA0;Calculating cluster properties</h2><!--SEC END --><!--TOC subsection id="subsec:clustercentroids" Calculating the cluster centroids-->
<h3 id="subsec:clustercentroids" class="subsection">Calculating the cluster centroids</h3><!--SEC END --><p>The centroid of a cluster can be defined either as the mean or as the median of each dimension over all cluster items. The function <code>clustercentroids</code> in <code>Bio.Cluster</code> can be used to calculate either:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Cluster import clustercentroids
&gt;&gt;&gt; cdata, cmask = clustercentroids(data)
</pre><p>where the following arguments are defined:
</p><ul class="itemize"><li class="li-itemize">
<code>data</code> (required) <br>
Array containing the data for the items.
</li><li class="li-itemize"><code>mask</code> (default: <code>None</code>) <br>
Array of integers showing which data are missing. If <code>mask[i,j]==0</code>, then <code>data[i,j]</code> is missing. If <code>mask==None</code>, then all data are present.
</li><li class="li-itemize"><code>clusterid</code> (default: <code>None</code>) <br>
Vector of integers showing to which cluster each item belongs. If <code>clusterid</code> is <code>None</code>, then all items are assumed to belong to the same cluster.
</li><li class="li-itemize"><code>method</code> (default: <code>'a'</code>) <br>
Specifies whether the arithmetic mean (<code>method=='a'</code>) or the median (<code>method=='m'</code>) is used to calculate the cluster center.
</li><li class="li-itemize"><code>transpose</code> (default: <code>0</code>) <br>
Determines if the centroids of the rows of <code>data</code> are to be calculated (<code>transpose==0</code>), or the centroids of the columns of <code>data</code> (<code>transpose==1</code>).
</li></ul><p>This function returns the tuple <code>(cdata, cmask)</code>. The centroid data are stored in the 2D Numerical Python array <code>cdata</code>, with missing data indicated by the 2D Numerical Python integer array <code>cmask</code>. The dimensions of these arrays are (number of clusters, number of columns) if <code>transpose</code> is <code>0</code>, or (number of rows, number of clusters) if <code>transpose</code> is <code>1</code>. Each row (if <code>transpose</code> is <code>0</code>) or column (if <code>transpose</code> is <code>1</code>) contains the averaged data corresponding to the centroid of each cluster.</p><!--TOC subsection id="sec288" Calculating the distance between clusters-->
<h3 id="sec288" class="subsection">Calculating the distance between clusters</h3><!--SEC END --><p>Given a distance function between <em>items</em>, we can define the distance between two <em>clusters</em> in several ways. The distance between the arithmetic means of the two clusters is used in pairwise centroid-linkage clustering and in <span style="font-style:italic">k</span>-means clustering. In <span style="font-style:italic">k</span>-medoids clustering, the distance between the medians of the two clusters is used instead. The shortest pairwise distance between items of the two clusters is used in pairwise single-linkage clustering, while the longest pairwise distance is used in pairwise maximum-linkage clustering. In pairwise average-linkage clustering, the distance between two clusters is defined as the average over the pairwise distances.</p><p>To calculate the distance between two clusters, use
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Cluster import clusterdistance
&gt;&gt;&gt; distance = clusterdistance(data)
</pre><p>where the following arguments are defined:
</p><ul class="itemize"><li class="li-itemize">
<code>data</code> (required)<br>
Array containing the data for the items.
</li><li class="li-itemize"><code>mask</code> (default: <code>None</code>) <br>
Array of integers showing which data are missing. If <code>mask[i,j]==0</code>, then <code>data[i,j]</code> is missing. If <code>mask==None</code>, then all data are present.
</li><li class="li-itemize"><code>weight</code> (default: <code>None</code>) <br>
The weights to be used when calculating distances. If <code>weight==None</code>, then equal weights are assumed.
</li><li class="li-itemize"><code>index1</code> (default: <code>0</code>) <br>
A list containing the indices of the items belonging to the first cluster. A cluster containing only one item <span style="font-style:italic">i</span> can be represented either as a list <code>[i]</code>, or as an integer <code>i</code>.
</li><li class="li-itemize"><code>index2</code> (default: <code>0</code>) <br>
A list containing the indices of the items belonging to the second cluster. A cluster containing only one items <span style="font-style:italic">i</span> can be represented either as a list <code>[i]</code>, or as an integer <code>i</code>.
</li><li class="li-itemize"><code>method</code> (default: <code>'a'</code>) <br>
Specifies how the distance between clusters is defined:
<ul class="itemize"><li class="li-itemize">
<code>'a'</code>: Distance between the two cluster centroids (arithmetic mean);
</li><li class="li-itemize"><code>'m'</code>: Distance between the two cluster centroids (median);
</li><li class="li-itemize"><code>'s'</code>: Shortest pairwise distance between items in the two clusters;
</li><li class="li-itemize"><code>'x'</code>: Longest pairwise distance between items in the two clusters;
</li><li class="li-itemize"><code>'v'</code>: Average over the pairwise distances between items in the two clusters.
</li></ul>
</li><li class="li-itemize"><code>dist</code> (default: <code>'e'</code>, Euclidean distance) <br>
Defines the distance function to be used (see <a href="#sec%3Adistancefunctions">15.1</a>).
</li><li class="li-itemize"><code>transpose</code> (default: <code>0</code>) <br>
If <code>transpose==0</code>, calculate the distance between the rows of <code>data</code>. If <code>transpose==1</code>, calculate the distance between the columns of <code>data</code>.
</li></ul>
<!--TOC section id="sec289" Partitioning algorithms-->
<h2 id="sec289" class="section">15.3&#XA0;&#XA0;Partitioning algorithms</h2><!--SEC END --><p>Partitioning algorithms divide items into <span style="font-style:italic">k</span> clusters such that the sum of distances over the items to their cluster centers is minimal.
The number of clusters <span style="font-style:italic">k</span> is specified by the user.
Three partitioning algorithms are available in <code>Bio.Cluster</code>:
</p><ul class="itemize"><li class="li-itemize">
<span style="font-style:italic">k</span>-means clustering
</li><li class="li-itemize"><span style="font-style:italic">k</span>-medians clustering
</li><li class="li-itemize"><span style="font-style:italic">k</span>-medoids clustering
</li></ul><p>
These algorithms differ in how the cluster center is defined. In <span style="font-style:italic">k</span>-means clustering, the cluster center is defined as the mean data vector averaged over all items in the cluster. Instead of the mean, in <span style="font-style:italic">k</span>-medians clustering the median is calculated for each dimension in the data vector. Finally, in <span style="font-style:italic">k</span>-medoids clustering the cluster center is defined as the item which has the smallest sum of distances to the other items in the cluster. This clustering algorithm is suitable for cases in which the distance matrix is known but the original data matrix is not available, for example when clustering proteins based on their structural similarity.</p><p>The expectation-maximization (EM) algorithm is used to find this partitioning into <span style="font-style:italic">k</span> groups.
In the initialization of the EM algorithm, we randomly assign items to clusters. To ensure that no empty clusters are produced, we use the binomial distribution to randomly choose the number of items in each cluster to be one or more. We then randomly permute the cluster assignments to items such that each item has an equal probability to be in any cluster. Each cluster is thus guaranteed to contain at least one item.</p><p>We then iterate:
</p><ul class="itemize"><li class="li-itemize">
Calculate the centroid of each cluster, defined as either the mean, the median, or the medoid of the cluster;
</li><li class="li-itemize">Calculate the distances of each item to the cluster centers;
</li><li class="li-itemize">For each item, determine which cluster centroid is closest;
</li><li class="li-itemize">Reassign each item to its closest cluster, or stop the iteration if no further item reassignments take place.
</li></ul><p>To avoid clusters becoming empty during the iteration, in <span style="font-style:italic">k</span>-means and <span style="font-style:italic">k</span>-medians clustering the algorithm keeps track of the number of items in each cluster, and prohibits the last remaining item in a cluster from being reassigned to a different cluster. For <span style="font-style:italic">k</span>-medoids clustering, such a check is not needed, as the item that functions as the cluster centroid has a zero distance to itself, and will therefore never be closer to a different cluster.</p><p>As the initial assignment of items to clusters is done randomly, usually a different clustering solution is found each time the EM algorithm is executed.
To find the optimal clustering solution, the <span style="font-style:italic">k</span>-means algorithm is repeated many times, each time starting from a different initial random clustering. The sum of distances of the items to their cluster center is saved for each run, and the solution with the smallest value of this sum will be returned as the overall clustering solution.</p><p>How often the EM algorithm should be run depends on the number of items being clustered. As a rule of thumb, we can consider how often the optimal solution was found; this number is returned by the partitioning algorithms as implemented in this library. If the optimal solution was found many times, it is unlikely that better solutions exist than the one that was found. However, if the optimal solution was found only once, there may well be other solutions with a smaller within-cluster sum of distances. If the number of items is large (more than several hundreds), it may be difficult to find the globally optimal solution.</p><p>The EM algorithm terminates when no further reassignments take place. We noticed that for some sets of initial cluster assignments, the EM algorithm fails to converge due to the same clustering solution reappearing periodically after a small number of iteration steps. We therefore check for the occurrence of such periodic solutions during the iteration. After a given number of iteration steps, the current clustering result is saved as a reference. By comparing the clustering result after each subsequent iteration step to the reference state, we can determine if a previously encountered clustering result is found. In such a case, the iteration is halted. If after a given number of iterations the reference state has not yet been encountered, the current clustering solution is saved to be used as the new reference state. Initially, ten iteration steps are executed before resaving the reference state. This number of iteration steps is doubled each time, to ensure that periodic behavior with longer periods can also be detected.</p><!--TOC subsection id="sec290" <span style="font-style:italic">k</span>-means and <span style="font-style:italic">k</span>-medians-->
<h3 id="sec290" class="subsection"><span style="font-style:italic">k</span>-means and <span style="font-style:italic">k</span>-medians</h3><!--SEC END --><p>The <span style="font-style:italic">k</span>-means and <span style="font-style:italic">k</span>-medians algorithms are implemented as the function <code>kcluster</code> in <code>Bio.Cluster</code>:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Cluster import kcluster
&gt;&gt;&gt; clusterid, error, nfound = kcluster(data)
</pre><p>where the following arguments are defined:
</p><ul class="itemize"><li class="li-itemize">
<code>data</code> (required)<br>
Array containing the data for the items.
</li><li class="li-itemize"><code>nclusters</code> (default: <code>2</code>) <br>
The number of clusters <span style="font-style:italic">k</span>.
</li><li class="li-itemize"><code>mask</code> (default: <code>None</code>) <br>
Array of integers showing which data are missing. If <code>mask[i,j]==0</code>, then <code>data[i,j]</code> is missing. If <code>mask==None</code>, then all data are present.
</li><li class="li-itemize"><code>weight</code> (default: <code>None</code>) <br>
The weights to be used when calculating distances. If <code>weight==None</code>, then equal weights are assumed.
</li><li class="li-itemize"><code>transpose</code> (default: <code>0</code>) <br>
Determines if rows (<code>transpose</code> is <code>0</code>) or columns (<code>transpose</code> is <code>1</code>) are to be clustered.
</li><li class="li-itemize"><code>npass</code> (default: <code>1</code>) <br>
The number of times the <span style="font-style:italic">k</span>-means/-medians clustering algorithm is performed, each time with a different (random) initial condition. If <code>initialid</code> is given, the value of <code>npass</code> is ignored and the clustering algorithm is run only once, as it behaves deterministically in that case.
</li><li class="li-itemize"><code>method</code> (default: <code>a</code>) <br>
describes how the center of a cluster is found:
<ul class="itemize"><li class="li-itemize">
<code>method=='a'</code>: arithmetic mean (<span style="font-style:italic">k</span>-means clustering);
</li><li class="li-itemize"><code>method=='m'</code>: median (<span style="font-style:italic">k</span>-medians clustering).
</li></ul>
For other values of <code>method</code>, the arithmetic mean is used.
</li><li class="li-itemize"><code>dist</code> (default: <code>'e'</code>, Euclidean distance) <br>
Defines the distance function to be used (see <a href="#sec%3Adistancefunctions">15.1</a>).
Whereas all eight distance measures are accepted by <code>kcluster</code>, from a theoretical viewpoint it is best to use the Euclidean distance for the <span style="font-style:italic">k</span>-means algorithm, and the city-block distance for <span style="font-style:italic">k</span>-medians.
</li><li class="li-itemize"><code>initialid</code> (default: <code>None</code>) <br>
Specifies the initial clustering to be used for the EM algorithm. If <code>initialid==None</code>, then a different random initial clustering is used for each of the <code>npass</code> runs of the EM algorithm. If <code>initialid</code> is not <code>None</code>, then it should be equal to a 1D array containing the cluster number (between <code>0</code> and <code>nclusters-1</code>) for each item. Each cluster should contain at least one item. With the initial clustering specified, the EM algorithm is deterministic.
</li></ul><p>This function returns a tuple <code>(clusterid, error, nfound)</code>, where <code>clusterid</code> is an integer array containing the number of the cluster to which each row or cluster was assigned, <code>error</code> is the within-cluster sum of distances for the optimal clustering solution, and <code>nfound</code> is the number of times this optimal solution was found.</p><!--TOC subsection id="sec291" <span style="font-style:italic">k</span>-medoids clustering-->
<h3 id="sec291" class="subsection"><span style="font-style:italic">k</span>-medoids clustering</h3><!--SEC END --><p>The <code>kmedoids</code> routine performs <span style="font-style:italic">k</span>-medoids clustering on a given set of items, using the distance matrix and the number of clusters passed by the user:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Cluster import kmedoids
&gt;&gt;&gt; clusterid, error, nfound = kmedoids(distance)
</pre><p>where the following arguments are defined:
, nclusters=2, npass=1, initialid=None)|</p><ul class="itemize"><li class="li-itemize">
<code>distance</code> (required) <br>
The matrix containing the distances between the items; this matrix can be specified in three ways:
<ul class="itemize"><li class="li-itemize">
as a 2D Numerical Python array (in which only the left-lower part of the array will be accessed):
<pre class="verbatim">distance = array([[0.0, 1.1, 2.3],
                  [1.1, 0.0, 4.5],
                  [2.3, 4.5, 0.0]])
</pre></li><li class="li-itemize">as a 1D Numerical Python array containing consecutively the distances in the left-lower part of the distance matrix:
<pre class="verbatim">distance = array([1.1, 2.3, 4.5])
</pre></li><li class="li-itemize">as a list containing the rows of the left-lower part of the distance matrix:
<pre class="verbatim">distance = [array([]|,
            array([1.1]),
            array([2.3, 4.5])
           ]
</pre></li></ul>
These three expressions correspond to the same distance matrix.
</li><li class="li-itemize"><code>nclusters</code> (default: <code>2</code>) <br>
The number of clusters <span style="font-style:italic">k</span>.
</li><li class="li-itemize"><code>npass</code> (default: <code>1</code>) <br>
The number of times the <span style="font-style:italic">k</span>-medoids clustering algorithm is performed, each time with a different (random) initial condition. If <code>initialid</code> is given, the value of <code>npass</code> is ignored, as the clustering algorithm behaves deterministically in that case.
</li><li class="li-itemize"><code>initialid</code> (default: <code>None</code>) <br>
Specifies the initial clustering to be used for the EM algorithm. If <code>initialid==None</code>, then a different random initial clustering is used for each of the <code>npass</code> runs of the EM algorithm. If <code>initialid</code> is not <code>None</code>, then it should be equal to a 1D array containing the cluster number (between <code>0</code> and <code>nclusters-1</code>) for each item. Each cluster should contain at least one item. With the initial clustering specified, the EM algorithm is deterministic.
</li></ul><p>This function returns a tuple <code>(clusterid, error, nfound)</code>, where <code>clusterid</code> is an array containing the number of the cluster to which each item was assigned, <code>error</code> is the within-cluster sum of distances for the optimal <span style="font-style:italic">k</span>-medoids clustering solution, and <code>nfound</code> is the number of times the optimal solution was found. Note that the cluster number in <code>clusterid</code> is defined as the item number of the item representing the cluster centroid.</p>
<!--TOC section id="sec292" Hierarchical clustering-->
<h2 id="sec292" class="section">15.4&#XA0;&#XA0;Hierarchical clustering</h2><!--SEC END --><p>Hierarchical clustering methods are inherently different from the <span style="font-style:italic">k</span>-means clustering method. In hierarchical clustering, the similarity in the expression profile between genes or experimental conditions are represented in the form of a tree structure. This tree structure can be shown graphically by programs such as Treeview and Java Treeview, which has contributed to the popularity of hierarchical clustering in the analysis of gene expression data.</p><p>The first step in hierarchical clustering is to calculate the distance matrix, specifying all the distances between the items to be clustered. Next, we create a node by joining the two closest items. Subsequent nodes are created by pairwise joining of items or nodes based on the distance between them, until all items belong to the same node. A tree structure can then be created by retracing which items and nodes were merged. Unlike the EM algorithm, which is used in <span style="font-style:italic">k</span>-means clustering, the complete process of hierarchical clustering is deterministic.</p><p>Several flavors of hierarchical clustering exist, which differ in how the distance between subnodes is defined in terms of their members. In <code>Bio.Cluster</code>, pairwise single, maximum, average, and centroid linkage are available.</p><ul class="itemize"><li class="li-itemize">
In pairwise single-linkage clustering, the distance between two nodes is defined as the shortest distance among the pairwise distances between the members of the two nodes.
</li><li class="li-itemize">In pairwise maximum-linkage clustering, alternatively known as pairwise complete-linkage clustering, the distance between two nodes is defined as the longest distance among the pairwise distances between the members of the two nodes.
</li><li class="li-itemize">In pairwise average-linkage clustering, the distance between two nodes is defined as the average over all pairwise distances between the items of the two nodes.
</li><li class="li-itemize">In pairwise centroid-linkage clustering, the distance between two nodes is defined as the distance between their centroids. The centroids are calculated by taking the mean over all the items in a cluster. As the distance from each newly formed node to existing nodes and items need to be calculated at each step, the computing time of pairwise centroid-linkage clustering may be significantly longer than for the other hierarchical clustering methods. Another peculiarity is that (for a distance measure based on the Pearson correlation), the distances do not necessarily increase when going up in the clustering tree, and may even decrease. This is caused by an inconsistency between the centroid calculation and the distance calculation when using the Pearson correlation: Whereas the Pearson correlation effectively normalizes the data for the distance calculation, no such normalization occurs for the centroid calculation.
</li></ul><p>For pairwise single-, complete-, and average-linkage clustering, the distance between two nodes can be found directly from the distances between the individual items. Therefore, the clustering algorithm does not need access to the original gene expression data, once the distance matrix is known. For pairwise centroid-linkage clustering, however, the centroids of newly formed subnodes can only be calculated from the original data and not from the distance matrix.</p><p>The implementation of pairwise single-linkage hierarchical clustering is based on the SLINK algorithm (R. Sibson, 1973), which is much faster and more memory-efficient than a straightforward implementation of pairwise single-linkage clustering. The clustering result produced by this algorithm is identical to the clustering solution found by the conventional single-linkage algorithm. The single-linkage hierarchical clustering algorithm implemented in this library can be used to cluster large gene expression data sets, for which conventional hierarchical clustering algorithms fail due to excessive memory requirements and running time.</p><!--TOC subsection id="sec293" Representing a hierarchical clustering solution-->
<h3 id="sec293" class="subsection">Representing a hierarchical clustering solution</h3><!--SEC END --><p>The result of hierarchical clustering consists of a tree of nodes, in which each node joins two items or subnodes. Usually, we are not only interested in which items or subnodes are joined at each node, but also in their similarity (or distance) as they are joined. To store one node in the hierarchical clustering tree, we make use of the class <code>Node</code>, which defined in <code>Bio.Cluster</code>. An instance of <code>Node</code> has three attributes:
</p><ul class="itemize"><li class="li-itemize">
<code>left</code>
</li><li class="li-itemize"><code>right</code>
</li><li class="li-itemize"><code>distance</code>
</li></ul><p>
Here, <code>left</code> and <code>right</code> are integers referring to the two items or subnodes that are joined at this node, and <code>distance</code> is the distance between them. The items being clustered are numbered from 0 to (number of items &#X2212; 1), while clusters are numbered from -1 to &#X2212;(number of items&#X2212;1). Note that the number of nodes is one less than the number of items.</p><p>To create a new <code>Node</code> object, we need to specify <code>left</code> and <code>right</code>; <code>distance</code> is optional.</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Cluster import Node
&gt;&gt;&gt; Node(2, 3)
(2, 3): 0
&gt;&gt;&gt; Node(2, 3, 0.91)
(2, 3): 0.91
</pre><p>The attributes <code>left</code>, <code>right</code>, and <code>distance</code> of an existing <code>Node</code> object can be modified directly:</p><pre class="verbatim">&gt;&gt;&gt; node = Node(4, 5)
&gt;&gt;&gt; node.left = 6
&gt;&gt;&gt; node.right = 2
&gt;&gt;&gt; node.distance = 0.73
&gt;&gt;&gt; node
(6, 2): 0.73
</pre><p>An error is raised if <code>left</code> and <code>right</code> are not integers, or if <code>distance</code> cannot be converted to a floating-point value.</p><p>The Python class <code>Tree</code> represents a full hierarchical clustering solution. A <code>Tree</code> object can be created from a list of <code>Node</code> objects:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Cluster import Node, Tree
&gt;&gt;&gt; nodes = [Node(1, 2, 0.2), Node(0, 3, 0.5), Node(-2, 4, 0.6), Node(-1, -3, 0.9)]
&gt;&gt;&gt; tree = Tree(nodes)
&gt;&gt;&gt; print(tree)
(1, 2): 0.2
(0, 3): 0.5
(-2, 4): 0.6
(-1, -3): 0.9
</pre><p>The <code>Tree</code> initializer checks if the list of nodes is a valid hierarchical clustering result:</p><pre class="verbatim">&gt;&gt;&gt; nodes = [Node(1, 2, 0.2), Node(0, 2, 0.5)]
&gt;&gt;&gt; Tree(nodes)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in ?
ValueError: Inconsistent tree
</pre><p>Individual nodes in a <code>Tree</code> object can be accessed using square brackets:</p><pre class="verbatim">&gt;&gt;&gt; nodes = [Node(1, 2, 0.2), Node(0, -1, 0.5)]
&gt;&gt;&gt; tree = Tree(nodes)
&gt;&gt;&gt; tree[0]
(1, 2): 0.2
&gt;&gt;&gt; tree[1]
(0, -1): 0.5
&gt;&gt;&gt; tree[-1]
(0, -1): 0.5
</pre><p>As a <code>Tree</code> object is read-only, we cannot change individual nodes in a <code>Tree</code> object. However, we can convert the tree to a list of nodes, modify this list, and create a new tree from this list:</p><pre class="verbatim">&gt;&gt;&gt; tree = Tree([Node(1, 2, 0.1), Node(0, -1, 0.5), Node(-2, 3, 0.9)])
&gt;&gt;&gt; print(tree)
(1, 2): 0.1
(0, -1): 0.5
(-2, 3): 0.9
&gt;&gt;&gt; nodes = tree[:]
&gt;&gt;&gt; nodes[0] = Node(0, 1, 0.2)
&gt;&gt;&gt; nodes[1].left = 2
&gt;&gt;&gt; tree = Tree(nodes)
&gt;&gt;&gt; print(tree)
(0, 1): 0.2
(2, -1): 0.5
(-2, 3): 0.9
</pre><p>This guarantees that any <code>Tree</code> object is always well-formed. </p><p>To display a hierarchical clustering solution with visualization programs such as Java Treeview, it is better to scale all node distances such that they are between zero and one. This can be accomplished by calling the <code>scale</code> method on an existing <code>Tree</code> object:
</p><pre class="verbatim">&gt;&gt;&gt; tree.scale()
</pre><p>This method takes no arguments, and returns <code>None</code>.</p><p>After hierarchical clustering, the items can be grouped into <span style="font-style:italic">k</span> clusters based on the tree structure stored in the <code>Tree</code> object by cutting the tree:
</p><pre class="verbatim">&gt;&gt;&gt; clusterid = tree.cut(nclusters=1)
</pre><p>where <code>nclusters</code> (defaulting to <code>1</code>) is the desired number of clusters <span style="font-style:italic">k</span>.
This method ignores the top <span style="font-style:italic">k</span>&#X2212;1 linking events in the tree structure, resulting in <span style="font-style:italic">k</span> separated clusters of items. The number of clusters <span style="font-style:italic">k</span> should be positive, and less than or equal to the number of items.
This method returns an array <code>clusterid</code> containing the number of the cluster to which each item is assigned.</p><!--TOC subsection id="sec294" Performing hierarchical clustering-->
<h3 id="sec294" class="subsection">Performing hierarchical clustering</h3><!--SEC END --><p>To perform hierarchical clustering, use the <code>treecluster</code> function in <code>Bio.Cluster</code>.
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Cluster import treecluster
&gt;&gt;&gt; tree = treecluster(data)
</pre><p>where the following arguments are defined:</p><ul class="itemize"><li class="li-itemize">
<code>data</code> <br>
Array containing the data for the items.
</li><li class="li-itemize"><code>mask</code> (default: <code>None</code>) <br>
Array of integers showing which data are missing. If <code>mask[i,j]==0</code>, then <code>data[i,j]</code> is missing. If <code>mask==None</code>, then all data are present.
</li><li class="li-itemize"><code>weight</code> (default: <code>None</code>) <br>
The weights to be used when calculating distances. If <code>weight==None</code>, then equal weights are assumed.
</li><li class="li-itemize"><code>transpose</code> (default: <code>0</code>) <br>
Determines if rows (<code>transpose==0</code>) or columns (<code>transpose==1</code>) are to be clustered.
</li><li class="li-itemize"><code>method</code> (default: <code>'m'</code>) <br>
defines the linkage method to be used:
<ul class="itemize"><li class="li-itemize">
<code>method=='s'</code>: pairwise single-linkage clustering
</li><li class="li-itemize"><code>method=='m'</code>: pairwise maximum- (or complete-) linkage clustering
</li><li class="li-itemize"><code>method=='c'</code>: pairwise centroid-linkage clustering
</li><li class="li-itemize"><code>method=='a'</code>: pairwise average-linkage clustering
</li></ul>
</li><li class="li-itemize"><code>dist</code> (default: <code>'e'</code>, Euclidean distance) <br>
Defines the distance function to be used (see <a href="#sec%3Adistancefunctions">15.1</a>).
</li></ul><p>To apply hierarchical clustering on a precalculated distance matrix, specify the <code>distancematrix</code> argument when calling <code>treecluster</code> function instead of the <code>data</code> argument:
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Cluster import treecluster
&gt;&gt;&gt; tree = treecluster(distancematrix=distance)
</pre><p>In this case, the following arguments are defined:
</p><ul class="itemize"><li class="li-itemize">
<code>distancematrix</code> <br>
The distance matrix, which can be specified in three ways:
<ul class="itemize"><li class="li-itemize">
as a 2D Numerical Python array (in which only the left-lower part of the array will be accessed):
<pre class="verbatim">distance = array([[0.0, 1.1, 2.3], 
                  [1.1, 0.0, 4.5],
                  [2.3, 4.5, 0.0]])
</pre></li><li class="li-itemize">as a 1D Numerical Python array containing consecutively the distances in the left-lower part of the distance matrix:
<pre class="verbatim">distance = array([1.1, 2.3, 4.5])
</pre></li><li class="li-itemize">as a list containing the rows of the left-lower part of the distance matrix:
<pre class="verbatim">distance = [array([]),
            array([1.1]),
            array([2.3, 4.5])
</pre></li></ul>
These three expressions correspond to the same distance matrix.
As <code>treecluster</code> may shuffle the values in the distance matrix as part of the clustering algorithm, be sure to save this array in a different variable before calling <code>treecluster</code> if you need it later.
</li><li class="li-itemize"><code>method</code> <br>
The linkage method to be used:
<ul class="itemize"><li class="li-itemize">
<code>method=='s'</code>: pairwise single-linkage clustering
</li><li class="li-itemize"><code>method=='m'</code>: pairwise maximum- (or complete-) linkage clustering
</li><li class="li-itemize"><code>method=='a'</code>: pairwise average-linkage clustering
</li></ul>
While pairwise single-, maximum-, and average-linkage clustering can be calculated from the distance matrix alone, pairwise centroid-linkage cannot.
</li></ul><p>When calling <code>treecluster</code>, either <code>data</code> or <code>distancematrix</code> should be <code>None</code>.</p><p>This function returns a <code>Tree</code> object. This object contains (number of items &#X2212; 1) nodes, where the number of items is the number of rows if rows were clustered, or the number of columns if columns were clustered. Each node describes a pairwise linking event, where the node attributes <code>left</code> and <code>right</code> each contain the number of one item or subnode, and <code>distance</code> the distance between them. Items are numbered from 0 to (number of items &#X2212; 1), while clusters are numbered -1 to &#X2212;(number of items&#X2212;1).</p>
<!--TOC section id="sec295" Self-Organizing Maps-->
<h2 id="sec295" class="section">15.5&#XA0;&#XA0;Self-Organizing Maps</h2><!--SEC END --><p>Self-Organizing Maps (SOMs) were invented by Kohonen to describe neural networks (see for instance Kohonen, 1997 [<a href="#kohonen1997">24</a>]). Tamayo (1999) first applied Self-Organizing Maps to gene expression data [<a href="#tamayo1999">30</a>].</p><p>SOMs organize items into clusters that are situated in some topology. Usually a rectangular topology is chosen. The clusters generated by SOMs are such that neighboring clusters in the topology are more similar to each other than clusters far from each other in the topology.</p><p>The first step to calculate a SOM is to randomly assign a data vector to each cluster in the topology. If rows are being clustered, then the number of elements in each data vector is equal to the number of columns.</p><p>An SOM is then generated by taking rows one at a time, and finding which cluster in the topology has the closest data vector. The data vector of that cluster, as well as those of the neighboring clusters, are adjusted using the data vector of the row under consideration. The adjustment is given by
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell">&#X394;&#XA0;</td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">x</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell"><sub>cell</sub>&#XA0;=&#XA0;&#X3C4;&#XA0;&#XB7;&#XA0;</td><td class="dcell">&#X239B;<br>
&#X239C;<br>
&#X239D;</td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">x</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell"><sub>row</sub>&#XA0;&#X2212;&#XA0;</td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">x</span></td></tr>
<tr><td class="hbar"></td></tr>
</table></td><td class="dcell"><sub>cell</sub>&#XA0;</td><td class="dcell">&#X239E;<br>
&#X239F;<br>
&#X23A0;</td><td class="dcell">.</td></tr>
</table><p>
The parameter
&#X3C4;
is a parameter that decreases at each iteration step. We have used a simple linear function of the iteration step:
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell">&#X3C4;&#XA0;=&#XA0;&#X3C4;<sub>init</sub>&#XA0;&#XB7;&#XA0;</td><td class="dcell">&#X239B;<br>
&#X239C;<br>
&#X239C;<br>
&#X239D;</td><td class="dcell">1&#XA0;&#X2212;&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">i</span>&#XA0;</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
</table></td><td class="dcell">&#X239E;<br>
&#X239F;<br>
&#X239F;<br>
&#X23A0;</td><td class="dcell">,</td></tr>
</table><p>
&#X3C4;<sub>init</sub>
is the initial value of &#X3C4; as specified by the user, <span style="font-style:italic">i</span> is the number of the current iteration step, and <span style="font-style:italic">n</span> is the total number of iteration steps to be performed. While changes are made rapidly in the beginning of the iteration, at the end of iteration only small changes are made.</p><p>All clusters within a radius <span style="font-style:italic">R</span> are adjusted to the gene under consideration. This radius decreases as the calculation progresses as
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">R</span>&#XA0;=&#XA0;<span style="font-style:italic">R</span><sub>max</sub>&#XA0;&#XB7;&#XA0;</td><td class="dcell">&#X239B;<br>
&#X239C;<br>
&#X239C;<br>
&#X239D;</td><td class="dcell">1&#XA0;&#X2212;&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">i</span>&#XA0;</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">n</span></td></tr>
</table></td><td class="dcell">&#X239E;<br>
&#X239F;<br>
&#X239F;<br>
&#X23A0;</td><td class="dcell">,</td></tr>
</table><p>
in which the maximum radius is defined as
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">R</span><sub>max</sub>&#XA0;=&#XA0;</td><td class="dcell"><span style="font-size:x-large">&#X221A;</span></td><td class="dcell"><table style="border:0;border-spacing:1;border-collapse:separate;" class="cellpadding0"><tr><td class="hbar"></td></tr>
<tr><td style="text-align:center;white-space:nowrap" ><span style="font-style:italic">N</span><sub><span style="font-style:italic">x</span></sub><sup>2</sup>&#XA0;+&#XA0;<span style="font-style:italic">N</span><sub><span style="font-style:italic">y</span></sub><sup>2</sup></td></tr>
</table></td><td class="dcell">,</td></tr>
</table><p>
where
(<span style="font-style:italic">N</span><sub><span style="font-style:italic">x</span></sub>, <span style="font-style:italic">N</span><sub><span style="font-style:italic">y</span></sub>)
are the dimensions of the rectangle defining the topology.</p><p>The function <code>somcluster</code> implements the complete algorithm to calculate a Self-Organizing Map on a rectangular grid. First it initializes the random number generator. The node data are then initialized using the random number generator. The order in which genes or microarrays are used to modify the SOM is also randomized. The total number of iterations in the SOM algorithm is specified by the user.</p><p>To run <code>somcluster</code>, use
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Cluster import somcluster
&gt;&gt;&gt; clusterid, celldata = somcluster(data)
</pre><p>where the following arguments are defined:
</p><ul class="itemize"><li class="li-itemize">
<code>data</code> (required) <br>
Array containing the data for the items.
</li><li class="li-itemize"><code>mask</code> (default: <code>None</code>) <br>
Array of integers showing which data are missing. If <code>mask[i,j]==0</code>, then <code>data[i,j]</code> is missing. If <code>mask==None</code>, then all data are present.
</li><li class="li-itemize"><code>weight</code> (default: <code>None</code>) <br>
contains the weights to be used when calculating distances. If <code>weight==None</code>, then equal weights are assumed.
</li><li class="li-itemize"><code>transpose</code> (default: <code>0</code>) <br>
Determines if rows (<code>transpose</code> is <code>0</code>) or columns (<code>transpose</code> is <code>1</code>) are to be clustered.
</li><li class="li-itemize"><code>nxgrid, nygrid</code> (default: <code>2, 1</code>) <br>
The number of cells horizontally and vertically in the rectangular grid on which the Self-Organizing Map is calculated.
</li><li class="li-itemize"><code>inittau</code> (default: <code>0.02</code>) <br>
The initial value for the parameter &#X3C4; that is used in the SOM algorithm. The default value for <code>inittau</code> is 0.02, which was used in Michael Eisen&#X2019;s Cluster/TreeView program.
</li><li class="li-itemize"><code>niter</code> (default: <code>1</code>) <br>
The number of iterations to be performed.
</li><li class="li-itemize"><code>dist</code> (default: <code>'e'</code>, Euclidean distance) <br>
Defines the distance function to be used (see <a href="#sec%3Adistancefunctions">15.1</a>).
</li></ul><p>This function returns the tuple <code>(clusterid, celldata)</code>:
</p><ul class="itemize"><li class="li-itemize">
<code>clusterid</code>: <br>
An array with two columns, where the number of rows is equal to the number of items that were clustered. Each row contains the <span style="font-style:italic">x</span> and <span style="font-style:italic">y</span> coordinates of the cell in the rectangular SOM grid to which the item was assigned.
</li><li class="li-itemize"><code>celldata</code>: <br>
An array with dimensions (<code>nxgrid</code>, <code>nygrid</code>, number of columns) if rows are being clustered, or (<code>nxgrid</code>, <code>nygrid</code>, number of rows) if columns are being clustered. Each element <code>[ix][iy]</code> of this array is a 1D vector containing the gene expression data for the centroid of the cluster in the grid cell with coordinates <code>[ix][iy]</code>.
</li></ul>
<!--TOC section id="sec296" Principal Component Analysis-->
<h2 id="sec296" class="section">15.6&#XA0;&#XA0;Principal Component Analysis</h2><!--SEC END --><p>Principal Component Analysis (PCA) is a widely used technique for analyzing multivariate data. A practical example of applying Principal Component Analysis to gene expression data is presented by Yeung and Ruzzo (2001) [<a href="#yeung2001">33</a>].</p><p>In essence, PCA is a coordinate transformation in which each row in the data matrix is written as a linear sum over basis vectors called principal components, which are ordered and chosen such that each maximally explains the remaining variance in the data vectors. For example, an <span style="font-style:italic">n</span> &#XD7; 3 data matrix can be represented as an ellipsoidal cloud of <span style="font-style:italic">n</span> points in three dimensional space. The first principal component is the longest axis of the ellipsoid, the second principal component the second longest axis of the ellipsoid, and the third principal component is the shortest axis. Each row in the data matrix can be reconstructed as a suitable linear combination of the principal components. However, in order to reduce the dimensionality of the data, usually only the most important principal components are retained. The remaining variance present in the data is then regarded as unexplained variance.</p><p>The principal components can be found by calculating the eigenvectors of the covariance matrix of the data. The corresponding eigenvalues determine how much of the variance present in the data is explained by each principal component.</p><p>Before applying principal component analysis, typically the mean is subtracted from each column in the data matrix. In the example above, this effectively centers the ellipsoidal cloud around its centroid in 3D space, with the principal components describing the variation of points in the ellipsoidal cloud with respect to their centroid.</p><p>The function <code>pca</code> below first uses the singular value decomposition to calculate the eigenvalues and eigenvectors of the data matrix. The singular value decomposition is implemented as a translation in C of the Algol procedure <code>svd</code> [<a href="#golub1971">16</a>], which uses Householder bidiagonalization and a variant of the QR algorithm. The principal components, the coordinates of each data vector along the principal components, and the eigenvalues corresponding to the principal components are then evaluated and returned in decreasing order of the magnitude of the eigenvalue. If data centering is desired, the mean should be subtracted from each column in the data matrix before calling the <code>pca</code> routine.</p><p>To apply Principal Component Analysis to a rectangular matrix <code>data</code>, use
</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Cluster import pca
&gt;&gt;&gt; columnmean, coordinates, components, eigenvalues = pca(data)
</pre><p>This function returns a tuple <code>columnmean, coordinates, components, eigenvalues</code>:
</p><ul class="itemize"><li class="li-itemize">
<code>columnmean</code> <br>
Array containing the mean over each column in <code>data</code>.
</li><li class="li-itemize"><code>coordinates</code> <br>
The coordinates of each row in <code>data</code> with respect to the principal components.
</li><li class="li-itemize"><code>components</code> <br>
The principal components.
</li><li class="li-itemize"><code>eigenvalues</code> <br>
The eigenvalues corresponding to each of the principal components.
</li></ul><p>
The original matrix <code>data</code> can be recreated by calculating <code>columnmean +  dot(coordinates, components)</code>.</p>
<!--TOC section id="sec297" Handling Cluster/TreeView-type files-->
<h2 id="sec297" class="section">15.7&#XA0;&#XA0;Handling Cluster/TreeView-type files</h2><!--SEC END --><p>Cluster/TreeView are GUI-based codes for clustering gene expression data. They were originally written by <a href="http://rana.lbl.gov">Michael Eisen</a> while at Stanford University. <code>Bio.Cluster</code> contains functions for reading and writing data files that correspond to the format specified for Cluster/TreeView. In particular, by saving a clustering result in that format, TreeView can be used to visualize the clustering results. We recommend using Alok Saldanha&#X2019;s <a href="http://jtreeview.sourceforge.net/"><span style="font-family:monospace">http://jtreeview.sourceforge.net/</span></a>Java TreeView program, which can display hierarchical as well as <span style="font-style:italic">k</span>-means clustering results.</p><p>An object of the class <code>Record</code> contains all information stored in a Cluster/TreeView-type data file. To store the information contained in the data file in a <code>Record</code> object, we first open the file and then read it:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Cluster
&gt;&gt;&gt; handle = open("mydatafile.txt")
&gt;&gt;&gt; record = Cluster.read(handle)
&gt;&gt;&gt; handle.close()
</pre><p>This two-step process gives you some flexibility in the source of the data.
For example, you can use</p><pre class="verbatim">&gt;&gt;&gt; import gzip # Python standard library
&gt;&gt;&gt; handle = gzip.open("mydatafile.txt.gz")
</pre><p>to open a gzipped file, or
</p><pre class="verbatim">&gt;&gt;&gt; import urllib # Python standard library
&gt;&gt;&gt; handle = urllib.urlopen("http://somewhere.org/mydatafile.txt")
</pre><p>to open a file stored on the Internet before calling <code>read</code>.</p><p>The <code>read</code> command reads the tab-delimited text file <code>mydatafile.txt</code> containing gene expression data in the format specified for Michael Eisen&#X2019;s Cluster/TreeView program. For a description of this file format, see the manual to Cluster/TreeView. It is available at <a href="http://rana.lbl.gov/manuals/ClusterTreeView.pdf">Michael Eisen&#X2019;s lab website</a> and at <a href="http://bonsai.ims.u-tokyo.ac.jp/~mdehoon/software/cluster/cluster3.pdf">our website</a>.</p><p>A <code>Record</code> object has the following attributes:</p><ul class="itemize"><li class="li-itemize">
<code>data</code> <br>
The data array containing the gene expression data. Genes are stored row-wise, while microarrays are stored column-wise.</li><li class="li-itemize"><code>mask</code> <br>
This array shows which elements in the <code>data</code> array, if any, are missing. If <code>mask[i,j]==0</code>, then <code>data[i,j]</code> is missing. If no data were found to be missing, <code>mask</code> is set to <code>None</code>.</li><li class="li-itemize"><code>geneid</code> <br>
This is a list containing a unique description for each gene (i.e., ORF numbers).</li><li class="li-itemize"><code>genename</code> <br>
This is a list containing a description for each gene (i.e., gene name). If not present in the data file, <code>genename</code> is set to <code>None</code>.</li><li class="li-itemize"><code>gweight</code> <br>
The weights that are to be used to calculate the distance in expression profile between genes. If not present in the data file, <code>gweight</code> is set to <code>None</code>.</li><li class="li-itemize"><code>gorder</code> <br>
The preferred order in which genes should be stored in an output file. If not present in the data file, <code>gorder</code> is set to <code>None</code>.</li><li class="li-itemize"><code>expid</code> <br>
This is a list containing a description of each microarray, e.g. experimental condition.</li><li class="li-itemize"><code>eweight</code> <br>
The weights that are to be used to calculate the distance in expression profile between microarrays. If not present in the data file, <code>eweight</code> is set to <code>None</code>.</li><li class="li-itemize"><code>eorder</code> <br>
The preferred order in which microarrays should be stored in an output file. If not present in the data file, <code>eorder</code> is set to <code>None</code>.</li><li class="li-itemize"><code>uniqid</code> <br>
The string that was used instead of UNIQID in the data file.
</li></ul><p>After loading a <code>Record</code> object, each of these attributes can be accessed and modified directly. For example, the data can be log-transformed by taking the logarithm of <code>record.data</code>.</p><!--TOC subsection id="sec298" Calculating the distance matrix-->
<h3 id="sec298" class="subsection">Calculating the distance matrix</h3><!--SEC END --><p>To calculate the distance matrix between the items stored in the record, use
</p><pre class="verbatim">&gt;&gt;&gt; matrix = record.distancematrix()
</pre><p>where the following arguments are defined:
</p><ul class="itemize"><li class="li-itemize">
<code>transpose</code> (default: <code>0</code>) <br>
Determines if the distances between the rows of <code>data</code> are to be calculated (<code>transpose==0</code>), or between the columns of <code>data</code> (<code>transpose==1</code>).
</li><li class="li-itemize"><code>dist</code> (default: <code>'e'</code>, Euclidean distance) <br>
Defines the distance function to be used (see <a href="#sec%3Adistancefunctions">15.1</a>).
</li></ul><p>This function returns the distance matrix as a list of rows, where the number of columns of each row is equal to the row number (see section <a href="#subsec%3Adistancematrix">15.1</a>).</p><!--TOC subsection id="sec299" Calculating the cluster centroids-->
<h3 id="sec299" class="subsection">Calculating the cluster centroids</h3><!--SEC END --><p>To calculate the centroids of clusters of items stored in the record, use
</p><pre class="verbatim">&gt;&gt;&gt; cdata, cmask = record.clustercentroids()
</pre><ul class="itemize"><li class="li-itemize">
<code>clusterid</code> (default: <code>None</code>) <br>
Vector of integers showing to which cluster each item belongs. If <code>clusterid</code> is not given, then all items are assumed to belong to the same cluster.
</li><li class="li-itemize"><code>method</code> (default: <code>'a'</code>) <br>
Specifies whether the arithmetic mean (<code>method=='a'</code>) or the median (<code>method=='m'</code>) is used to calculate the cluster center.
</li><li class="li-itemize"><code>transpose</code> (default: <code>0</code>) <br>
Determines if the centroids of the rows of <code>data</code> are to be calculated (<code>transpose==0</code>), or the centroids of the columns of <code>data</code> (<code>transpose==1</code>).
</li></ul><p>This function returns the tuple <code>cdata, cmask</code>; see section <a href="#subsec%3Aclustercentroids">15.2</a> for a description.</p><!--TOC subsection id="sec300" Calculating the distance between clusters-->
<h3 id="sec300" class="subsection">Calculating the distance between clusters</h3><!--SEC END --><p>
To calculate the distance between clusters of items stored in the record, use
</p><pre class="verbatim">&gt;&gt;&gt; distance = record.clusterdistance()
</pre><p>where the following arguments are defined:
</p><ul class="itemize"><li class="li-itemize">
<code>index1</code> (default: <code>0</code>) <br>
A list containing the indices of the items belonging to the first cluster. A cluster containing only one item <span style="font-style:italic">i</span> can be represented either as a list <code>[i]</code>, or as an integer <code>i</code>.
</li><li class="li-itemize"><code>index2</code> (default: <code>0</code>) <br>
A list containing the indices of the items belonging to the second cluster. A cluster containing only one item <span style="font-style:italic">i</span> can be represented either as a list <code>[i]</code>, or as an integer <code>i</code>.
</li><li class="li-itemize"><code>method</code> (default: <code>'a'</code>) <br>
Specifies how the distance between clusters is defined:
<ul class="itemize"><li class="li-itemize">
<code>'a'</code>: Distance between the two cluster centroids (arithmetic mean);
</li><li class="li-itemize"><code>'m'</code>: Distance between the two cluster centroids (median);
</li><li class="li-itemize"><code>'s'</code>: Shortest pairwise distance between items in the two clusters;
</li><li class="li-itemize"><code>'x'</code>: Longest pairwise distance between items in the two clusters;
</li><li class="li-itemize"><code>'v'</code>: Average over the pairwise distances between items in the two clusters.
</li></ul>
</li><li class="li-itemize"><code>dist</code> (default: <code>'e'</code>, Euclidean distance) <br>
Defines the distance function to be used (see <a href="#sec%3Adistancefunctions">15.1</a>).
</li><li class="li-itemize"><code>transpose</code> (default: <code>0</code>) <br>
If <code>transpose==0</code>, calculate the distance between the rows of <code>data</code>. If <code>transpose==1</code>, calculate the distance between the columns of <code>data</code>.
</li></ul><!--TOC subsection id="sec301" Performing hierarchical clustering-->
<h3 id="sec301" class="subsection">Performing hierarchical clustering</h3><!--SEC END --><p>To perform hierarchical clustering on the items stored in the record, use
</p><pre class="verbatim">&gt;&gt;&gt; tree = record.treecluster()
</pre><p>where the following arguments are defined:
</p><ul class="itemize"><li class="li-itemize">
<code>transpose</code> (default: <code>0</code>) <br>
Determines if rows (<code>transpose==0</code>) or columns (<code>transpose==1</code>) are to be clustered.
</li><li class="li-itemize"><code>method</code> (default: <code>'m'</code>) <br>
defines the linkage method to be used:
<ul class="itemize"><li class="li-itemize">
<code>method=='s'</code>: pairwise single-linkage clustering
</li><li class="li-itemize"><code>method=='m'</code>: pairwise maximum- (or complete-) linkage clustering
</li><li class="li-itemize"><code>method=='c'</code>: pairwise centroid-linkage clustering
</li><li class="li-itemize"><code>method=='a'</code>: pairwise average-linkage clustering
</li></ul>
</li><li class="li-itemize"><code>dist</code> (default: <code>'e'</code>, Euclidean distance) <br>
Defines the distance function to be used (see <a href="#sec%3Adistancefunctions">15.1</a>).
</li><li class="li-itemize"><code>transpose</code> <br>
Determines if genes or microarrays are being clustered. If <code>transpose==0</code>, genes (rows) are being clustered. If <code>transpose==1</code>, microarrays (columns) are clustered.
</li></ul><p>This function returns a <code>Tree</code> object. This object contains (number of items &#X2212; 1) nodes, where the number of items is the number of rows if rows were clustered, or the number of columns if columns were clustered. Each node describes a pairwise linking event, where the node attributes <code>left</code> and <code>right</code> each contain the number of one item or subnode, and <code>distance</code> the distance between them. Items are numbered from 0 to (number of items &#X2212; 1), while clusters are numbered -1 to &#X2212;(number of items&#X2212;1).</p><!--TOC subsection id="sec302" Performing <span style="font-style:italic">k</span>-means or <span style="font-style:italic">k</span>-medians clustering-->
<h3 id="sec302" class="subsection">Performing <span style="font-style:italic">k</span>-means or <span style="font-style:italic">k</span>-medians clustering</h3><!--SEC END --><p>To perform <span style="font-style:italic">k</span>-means or <span style="font-style:italic">k</span>-medians clustering on the items stored in the record, use
</p><pre class="verbatim">&gt;&gt;&gt; clusterid, error, nfound = record.kcluster()
</pre><p>where the following arguments are defined:
</p><ul class="itemize"><li class="li-itemize">
<code>nclusters</code> (default: <code>2</code>) <br>
The number of clusters <span style="font-style:italic">k</span>.
</li><li class="li-itemize"><code>transpose</code> (default: <code>0</code>) <br>
Determines if rows (<code>transpose</code> is <code>0</code>) or columns (<code>transpose</code> is <code>1</code>) are to be clustered.
</li><li class="li-itemize"><code>npass</code> (default: <code>1</code>) <br>
The number of times the <span style="font-style:italic">k</span>-means/-medians clustering algorithm is performed, each time with a different (random) initial condition. If <code>initialid</code> is given, the value of <code>npass</code> is ignored and the clustering algorithm is run only once, as it behaves deterministically in that case.
</li><li class="li-itemize"><code>method</code> (default: <code>a</code>) <br>
describes how the center of a cluster is found:
<ul class="itemize"><li class="li-itemize">
<code>method=='a'</code>: arithmetic mean (<span style="font-style:italic">k</span>-means clustering);
</li><li class="li-itemize"><code>method=='m'</code>: median (<span style="font-style:italic">k</span>-medians clustering).
</li></ul>
For other values of <code>method</code>, the arithmetic mean is used.
</li><li class="li-itemize"><code>dist</code> (default: <code>'e'</code>, Euclidean distance) <br>
Defines the distance function to be used (see <a href="#sec%3Adistancefunctions">15.1</a>).
</li></ul><p>This function returns a tuple <code>(clusterid, error, nfound)</code>, where <code>clusterid</code> is an integer array containing the number of the cluster to which each row or cluster was assigned, <code>error</code> is the within-cluster sum of distances for the optimal clustering solution, and <code>nfound</code> is the number of times this optimal solution was found.</p><!--TOC subsection id="sec303" Calculating a Self-Organizing Map-->
<h3 id="sec303" class="subsection">Calculating a Self-Organizing Map</h3><!--SEC END --><p>To calculate a Self-Organizing Map of the items stored in the record, use
</p><pre class="verbatim">&gt;&gt;&gt; clusterid, celldata = record.somcluster()
</pre><p>where the following arguments are defined:
</p><ul class="itemize"><li class="li-itemize">
<code>transpose</code> (default: <code>0</code>) <br>
Determines if rows (<code>transpose</code> is <code>0</code>) or columns (<code>transpose</code> is <code>1</code>) are to be clustered.
</li><li class="li-itemize"><code>nxgrid, nygrid</code> (default: <code>2, 1</code>) <br>
The number of cells horizontally and vertically in the rectangular grid on which the Self-Organizing Map is calculated.
</li><li class="li-itemize"><code>inittau</code> (default: <code>0.02</code>) <br>
The initial value for the parameter &#X3C4; that is used in the SOM algorithm. The default value for <code>inittau</code> is 0.02, which was used in Michael Eisen&#X2019;s Cluster/TreeView program.
</li><li class="li-itemize"><code>niter</code> (default: <code>1</code>) <br>
The number of iterations to be performed.
</li><li class="li-itemize"><code>dist</code> (default: <code>'e'</code>, Euclidean distance) <br>
Defines the distance function to be used (see <a href="#sec%3Adistancefunctions">15.1</a>).
</li></ul><p>This function returns the tuple <code>(clusterid, celldata)</code>:
</p><ul class="itemize"><li class="li-itemize">
<code>clusterid</code>: <br>
An array with two columns, where the number of rows is equal to the number of items that were clustered. Each row contains the <span style="font-style:italic">x</span> and <span style="font-style:italic">y</span> coordinates of the cell in the rectangular SOM grid to which the item was assigned.
</li><li class="li-itemize"><code>celldata</code>: <br>
An array with dimensions (<code>nxgrid</code>, <code>nygrid</code>, number of columns) if rows are being clustered, or (<code>nxgrid</code>, <code>nygrid</code>, number of rows) if columns are being clustered. Each element <code>[ix][iy]</code> of this array is a 1D vector containing the gene expression data for the centroid of the cluster in the grid cell with coordinates <code>[ix][iy]</code>.
</li></ul><!--TOC subsection id="sec304" Saving the clustering result-->
<h3 id="sec304" class="subsection">Saving the clustering result</h3><!--SEC END --><p>To save the clustering result, use
</p><pre class="verbatim">&gt;&gt;&gt; record.save(jobname, geneclusters, expclusters)
</pre><p>where the following arguments are defined:
</p><ul class="itemize"><li class="li-itemize">
<code>jobname</code> <br>
The string <code>jobname</code> is used as the base name for names of the files that are to be saved.
</li><li class="li-itemize"><code>geneclusters</code> <br>
This argument describes the gene (row-wise) clustering result. In case of <span style="font-style:italic">k</span>-means clustering, this is a 1D array containing the number of the cluster each gene belongs to. It can be calculated using <code>kcluster</code>. In case of hierarchical clustering, <code>geneclusters</code> is a <code>Tree</code> object.
</li><li class="li-itemize"><code>expclusters</code> <br>
This argument describes the (column-wise) clustering result for the experimental conditions. In case of <span style="font-style:italic">k</span>-means clustering, this is a 1D array containing the number of the cluster each experimental condition belongs to. It can be calculated using <code>kcluster</code>. In case of hierarchical clustering, <code>expclusters</code> is a <code>Tree</code> object.
</li></ul><p>This method writes the text file <code>jobname.cdt</code>, <code>jobname.gtr</code>, <code>jobname.atr</code>, <code>jobname*.kgg</code>, and/or <code>jobname*.kag</code> for subsequent reading by the Java TreeView program. If <code>geneclusters</code> and <code>expclusters</code> are both <code>None</code>, this method only writes the text file <code>jobname.cdt</code>; this file can subsequently be read into a new <code>Record</code> object.
</p>
<!--TOC section id="sec305" Example calculation-->
<h2 id="sec305" class="section">15.8&#XA0;&#XA0;Example calculation</h2><!--SEC END --><p>This is an example of a hierarchical clustering calculation, using single linkage clustering for genes and maximum linkage clustering for experimental conditions. As the Euclidean distance is being used for gene clustering, it is necessary to scale the node distances <code>genetree</code> such that they are all between zero and one. This is needed for the Java TreeView code to display the tree diagram correctly. To cluster the experimental conditions, the uncentered correlation is being used. No scaling is needed in this case, as the distances in <code>exptree</code> are already between zero and two. The example data <code>cyano.txt</code> can be found in the <code>data</code> subdirectory.</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Cluster
&gt;&gt;&gt; handle = open("cyano.txt")
&gt;&gt;&gt; record = Cluster.read(handle)
&gt;&gt;&gt; handle.close()
&gt;&gt;&gt; genetree = record.treecluster(method='s')
&gt;&gt;&gt; genetree.scale()
&gt;&gt;&gt; exptree = record.treecluster(dist='u', transpose=1)
&gt;&gt;&gt; record.save("cyano_result", genetree, exptree)
</pre><p>This will create the files <code>cyano_result.cdt</code>, <code>cyano_result.gtr</code>, and <code>cyano_result.atr</code>.</p><p>Similarly, we can save a <span style="font-style:italic">k</span>-means clustering solution:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import Cluster
&gt;&gt;&gt; handle = open("cyano.txt")
&gt;&gt;&gt; record = Cluster.read(handle)
&gt;&gt;&gt; handle.close()
&gt;&gt;&gt; (geneclusters, error, ifound) = record.kcluster(nclusters=5, npass=1000)
&gt;&gt;&gt; (expclusters, error, ifound) = record.kcluster(nclusters=2, npass=100, transpose=1)
&gt;&gt;&gt; record.save("cyano_result", geneclusters, expclusters)
</pre><p>This will create the files <code>cyano_result_K_G2_A2.cdt</code>, <code>cyano_result_K_G2.kgg</code>, and <code>cyano_result_K_A2.kag</code>.</p>
<!--TOC section id="sec306" Auxiliary functions-->
<h2 id="sec306" class="section">15.9&#XA0;&#XA0;Auxiliary functions</h2><!--SEC END --><p><code>median(data)</code>
returns the median of the 1D array <code>data</code>.</p><p><code>mean(data)</code>
returns the mean of the 1D array <code>data</code>.</p><p><code>version()</code>
returns the version number of the underlying C Clustering Library as a string.</p>
<!--TOC chapter id="sec307" Supervised learning methods-->
<h1 id="sec307" class="chapter">Chapter&#XA0;16&#XA0;&#XA0;Supervised learning methods</h1><!--SEC END --><p>Note the supervised learning methods described in this chapter all require Numerical Python (numpy) to be installed.</p>
<!--TOC section id="sec308" The Logistic Regression Model-->
<h2 id="sec308" class="section">16.1&#XA0;&#XA0;The Logistic Regression Model</h2><!--SEC END --><p>
<a id="sec:LogisticRegression"></a></p>
<!--TOC subsection id="sec309" Background and Purpose-->
<h3 id="sec309" class="subsection">16.1.1&#XA0;&#XA0;Background and Purpose</h3><!--SEC END --><p>Logistic regression is a supervised learning approach that attempts to distinguish <span style="font-style:italic">K</span> classes from each other using a weighted sum of some predictor variables <span style="font-style:italic">x</span><sub><span style="font-style:italic">i</span></sub>. The logistic regression model is used to calculate the weights &#X3B2;<sub><span style="font-style:italic">i</span></sub> of the predictor variables. In Biopython, the logistic regression model is currently implemented for two classes only (<span style="font-style:italic">K</span> = 2); the number of predictor variables has no predefined limit.</p><p>As an example, let&#X2019;s try to predict the operon structure in bacteria. An operon is a set of adjacent genes on the same strand of DNA that are transcribed into a single mRNA molecule. Translation of the single mRNA molecule then yields the individual proteins. For <span style="font-style:italic">Bacillus subtilis</span>, whose data we will be using, the average number of genes in an operon is about 2.4.</p><p>As a first step in understanding gene regulation in bacteria, we need to know the operon structure. For about 10% of the genes in <span style="font-style:italic">Bacillus subtilis</span>, the operon structure is known from experiments. A supervised learning method can be used to predict the operon structure for the remaining 90% of the genes.</p><p>For such a supervised learning approach, we need to choose some predictor variables <span style="font-style:italic">x</span><sub><span style="font-style:italic">i</span></sub> that can be measured easily and are somehow related to the operon structure. One predictor variable might be the distance in base pairs between genes. Adjacent genes belonging to the same operon tend to be separated by a relatively short distance, whereas adjacent genes in different operons tend to have a larger space between them to allow for promoter and terminator sequences. Another predictor variable is based on gene expression measurements. By definition, genes belonging to the same operon have equal gene expression profiles, while genes in different operons are expected to have different expression profiles. In practice, the measured expression profiles of genes in the same operon are not quite identical due to the presence of measurement errors. To assess the similarity in the gene expression profiles, we assume that the measurement errors follow a normal distribution and calculate the corresponding log-likelihood score.</p><p>We now have two predictor variables that we can use to predict if two adjacent genes on the same strand of DNA belong to the same operon:
</p><ul class="itemize"><li class="li-itemize">
<span style="font-style:italic">x</span><sub>1</sub>: the number of base pairs between them;
</li><li class="li-itemize"><span style="font-style:italic">x</span><sub>2</sub>: their similarity in expression profile.
</li></ul><p>In a logistic regression model, we use a weighted sum of these two predictors to calculate a joint score <span style="font-style:italic">S</span>:
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell">
<span style="font-style:italic">S</span>&#XA0;=&#XA0;&#X3B2;<sub>0</sub>&#XA0;+&#XA0;&#X3B2;<sub>1</sub>&#XA0;<span style="font-style:italic">x</span><sub>1</sub>&#XA0;+&#XA0;&#X3B2;<sub>2</sub>&#XA0;<span style="font-style:italic">x</span><sub>2</sub>.
&#XA0;&#XA0;&#XA0;&#XA0;(16.1)</td></tr>
</table><p>
The logistic regression model gives us appropriate values for the parameters &#X3B2;<sub>0</sub>, &#X3B2;<sub>1</sub>, &#X3B2;<sub>2</sub> using two sets of example genes:
</p><ul class="itemize"><li class="li-itemize">
OP: Adjacent genes, on the same strand of DNA, known to belong to the same operon;
</li><li class="li-itemize">NOP: Adjacent genes, on the same strand of DNA, known to belong to different operons.
</li></ul><p>In the logistic regression model, the probability of belonging to a class depends on the score via the logistic function. For the two classes OP and NOP, we can write this as
</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell">


&#XA0;&#XA0;&#XA0;&#XA0;&#XA0;

</td><td class="dcell"><table style="border-spacing:6px;border-collapse:separate;" class="cellpading0"><tr><td style="text-align:right;white-space:nowrap" >Pr(<span style="font-style:italic">OP</span>|<span style="font-style:italic">x</span><sub>1</sub>,&#XA0;<span style="font-style:italic">x</span><sub>2</sub>)</td><td style="text-align:center;white-space:nowrap" >&#XA0;=</td><td style="text-align:left;white-space:nowrap" ><table class="display"><tr style="vertical-align:middle"><td class="dcell">&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center">exp(&#X3B2;<sub>0</sub>&#XA0;+&#XA0;&#X3B2;<sub>1</sub>&#XA0;<span style="font-style:italic">x</span><sub>1</sub>&#XA0;+&#XA0;&#X3B2;<sub>2</sub>&#XA0;<span style="font-style:italic">x</span><sub>2</sub>)</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center">1+exp(&#X3B2;<sub>0</sub>&#XA0;+&#XA0;&#X3B2;<sub>1</sub>&#XA0;<span style="font-style:italic">x</span><sub>1</sub>&#XA0;+&#XA0;&#X3B2;<sub>2</sub>&#XA0;<span style="font-style:italic">x</span><sub>2</sub>)</td></tr>
</table></td><td class="dcell">&#XA0;<a id="eq:OP">&#XA0;</a>&#XA0;</td></tr>
</table></td><td style="text-align:right;white-space:nowrap" >&#XA0;&#XA0;&#XA0;&#XA0;(16.2)</td></tr>
<tr><td style="text-align:right;white-space:nowrap" >Pr(<span style="font-style:italic">NOP</span>|<span style="font-style:italic">x</span><sub>1</sub>,&#XA0;<span style="font-style:italic">x</span><sub>2</sub>)</td><td style="text-align:center;white-space:nowrap" >&#XA0;=</td><td style="text-align:left;white-space:nowrap" ><table class="display"><tr style="vertical-align:middle"><td class="dcell">&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center">1</td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center">1+exp(&#X3B2;<sub>0</sub>&#XA0;+&#XA0;&#X3B2;<sub>1</sub>&#XA0;<span style="font-style:italic">x</span><sub>1</sub>&#XA0;+&#XA0;&#X3B2;<sub>2</sub>&#XA0;<span style="font-style:italic">x</span><sub>2</sub>)</td></tr>
</table></td><td class="dcell">&#XA0;<a id="eq:NOP">&#XA0;</a>&#XA0;
</td></tr>
</table></td><td style="text-align:right;white-space:nowrap" >&#XA0;&#XA0;&#XA0;&#XA0;(16.3)</td></tr>
</table></td></tr>
</table><p>
Using a set of gene pairs for which it is known whether they belong to the same operon (class OP) or to different operons (class NOP), we can calculate the weights &#X3B2;<sub>0</sub>, &#X3B2;<sub>1</sub>, &#X3B2;<sub>2</sub> by maximizing the log-likelihood corresponding to the probability functions (<a href="#eq%3AOP">16.2</a>) and (<a href="#eq%3ANOP">16.3</a>).</p>
<!--TOC subsection id="sec310" Training the logistic regression model-->
<h3 id="sec310" class="subsection">16.1.2&#XA0;&#XA0;Training the logistic regression model</h3><!--SEC END --><p>
<a id="subsec:LogisticRegressionTraining"></a></p><blockquote class="table"><div class="center"><hr style="width:80%;height:2"></div>
<div class="center">
<div class="caption"><table style="border-spacing:6px;border-collapse:separate;" class="cellpading0"><tr><td style="vertical-align:top;text-align:left;" >Table 16.1: Adjacent gene pairs known to belong to the same operon (class OP) or to different operons (class NOP). Intergene distances are negative if the two genes overlap.</td></tr>
</table></div>
<table border=1  style="border-spacing:0;" class="cellpadding1"><tr><td style="text-align:center;border:solid 1px;white-space:nowrap" >Gene pair</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >Intergene distance (<span style="font-style:italic">x</span><sub>1</sub>)</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >Gene expression score (<span style="font-style:italic">x</span><sub>2</sub>)</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >Class</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">cotJA</span> &#X2014; <span style="font-style:italic">cotJB</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-53</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-200.78</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >OP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">yesK</span> &#X2014; <span style="font-style:italic">yesL</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >117</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-267.14</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >OP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">lplA</span> &#X2014; <span style="font-style:italic">lplB</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >57</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-163.47</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >OP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">lplB</span> &#X2014; <span style="font-style:italic">lplC</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >16</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-190.30</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >OP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">lplC</span> &#X2014; <span style="font-style:italic">lplD</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >11</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-220.94</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >OP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">lplD</span> &#X2014; <span style="font-style:italic">yetF</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >85</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-193.94</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >OP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">yfmT</span> &#X2014; <span style="font-style:italic">yfmS</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >16</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-182.71</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >OP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">yfmF</span> &#X2014; <span style="font-style:italic">yfmE</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >15</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-180.41</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >OP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">citS</span> &#X2014; <span style="font-style:italic">citT</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-26</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-181.73</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >OP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">citM</span> &#X2014; <span style="font-style:italic">yflN</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >58</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-259.87</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >OP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">yfiI</span> &#X2014; <span style="font-style:italic">yfiJ</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >126</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-414.53</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >NOP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">lipB</span> &#X2014; <span style="font-style:italic">yfiQ</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >191</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-249.57</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >NOP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">yfiU</span> &#X2014; <span style="font-style:italic">yfiV</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >113</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-265.28</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >NOP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">yfhH</span> &#X2014; <span style="font-style:italic">yfhI</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >145</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-312.99</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >NOP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">cotY</span> &#X2014; <span style="font-style:italic">cotX</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >154</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-213.83</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >NOP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">yjoB</span> &#X2014; <span style="font-style:italic">rapA</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >147</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-380.85</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >NOP</td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">ptsI</span> &#X2014; <span style="font-style:italic">splA</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >93</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-291.13</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >NOP </td></tr>
</table>
<a id="table:training"></a>
</div>
<div class="center"><hr style="width:80%;height:2"></div></blockquote><p>Table <a href="#table%3Atraining">16.1</a> lists some of the <span style="font-style:italic">Bacillus subtilis</span> gene pairs for which the operon structure is known.
Let&#X2019;s calculate the logistic regression model from these data:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import LogisticRegression
&gt;&gt;&gt; xs = [[-53, -200.78],
          [117, -267.14],
          [57, -163.47],
          [16, -190.30],
          [11, -220.94],
          [85, -193.94],
          [16, -182.71],
          [15, -180.41],
          [-26, -181.73],
          [58, -259.87],
          [126, -414.53],
          [191, -249.57],
          [113, -265.28],
          [145, -312.99],
          [154, -213.83],
          [147, -380.85],
          [93, -291.13]]
&gt;&gt;&gt; ys = [1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          0,
          0,
          0,
          0,
          0,
          0,
          0]
&gt;&gt;&gt; model = LogisticRegression.train(xs, ys)
</pre><p>Here, <code>xs</code> and <code>ys</code> are the training data: <code>xs</code> contains the predictor variables for each gene pair, and <code>ys</code> specifies if the gene pair belongs to the same operon (<code>1</code>, class OP) or different operons (<code>0</code>, class NOP). The resulting logistic regression model is stored in <code>model</code>, which contains the weights &#X3B2;<sub>0</sub>, &#X3B2;<sub>1</sub>, and &#X3B2;<sub>2</sub>:</p><pre class="verbatim">&gt;&gt;&gt; model.beta
[8.9830290157144681, -0.035968960444850887, 0.02181395662983519]
</pre><p>Note that &#X3B2;<sub>1</sub> is negative, as gene pairs with a shorter intergene distance have a higher probability of belonging to the same operon (class OP). On the other hand, &#X3B2;<sub>2</sub> is positive, as gene pairs belonging to the same operon typically have a higher similarity score of their gene expression profiles.
The parameter &#X3B2;<sub>0</sub> is positive due to the higher prevalence of operon gene pairs than non-operon gene pairs in the training data.</p><p>The function <code>train</code> has two optional arguments: <code>update_fn</code> and <code>typecode</code>. The <code>update_fn</code> can be used to specify a callback function, taking as arguments the iteration number and the log-likelihood. With the callback function, we can for example track the progress of the model calculation (which uses a Newton-Raphson iteration to maximize the log-likelihood function of the logistic regression model):</p><pre class="verbatim">&gt;&gt;&gt; def show_progress(iteration, loglikelihood):
        print("Iteration:", iteration, "Log-likelihood function:", loglikelihood)
&gt;&gt;&gt;
&gt;&gt;&gt; model = LogisticRegression.train(xs, ys, update_fn=show_progress)
Iteration: 0 Log-likelihood function: -11.7835020695
Iteration: 1 Log-likelihood function: -7.15886767672
Iteration: 2 Log-likelihood function: -5.76877209868
Iteration: 3 Log-likelihood function: -5.11362294338
Iteration: 4 Log-likelihood function: -4.74870642433
Iteration: 5 Log-likelihood function: -4.50026077146
Iteration: 6 Log-likelihood function: -4.31127773737
Iteration: 7 Log-likelihood function: -4.16015043396
Iteration: 8 Log-likelihood function: -4.03561719785
Iteration: 9 Log-likelihood function: -3.93073282192
Iteration: 10 Log-likelihood function: -3.84087660929
Iteration: 11 Log-likelihood function: -3.76282560605
Iteration: 12 Log-likelihood function: -3.69425027154
Iteration: 13 Log-likelihood function: -3.6334178602
Iteration: 14 Log-likelihood function: -3.57900855837
Iteration: 15 Log-likelihood function: -3.52999671386
Iteration: 16 Log-likelihood function: -3.48557145163
Iteration: 17 Log-likelihood function: -3.44508206139
Iteration: 18 Log-likelihood function: -3.40799948447
Iteration: 19 Log-likelihood function: -3.3738885624
Iteration: 20 Log-likelihood function: -3.3423876581
Iteration: 21 Log-likelihood function: -3.31319343769
Iteration: 22 Log-likelihood function: -3.2860493346
Iteration: 23 Log-likelihood function: -3.2607366863
Iteration: 24 Log-likelihood function: -3.23706784091
Iteration: 25 Log-likelihood function: -3.21488073614
Iteration: 26 Log-likelihood function: -3.19403459259
Iteration: 27 Log-likelihood function: -3.17440646052
Iteration: 28 Log-likelihood function: -3.15588842703
Iteration: 29 Log-likelihood function: -3.13838533947
Iteration: 30 Log-likelihood function: -3.12181293595
Iteration: 31 Log-likelihood function: -3.10609629966
Iteration: 32 Log-likelihood function: -3.09116857282
Iteration: 33 Log-likelihood function: -3.07696988017
Iteration: 34 Log-likelihood function: -3.06344642288
Iteration: 35 Log-likelihood function: -3.05054971191
Iteration: 36 Log-likelihood function: -3.03823591619
Iteration: 37 Log-likelihood function: -3.02646530573
Iteration: 38 Log-likelihood function: -3.01520177394
Iteration: 39 Log-likelihood function: -3.00441242601
Iteration: 40 Log-likelihood function: -2.99406722296
Iteration: 41 Log-likelihood function: -2.98413867259
</pre><p>The iteration stops once the increase in the log-likelihood function is less than 0.01. If no convergence is reached after 500 iterations, the <code>train</code> function returns with an <code>AssertionError</code>.</p><p>The optional keyword <code>typecode</code> can almost always be ignored. This keyword allows the user to choose the type of Numeric matrix to use. In particular, to avoid memory problems for very large problems, it may be necessary to use single-precision floats (Float8, Float16, etc.) rather than double, which is used by default.</p>
<!--TOC subsection id="sec311" Using the logistic regression model for classification-->
<h3 id="sec311" class="subsection">16.1.3&#XA0;&#XA0;Using the logistic regression model for classification</h3><!--SEC END --><p>Classification is performed by calling the <code>classify</code> function. Given a logistic regression model and the values for <span style="font-style:italic">x</span><sub>1</sub> and <span style="font-style:italic">x</span><sub>2</sub> (e.g. for a gene pair of unknown operon structure), the <code>classify</code> function returns <code>1</code> or <code>0</code>, corresponding to class OP and class NOP, respectively. For example, let&#X2019;s consider the gene pairs <span style="font-style:italic">yxcE</span>, <span style="font-style:italic">yxcD</span> and <span style="font-style:italic">yxiB</span>, <span style="font-style:italic">yxiA</span>:</p><blockquote class="table"><div class="center"><hr style="width:80%;height:2"></div>
<div class="center">
<div class="caption"><table style="border-spacing:6px;border-collapse:separate;" class="cellpading0"><tr><td style="vertical-align:top;text-align:left;" >Table 16.2: Adjacent gene pairs of unknown operon status.</td></tr>
</table></div>
<table border=1  style="border-spacing:0;" class="cellpadding1"><tr><td style="text-align:center;border:solid 1px;white-space:nowrap" >Gene pair</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >Intergene distance <span style="font-style:italic">x</span><sub>1</sub></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >Gene expression score <span style="font-style:italic">x</span><sub>2</sub> </td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">yxcE</span> &#X2014; <span style="font-style:italic">yxcD</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >6</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-173.143442352 </td></tr>
<tr><td style="text-align:center;border:solid 1px;white-space:nowrap" ><span style="font-style:italic">yxiB</span> &#X2014; <span style="font-style:italic">yxiA</span></td><td style="text-align:center;border:solid 1px;white-space:nowrap" >309</td><td style="text-align:center;border:solid 1px;white-space:nowrap" >-271.005880394 </td></tr>
</table>
</div>
<div class="center"><hr style="width:80%;height:2"></div></blockquote><p>The logistic regression model classifies <span style="font-style:italic">yxcE</span>, <span style="font-style:italic">yxcD</span> as belonging to the same operon (class OP), while <span style="font-style:italic">yxiB</span>, <span style="font-style:italic">yxiA</span> are predicted to belong to different operons:
</p><pre class="verbatim">&gt;&gt;&gt; print("yxcE, yxcD:", LogisticRegression.classify(model, [6, -173.143442352]))
yxcE, yxcD: 1
&gt;&gt;&gt; print("yxiB, yxiA:", LogisticRegression.classify(model, [309, -271.005880394]))
yxiB, yxiA: 0
</pre><p>(which, by the way, agrees with the biological literature).</p><p>To find out how confident we can be in these predictions, we can call the <code>calculate</code> function to obtain the probabilities (equations (<a href="#eq%3AOP">16.2</a>) and <a href="#eq%3ANOP">16.3</a>) for class OP and NOP. For <span style="font-style:italic">yxcE</span>, <span style="font-style:italic">yxcD</span> we find
</p><pre class="verbatim">&gt;&gt;&gt; q, p = LogisticRegression.calculate(model, [6, -173.143442352])
&gt;&gt;&gt; print("class OP: probability =", p, "class NOP: probability =", q)
class OP: probability = 0.993242163503 class NOP: probability = 0.00675783649744
</pre><p>and for <span style="font-style:italic">yxiB</span>, <span style="font-style:italic">yxiA</span>
</p><pre class="verbatim">&gt;&gt;&gt; q, p = LogisticRegression.calculate(model, [309, -271.005880394])
&gt;&gt;&gt; print("class OP: probability =", p, "class NOP: probability =", q)
class OP: probability = 0.000321211251817 class NOP: probability = 0.999678788748
</pre><p>To get some idea of the prediction accuracy of the logistic regression model, we can apply it to the training data:
</p><pre class="verbatim">&gt;&gt;&gt; for i in range(len(ys)):
        print("True:", ys[i], "Predicted:", LogisticRegression.classify(model, xs[i]))
True: 1 Predicted: 1
True: 1 Predicted: 0
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
</pre><p>showing that the prediction is correct for all but one of the gene pairs. A more reliable estimate of the prediction accuracy can be found from a leave-one-out analysis, in which the model is recalculated from the training data after removing the gene to be predicted:
</p><pre class="verbatim">&gt;&gt;&gt; for i in range(len(ys)):
        model = LogisticRegression.train(xs[:i]+xs[i+1:], ys[:i]+ys[i+1:])
        print("True:", ys[i], "Predicted:", LogisticRegression.classify(model, xs[i]))
True: 1 Predicted: 1
True: 1 Predicted: 0
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 1
True: 0 Predicted: 0
True: 0 Predicted: 0
</pre><p>The leave-one-out analysis shows that the prediction of the logistic regression model is incorrect for only two of the gene pairs, which corresponds to a prediction accuracy of 88%.</p>
<!--TOC subsection id="sec312" Logistic Regression, Linear Discriminant Analysis, and Support Vector Machines-->
<h3 id="sec312" class="subsection">16.1.4&#XA0;&#XA0;Logistic Regression, Linear Discriminant Analysis, and Support Vector Machines</h3><!--SEC END --><p>The logistic regression model is similar to linear discriminant analysis. In linear discriminant analysis, the class probabilities also follow equations (<a href="#eq%3AOP">16.2</a>) and (<a href="#eq%3ANOP">16.3</a>). However, instead of estimating the coefficients &#X3B2; directly, we first fit a normal distribution to the predictor variables <span style="font-style:italic">x</span>. The coefficients &#X3B2; are then calculated from the means and covariances of the normal distribution. If the distribution of <span style="font-style:italic">x</span> is indeed normal, then we expect linear discriminant analysis to perform better than the logistic regression model. The logistic regression model, on the other hand, is more robust to deviations from normality.</p><p>Another similar approach is a support vector machine with a linear kernel. Such an SVM also uses a linear combination of the predictors, but estimates the coefficients &#X3B2; from the predictor variables <span style="font-style:italic">x</span> near the boundary region between the classes. If the logistic regression model (equations (<a href="#eq%3AOP">16.2</a>) and (<a href="#eq%3ANOP">16.3</a>)) is a good description for <span style="font-style:italic">x</span> away from the boundary region, we expect the logistic regression model to perform better than an SVM with a linear kernel, as it relies on more data. If not, an SVM with a linear kernel may perform better.</p><p>Trevor Hastie, Robert Tibshirani, and Jerome Friedman: <span style="font-style:italic">The Elements of Statistical Learning. Data Mining, Inference, and Prediction</span>. Springer Series in Statistics, 2001. Chapter 4.4.</p>
<!--TOC section id="sec313" <span style="font-style:italic">k</span>-Nearest Neighbors-->
<h2 id="sec313" class="section">16.2&#XA0;&#XA0;<span style="font-style:italic">k</span>-Nearest Neighbors</h2><!--SEC END -->
<!--TOC subsection id="sec314" Background and purpose-->
<h3 id="sec314" class="subsection">16.2.1&#XA0;&#XA0;Background and purpose</h3><!--SEC END --><p>The <span style="font-style:italic">k</span>-nearest neighbors method is a supervised learning approach that does not need to fit a model to the data. Instead, data points are classified based on the categories of the <span style="font-style:italic">k</span> nearest neighbors in the training data set.</p><p>In Biopython, the <span style="font-style:italic">k</span>-nearest neighbors method is available in <code>Bio.kNN</code>. To illustrate the use of the <span style="font-style:italic">k</span>-nearest neighbor method in Biopython, we will use the same operon data set as in section <a href="#sec%3ALogisticRegression">16.1</a>.</p>
<!--TOC subsection id="sec315" Initializing a <span style="font-style:italic">k</span>-nearest neighbors model-->
<h3 id="sec315" class="subsection">16.2.2&#XA0;&#XA0;Initializing a <span style="font-style:italic">k</span>-nearest neighbors model</h3><!--SEC END --><p>Using the data in Table <a href="#table%3Atraining">16.1</a>, we create and initialize a <span style="font-style:italic">k</span>-nearest neighbors model as follows:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import kNN
&gt;&gt;&gt; k = 3
&gt;&gt;&gt; model = kNN.train(xs, ys, k)
</pre><p>where <code>xs</code> and <code>ys</code> are the same as in Section <a href="#subsec%3ALogisticRegressionTraining">16.1.2</a>. Here, <code>k</code> is the number of neighbors <span style="font-style:italic">k</span> that will be considered for the classification. For classification into two classes, choosing an odd number for <span style="font-style:italic">k</span> lets you avoid tied votes. The function name <code>train</code> is a bit of a misnomer, since no model training is done: this function simply stores <code>xs</code>, <code>ys</code>, and <code>k</code> in <code>model</code>.</p>
<!--TOC subsection id="sec316" Using a <span style="font-style:italic">k</span>-nearest neighbors model for classification-->
<h3 id="sec316" class="subsection">16.2.3&#XA0;&#XA0;Using a <span style="font-style:italic">k</span>-nearest neighbors model for classification</h3><!--SEC END --><p>To classify new data using the <span style="font-style:italic">k</span>-nearest neighbors model, we use the <code>classify</code> function. This function takes a data point (<span style="font-style:italic">x</span><sub>1</sub>,<span style="font-style:italic">x</span><sub>2</sub>) and finds the <span style="font-style:italic">k</span>-nearest neighbors in the training data set <code>xs</code>. The data point (<span style="font-style:italic">x</span><sub>1</sub>, <span style="font-style:italic">x</span><sub>2</sub>) is then classified based on which category (<code>ys</code>) occurs most among the <span style="font-style:italic">k</span> neighbors.</p><p>For the example of the gene pairs <span style="font-style:italic">yxcE</span>, <span style="font-style:italic">yxcD</span> and <span style="font-style:italic">yxiB</span>, <span style="font-style:italic">yxiA</span>, we find:
</p><pre class="verbatim">&gt;&gt;&gt; x = [6, -173.143442352]
&gt;&gt;&gt; print("yxcE, yxcD:", kNN.classify(model, x))
yxcE, yxcD: 1
&gt;&gt;&gt; x = [309, -271.005880394]
&gt;&gt;&gt; print("yxiB, yxiA:", kNN.classify(model, x))
yxiB, yxiA: 0
</pre><p>In agreement with the logistic regression model, <span style="font-style:italic">yxcE</span>, <span style="font-style:italic">yxcD</span> are classified as belonging to the same operon (class OP), while <span style="font-style:italic">yxiB</span>, <span style="font-style:italic">yxiA</span> are predicted to belong to different operons.</p><p>The <code>classify</code> function lets us specify both a distance function and a weight function as optional arguments. The distance function affects which <span style="font-style:italic">k</span> neighbors are chosen as the nearest neighbors, as these are defined as the neighbors with the smallest distance to the query point (<span style="font-style:italic">x</span>, <span style="font-style:italic">y</span>). By default, the Euclidean distance is used. Instead, we could for example use the city-block (Manhattan) distance:</p><pre class="verbatim">&gt;&gt;&gt; def cityblock(x1, x2):
...    assert len(x1)==2
...    assert len(x2)==2
...    distance = abs(x1[0]-x2[0]) + abs(x1[1]-x2[1])
...    return distance
... 
&gt;&gt;&gt; x = [6, -173.143442352]
&gt;&gt;&gt; print("yxcE, yxcD:", kNN.classify(model, x, distance_fn = cityblock))
yxcE, yxcD: 1
</pre><p>The weight function can be used for weighted voting. For example, we may want to give closer neighbors a higher weight than neighbors that are further away:</p><pre class="verbatim">&gt;&gt;&gt; def weight(x1, x2):
...    assert len(x1)==2
...    assert len(x2)==2
...    return exp(-abs(x1[0]-x2[0]) - abs(x1[1]-x2[1]))
... 
&gt;&gt;&gt; x = [6, -173.143442352]
&gt;&gt;&gt; print("yxcE, yxcD:", kNN.classify(model, x, weight_fn = weight))
yxcE, yxcD: 1
</pre><p>By default, all neighbors are given an equal weight.</p><p>To find out how confident we can be in these predictions, we can call the <code>calculate</code> function, which will calculate the total weight assigned to the classes OP and NOP. For the default weighting scheme, this reduces to the number of neighbors in each category. For <span style="font-style:italic">yxcE</span>, <span style="font-style:italic">yxcD</span>, we find
</p><pre class="verbatim">&gt;&gt;&gt; x = [6, -173.143442352]
&gt;&gt;&gt; weight = kNN.calculate(model, x)
&gt;&gt;&gt; print("class OP: weight =", weight[0], "class NOP: weight =", weight[1])
class OP: weight = 0.0 class NOP: weight = 3.0
</pre><p>which means that all three neighbors of <code>x1</code>, <code>x2</code> are in the NOP class. As another example, for <span style="font-style:italic">yesK</span>, <span style="font-style:italic">yesL</span> we find</p><pre class="verbatim">&gt;&gt;&gt; x = [117, -267.14]
&gt;&gt;&gt; weight = kNN.calculate(model, x)
&gt;&gt;&gt; print("class OP: weight =", weight[0], "class NOP: weight =", weight[1])
class OP: weight = 2.0 class NOP: weight = 1.0
</pre><p>which means that two neighbors are operon pairs and one neighbor is a non-operon pair.</p><p>To get some idea of the prediction accuracy of the <span style="font-style:italic">k</span>-nearest neighbors approach, we can apply it to the training data:
</p><pre class="verbatim">&gt;&gt;&gt; for i in range(len(ys)):
        print("True:", ys[i], "Predicted:", kNN.classify(model, xs[i]))
True: 1 Predicted: 1
True: 1 Predicted: 0
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
</pre><p>showing that the prediction is correct for all but two of the gene pairs. A more reliable estimate of the prediction accuracy can be found from a leave-one-out analysis, in which the model is recalculated from the training data after removing the gene to be predicted:
</p><pre class="verbatim">&gt;&gt;&gt; for i in range(len(ys)):
        model = kNN.train(xs[:i]+xs[i+1:], ys[:i]+ys[i+1:])
        print("True:", ys[i], "Predicted:", kNN.classify(model, xs[i]))
True: 1 Predicted: 1
True: 1 Predicted: 0
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 1
True: 1 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 1
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 0
True: 0 Predicted: 1
</pre><p>The leave-one-out analysis shows that <span style="font-style:italic">k</span>-nearest neighbors model is correct for 13 out of 17 gene pairs, which corresponds to a prediction accuracy of 76%.</p>
<!--TOC section id="sec317" Na&#XEF;ve Bayes-->
<h2 id="sec317" class="section">16.3&#XA0;&#XA0;Na&#XEF;ve Bayes</h2><!--SEC END --><p>This section will describe the <code>Bio.NaiveBayes</code> module.</p>
<!--TOC section id="sec318" Maximum Entropy-->
<h2 id="sec318" class="section">16.4&#XA0;&#XA0;Maximum Entropy</h2><!--SEC END --><p>This section will describe the <code>Bio.MaximumEntropy</code> module.</p>
<!--TOC section id="sec319" Markov Models-->
<h2 id="sec319" class="section">16.5&#XA0;&#XA0;Markov Models</h2><!--SEC END --><p>This section will describe the <code>Bio.MarkovModel</code> and/or <code>Bio.HMM.MarkovModel</code> modules.</p>
<!--TOC chapter id="sec320" Graphics including GenomeDiagram-->
<h1 id="sec320" class="chapter">Chapter&#XA0;17&#XA0;&#XA0;Graphics including GenomeDiagram</h1><!--SEC END --><p>
<a id="chapter:graphics"></a></p><p>The <code>Bio.Graphics</code> module depends on the third party Python library
<a href="http://www.reportlab.org">ReportLab</a>. Although focused on producing PDF files,
ReportLab can also create encapsulated postscript (EPS) and (SVG) files. In addition
to these vector based images, provided certain further dependencies such as the
<a href="http://www.pythonware.com/products/pil/">Python Imaging Library (PIL)</a> are
installed, ReportLab can also output bitmap images (including JPEG, PNG, GIF, BMP
and PICT formats).</p>
<!--TOC section id="sec321" GenomeDiagram-->
<h2 id="sec321" class="section">17.1&#XA0;&#XA0;GenomeDiagram</h2><!--SEC END --><p>
<a id="sec:genomediagram"></a>
</p>
<!--TOC subsection id="sec322" Introduction-->
<h3 id="sec322" class="subsection">17.1.1&#XA0;&#XA0;Introduction</h3><!--SEC END --><p>The <code>Bio.Graphics.GenomeDiagram</code> module was added to Biopython 1.50,
having previously been available as a separate Python module dependent on Biopython.
GenomeDiagram is described in the Bioinformatics journal publication by Pritchard et al. (2006) [<a href="#pritchard2006">2</a>],
which includes some examples images. There is a PDF copy of the old manual here,
<a href="http://biopython.org/DIST/docs/GenomeDiagram/userguide.pdf"><span style="font-family:monospace">http://biopython.org/DIST/docs/GenomeDiagram/userguide.pdf</span></a> which has some
more examples.
</p><p>As the name might suggest, GenomeDiagram was designed for drawing whole genomes, in
particular prokaryotic genomes, either as linear diagrams (optionally broken up into
fragments to fit better) or as circular wheel diagrams. Have a look at Figure 2 in
Toth <span style="font-style:italic">et al.</span> (2006) [<a href="#toth2006">3</a>]
for a good example. It proved also well suited to drawing quite detailed figures for
smaller genomes such as phage, plasmids or mitochrondia, for example see Figures 1
and 2 in Van der Auwera <span style="font-style:italic">et al.</span> (2009) [<a href="#vanderauwera2009">4</a>]
(shown with additional manual editing).</p><p>This module is easiest to use if you have your genome loaded as a <code>SeqRecord</code>
object containing lots of <code>SeqFeature</code> objects - for example as loaded from a
GenBank file (see Chapters&#XA0;<a href="#chapter%3ASeqRecord">4</a> and&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>).</p>
<!--TOC subsection id="sec323" Diagrams, tracks, feature-sets and features-->
<h3 id="sec323" class="subsection">17.1.2&#XA0;&#XA0;Diagrams, tracks, feature-sets and features</h3><!--SEC END --><p>GenomeDiagram uses a nested set of objects. At the top level, you have a diagram
object representing a sequence (or sequence region) along the horizontal axis (or
circle). A diagram can contain one or more tracks, shown stacked vertically (or
radially on circular diagrams). These will typically all have the same length
and represent the same sequence region. You might use one track to show the gene
locations, another to show regulatory regions, and a third track to show the GC
percentage. </p><p>The most commonly used type of track will contain features, bundled together in
feature-sets. You might choose to use one feature-set for all your CDS features,
and another for tRNA features. This isn&#X2019;t required - they can all go in the same
feature-set, but it makes it easier to update the properties of just selected
features (e.g. make all the tRNA features red).</p><p>There are two main ways to build up a complete diagram. Firstly, the top down
approach where you create a diagram object, and then using its methods add
track(s), and use the track methods to add feature-set(s), and use their
methods to add the features. Secondly, you can create the individual objects
separately (in whatever order suits your code), and then combine them.</p>
<!--TOC subsection id="sec324" A top down example-->
<h3 id="sec324" class="subsection">17.1.3&#XA0;&#XA0;A top down example</h3><!--SEC END --><p>
<a id="sec:gd_top_down"></a></p><p>We&#X2019;re going to draw a whole genome from a <code>SeqRecord</code> object read in from
a GenBank file (see Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>). This example uses the
pPCP1 plasmid from <span style="font-style:italic">Yersinia pestis biovar Microtus</span>, the file is
included with the Biopython unit tests under the GenBank folder, or online
<a href="http://biopython.org/SRC/biopython/Tests/GenBank/NC_005816.gb"><span style="font-family:monospace">NC_005816.gb</span></a> from our website.</p><pre class="verbatim">from reportlab.lib import colors
from reportlab.lib.units import cm
from Bio.Graphics import GenomeDiagram
from Bio import SeqIO
record = SeqIO.read("NC_005816.gb", "genbank")
</pre><p>We&#X2019;re using a top down approach, so after loading in our sequence we next
create an empty diagram, then add an (empty) track, and to that add an
(empty) feature set:</p><pre class="verbatim">gd_diagram = GenomeDiagram.Diagram("Yersinia pestis biovar Microtus plasmid pPCP1")
gd_track_for_features = gd_diagram.new_track(1, name="Annotated Features")
gd_feature_set = gd_track_for_features.new_set()
</pre><p>Now the fun part - we take each gene <code>SeqFeature</code> object in our
<code>SeqRecord</code>, and use it to generate a feature on the diagram. We&#X2019;re
going to color them blue, alternating between a dark blue and a light blue.
</p><pre class="verbatim">for feature in record.features:
    if feature.type != "gene":
        #Exclude this feature
        continue
    if len(gd_feature_set) % 2 == 0:
        color = colors.blue
    else:
        color = colors.lightblue
    gd_feature_set.add_feature(feature, color=color, label=True)
</pre><p>Now we come to actually making the output file. This happens in two steps,
first we call the <code>draw</code> method, which creates all the shapes using
ReportLab objects. Then we call the <code>write</code> method which renders these
to the requested file format. Note you can output in multiple file formats:</p><pre class="verbatim">gd_diagram.draw(format="linear", orientation="landscape", pagesize='A4',
                fragments=4, start=0, end=len(record))
gd_diagram.write("plasmid_linear.pdf", "PDF")
gd_diagram.write("plasmid_linear.eps", "EPS")
gd_diagram.write("plasmid_linear.svg", "SVG")
</pre><p>Also, provided you have the dependencies installed, you can also do bitmaps,
for example:</p><pre class="verbatim">gd_diagram.write("plasmid_linear.png", "PNG")
</pre><p>
<img src="images/plasmid_linear.png" width=550, height=400></p><p>

Notice that the <code>fragments</code> argument which we set to four controls how
many pieces the genome gets broken up into.</p><p>If you want to do a circular figure, then try this:</p><pre class="verbatim">gd_diagram.draw(format="circular", circular=True, pagesize=(20*cm,20*cm),
                start=0, end=len(record), circle_core=0.7)
gd_diagram.write("plasmid_circular.pdf", "PDF")
</pre><p>
<img src="images/plasmid_circular.png" width=400, height=400></p><p>

These figures are not very exciting, but we&#X2019;ve only just got started.</p>
<!--TOC subsection id="sec325" A bottom up example-->
<h3 id="sec325" class="subsection">17.1.4&#XA0;&#XA0;A bottom up example</h3><!--SEC END --><p>
Now let&#X2019;s produce exactly the same figures, but using the bottom up approach.
This means we create the different objects directly (and this can be done in
almost any order) and then combine them.</p><pre class="verbatim">from reportlab.lib import colors
from reportlab.lib.units import cm
from Bio.Graphics import GenomeDiagram
from Bio import SeqIO
record = SeqIO.read("NC_005816.gb", "genbank")

#Create the feature set and its feature objects,
gd_feature_set = GenomeDiagram.FeatureSet()
for feature in record.features:
    if feature.type != "gene":
        #Exclude this feature
        continue
    if len(gd_feature_set) % 2 == 0:
        color = colors.blue
    else:
        color = colors.lightblue
    gd_feature_set.add_feature(feature, color=color, label=True)
#(this for loop is the same as in the previous example)

#Create a track, and a diagram
gd_track_for_features = GenomeDiagram.Track(name="Annotated Features")
gd_diagram = GenomeDiagram.Diagram("Yersinia pestis biovar Microtus plasmid pPCP1")

#Now have to glue the bits together...
gd_track_for_features.add_set(gd_feature_set)
gd_diagram.add_track(gd_track_for_features, 1)
</pre><p>You can now call the <code>draw</code> and <code>write</code> methods as before to produce
a linear or circular diagram, using the code at the end of the top-down example
above. The figures should be identical.</p>
<!--TOC subsection id="sec326" Features without a SeqFeature-->
<h3 id="sec326" class="subsection">17.1.5&#XA0;&#XA0;Features without a SeqFeature</h3><!--SEC END --><p>
<a id="sec:gd_features_without_seqfeatures"></a></p><p>In the above example we used a <code>SeqRecord</code>&#X2019;s <code>SeqFeature</code> objects
to build our diagram (see also Section&#XA0;<a href="#sec%3Aseq_features">4.3</a>).
Sometimes you won&#X2019;t have <code>SeqFeature</code> objects,
but just the coordinates for a feature you want to draw. You have to create
minimal <code>SeqFeature</code> object, but this is easy:</p><pre class="verbatim">from Bio.SeqFeature import SeqFeature, FeatureLocation
my_seq_feature = SeqFeature(FeatureLocation(50,100),strand=+1)
</pre><p>For strand, use <span style="font-family:monospace">+1</span> for the forward strand, <span style="font-family:monospace">-1</span> for the
reverse strand, and <span style="font-family:monospace">None</span> for both. Here is a short self contained
example:</p><pre class="verbatim">from Bio.SeqFeature import SeqFeature, FeatureLocation
from Bio.Graphics import GenomeDiagram
from reportlab.lib.units import cm

gdd = GenomeDiagram.Diagram('Test Diagram')
gdt_features = gdd.new_track(1, greytrack=False)
gds_features = gdt_features.new_set()

#Add three features to show the strand options,
feature = SeqFeature(FeatureLocation(25, 125), strand=+1)
gds_features.add_feature(feature, name="Forward", label=True)
feature = SeqFeature(FeatureLocation(150, 250), strand=None)
gds_features.add_feature(feature, name="Strandless", label=True)
feature = SeqFeature(FeatureLocation(275, 375), strand=-1)
gds_features.add_feature(feature, name="Reverse", label=True)

gdd.draw(format='linear', pagesize=(15*cm,4*cm), fragments=1,
         start=0, end=400)
gdd.write("GD_labels_default.pdf", "pdf")
</pre><p>
The top part of the image in the next subsection shows the output


(in the default feature color, pale green).</p><p>Notice that we have used the <span style="font-family:monospace">name</span> argument here to specify the
caption text for these features. This is discussed in more detail next.</p>
<!--TOC subsection id="sec327" Feature captions-->
<h3 id="sec327" class="subsection">17.1.6&#XA0;&#XA0;Feature captions</h3><!--SEC END --><p>
<a id="sec:gd_feature_captions"></a></p><p>Recall we used the following (where <span style="font-family:monospace">feature</span> was a
<code>SeqFeature</code> object) to add a feature to the diagram:</p><pre class="verbatim">gd_feature_set.add_feature(feature, color=color, label=True)
</pre><p>In the example above the <code>SeqFeature</code> annotation was used to pick a
sensible caption for the features. By default the following possible entries
under the <code>SeqFeature</code> object&#X2019;s qualifiers dictionary are used:
<span style="font-family:monospace">gene</span>, <span style="font-family:monospace">label</span>, <span style="font-family:monospace">name</span>, <span style="font-family:monospace">locus_tag</span>, and
<span style="font-family:monospace">product</span>. More simply, you can specify a name directly:</p><pre class="verbatim">gd_feature_set.add_feature(feature, color=color, label=True, name="My Gene")
</pre><p>In addition to the caption text for each feature&#X2019;s label, you can also choose
the font, position (this defaults to the start of the sigil, you can also
choose the middle or at the end) and orientation (for linear diagrams only,
where this defaults to rotated by 45 degrees):</p><pre class="verbatim">#Large font, parallel with the track
gd_feature_set.add_feature(feature, label=True, color="green",
                           label_size=25, label_angle=0)

#Very small font, perpendicular to the track (towards it)
gd_feature_set.add_feature(feature, label=True, color="purple",
                           label_position="end",
                           label_size=4, label_angle=90)

#Small font, perpendicular to the track (away from it)
gd_feature_set.add_feature(feature, label=True, color="blue",
                           label_position="middle",
                           label_size=6, label_angle=-90)
</pre><p>Combining each of these three fragments with the complete example
in the previous section should give something like

this:</p><p><img src="images/GD_sigil_labels.png" width=600, height=700>
<a id="fig:gd_sigil_labels"></a></p><p>We&#X2019;ve not shown it here, but you can also set <span style="font-family:monospace">label_color</span> to
control the label&#X2019;s color (used in Section&#XA0;<a href="#sec%3Agd_nice_example">17.1.9</a>).</p><p>You&#X2019;ll notice the default font is quite small - this makes sense because
you will usually be drawing many (small) features on a page, not just a
few large ones as shown here.</p>
<!--TOC subsection id="sec328" Feature sigils-->
<h3 id="sec328" class="subsection">17.1.7&#XA0;&#XA0;Feature sigils</h3><!--SEC END --><p>
<a id="sec:gd_sigils"></a></p><p>The examples above have all just used the default sigil for the feature, a
plain box, which was all that was available in the last publicly released standalone version of GenomeDiagram. Arrow sigils were included when
GenomeDiagram was added to Biopython 1.50:</p><pre class="verbatim">#Default uses a BOX sigil
gd_feature_set.add_feature(feature)

#You can make this explicit:
gd_feature_set.add_feature(feature, sigil="BOX")

#Or opt for an arrow:
gd_feature_set.add_feature(feature, sigil="ARROW")
</pre><p>Biopython 1.61 added three more sigils,</p><pre class="verbatim">#Box with corners cut off (making it an octagon)
gd_feature_set.add_feature(feature, sigil="OCTO")

#Box with jagged edges (useful for showing breaks in contains)
gd_feature_set.add_feature(feature, sigil="JAGGY")

#Arrow which spans the axis with strand used only for direction
gd_feature_set.add_feature(feature, sigil="BIGARROW")
</pre><p>These are shown

below.


Most sigils fit into a bounding box (as given by the default BOX sigil),
either above or below the axis for the forward or reverse strand, or
straddling it (double the height) for strand-less features.
The BIGARROW sigil is different, always straddling the axis with the
direction taken from the feature&#X2019;s stand.</p><p>
<img src="images/GD_sigils.png" width=425, height=600>

</p>
<!--TOC subsection id="sec329" Arrow sigils-->
<h3 id="sec329" class="subsection">17.1.8&#XA0;&#XA0;Arrow sigils</h3><!--SEC END --><p>
<a id="sec:gd_arrow_sigils"></a></p><p>We introduced the arrow sigils in the previous section.
There are two additional options to adjust the shapes of the arrows, firstly
the thickness of the arrow shaft, given as a proportion of the height of the
bounding box:</p><pre class="verbatim">#Full height shafts, giving pointed boxes:
gd_feature_set.add_feature(feature, sigil="ARROW", color="brown",
                           arrowshaft_height=1.0)
#Or, thin shafts:                      
gd_feature_set.add_feature(feature, sigil="ARROW", color="teal",
                           arrowshaft_height=0.2)
#Or, very thin shafts:
gd_feature_set.add_feature(feature, sigil="ARROW", color="darkgreen",
                           arrowshaft_height=0.1)
</pre><p>
The results are shown below:</p><p><img src="images/GD_sigil_arrow_shafts.png" width=600, height=700></p><p>Secondly, the length of the arrow head - given as a proportion of the height
of the bounding box (defaulting to 0.5, or 50%):</p><pre class="verbatim">#Short arrow heads:
gd_feature_set.add_feature(feature, sigil="ARROW", color="blue",
                           arrowhead_length=0.25)
#Or, longer arrow heads:
gd_feature_set.add_feature(feature, sigil="ARROW", color="orange",
                           arrowhead_length=1)
#Or, very very long arrow heads (i.e. all head, no shaft, so triangles):
gd_feature_set.add_feature(feature, sigil="ARROW", color="red",
                           arrowhead_length=10000)
</pre><p>
The results are shown below:</p><p><img src="images/GD_sigil_arrow_heads.png" width=600, height=700></p><p>Biopython 1.61 adds a new <code>BIGARROW</code> sigil which always stradles
the axis, pointing left for the reverse strand or right otherwise:</p><pre class="verbatim">#A large arrow straddling the axis:
gd_feature_set.add_feature(feature, sigil="BIGARROW")
</pre><p>All the shaft and arrow head options shown above for the
<code>ARROW</code> sigil can be used for the <code>BIGARROW</code> sigil too.</p>
<!--TOC subsection id="sec330" A nice example-->
<h3 id="sec330" class="subsection">17.1.9&#XA0;&#XA0;A nice example</h3><!--SEC END --><p>
<a id="sec:gd_nice_example"></a></p><p>Now let&#X2019;s return to the pPCP1 plasmid from <span style="font-style:italic">Yersinia pestis biovar
Microtus</span>, and the top down approach used in Section&#XA0;<a href="#sec%3Agd_top_down">17.1.3</a>,
but take advantage of the sigil options we&#X2019;ve now discussed. This time
we&#X2019;ll use arrows for the genes, and overlay them with strand-less features
(as plain boxes) showing the position of some restriction digest sites.</p><pre class="verbatim">from reportlab.lib import colors
from reportlab.lib.units import cm
from Bio.Graphics import GenomeDiagram
from Bio import SeqIO
from Bio.SeqFeature import SeqFeature, FeatureLocation

record = SeqIO.read("NC_005816.gb", "genbank")

gd_diagram = GenomeDiagram.Diagram(record.id)
gd_track_for_features = gd_diagram.new_track(1, name="Annotated Features")
gd_feature_set = gd_track_for_features.new_set()

for feature in record.features:
    if feature.type != "gene":
        #Exclude this feature
        continue
    if len(gd_feature_set) % 2 == 0:
        color = colors.blue
    else:
        color = colors.lightblue
    gd_feature_set.add_feature(feature, sigil="ARROW",
                               color=color, label=True,
                               label_size = 14, label_angle=0)

#I want to include some strandless features, so for an example
#will use EcoRI recognition sites etc.
for site, name, color in [("GAATTC","EcoRI",colors.green),
                          ("CCCGGG","SmaI",colors.orange),
                          ("AAGCTT","HindIII",colors.red),
                          ("GGATCC","BamHI",colors.purple)]:
    index = 0
    while True:
        index  = record.seq.find(site, start=index)
        if index == -1 : break
        feature = SeqFeature(FeatureLocation(index, index+len(site)))
        gd_feature_set.add_feature(feature, color=color, name=name,
                                   label=True, label_size = 10,
                                   label_color=color)
        index += len(site)

gd_diagram.draw(format="linear", pagesize='A4', fragments=4,
                start=0, end=len(record))
gd_diagram.write("plasmid_linear_nice.pdf", "PDF")
gd_diagram.write("plasmid_linear_nice.eps", "EPS")
gd_diagram.write("plasmid_linear_nice.svg", "SVG")

gd_diagram.draw(format="circular", circular=True, pagesize=(20*cm,20*cm),
                start=0, end=len(record), circle_core = 0.5)
gd_diagram.write("plasmid_circular_nice.pdf", "PDF")
gd_diagram.write("plasmid_circular_nice.eps", "EPS")
gd_diagram.write("plasmid_circular_nice.svg", "SVG")
</pre><p>
And the output:</p><p><img src="images/plasmid_linear_nice.png" width=550, height=400></p><p><img src="images/plasmid_circular_nice.png" width=591, height=591></p>
<!--TOC subsection id="sec331" Multiple tracks-->
<h3 id="sec331" class="subsection">17.1.10&#XA0;&#XA0;Multiple tracks</h3><!--SEC END --><p>
<a id="sec:gd_multiple_tracks"></a></p><p>All the examples so far have used a single track, but you can have more than
one track &#X2013; for example show the genes on one, and repeat regions on another.
In this example we&#X2019;re going to show three phage genomes side by side to scale,
inspired by Figure 6 in Proux <span style="font-style:italic">e</span>t al. (2002) [<a href="#proux2002">5</a>].
We&#X2019;ll need the GenBank files for the following three phage:
</p><ul class="itemize"><li class="li-itemize">
<code>NC_002703</code> &#X2013; Lactococcus phage Tuc2009, complete genome (38347 bp)
</li><li class="li-itemize"><code>AF323668</code> &#X2013; Bacteriophage bIL285, complete genome (35538 bp)
</li><li class="li-itemize"><code>NC_003212</code> &#X2013; <span style="font-style:italic">Listeria innocua</span> Clip11262, complete genome,
of which we are focussing only on integrated prophage 5 (similar length).
</li></ul><p>You can download these using Entrez if you like, see Section&#XA0;<a href="#sec%3Aefetch">9.6</a>
for more details. For the third record we&#X2019;ve worked out where the phage is
integrated into the genome, and slice the record to extract it (with the
features preserved, see Section&#XA0;<a href="#sec%3ASeqRecord-slicing">4.6</a>), and must also
reverse complement to match the orientation of the first two phage (again
preserving the features, see Section&#XA0;<a href="#sec%3ASeqRecord-reverse-complement">4.8</a>):</p><pre class="verbatim">from Bio import SeqIO

A_rec = SeqIO.read("NC_002703.gbk", "gb")
B_rec = SeqIO.read("AF323668.gbk", "gb")
C_rec = SeqIO.read("NC_003212.gbk", "gb")[2587879:2625807].reverse_complement(name=True)
</pre><p>The figure we are imitating used different colors for different gene functions.
One way to do this is to edit the GenBank file to record color preferences for
each feature - something <a href="http://www.sanger.ac.uk/resources/software/artemis/">Sanger&#X2019;s Artemis editor</a> does, and which GenomeDiagram should understand. Here
however, we&#X2019;ll just hard code three lists of colors.</p><p>Note that the annotation in the GenBank files doesn&#X2019;t exactly match that shown
in Proux <span style="font-style:italic">et al.</span>, they have drawn some unannotated genes.</p><pre class="verbatim">from reportlab.lib.colors import red, grey, orange, green, brown, blue, lightblue, purple

A_colors = [red]*5 + [grey]*7 + [orange]*2 + [grey]*2 + [orange] + [grey]*11 + [green]*4 \
         + [grey] + [green]*2 + [grey, green] + [brown]*5 + [blue]*4 + [lightblue]*5 \
         + [grey, lightblue] + [purple]*2 + [grey]
B_colors = [red]*6 + [grey]*8 + [orange]*2 + [grey] + [orange] + [grey]*21 + [green]*5 \
         + [grey] + [brown]*4 + [blue]*3 + [lightblue]*3 + [grey]*5 + [purple]*2
C_colors = [grey]*30 + [green]*5 + [brown]*4 + [blue]*2 + [grey, blue] + [lightblue]*2 \
         + [grey]*5
</pre><p>Now to draw them &#X2013; this time we add three tracks to the diagram, and also notice they
are given different start/end values to reflect their different lengths (this requires
Biopython 1.59 or later).</p><pre class="verbatim">from Bio.Graphics import GenomeDiagram

name = "Proux Fig 6"
gd_diagram = GenomeDiagram.Diagram(name)
max_len = 0
for record, gene_colors in zip([A_rec, B_rec, C_rec], [A_colors, B_colors, C_colors]):
    max_len = max(max_len, len(record))
    gd_track_for_features = gd_diagram.new_track(1,
                            name=record.name,
                            greytrack=True,
                            start=0, end=len(record))
    gd_feature_set = gd_track_for_features.new_set()

    i = 0
    for feature in record.features:
        if feature.type != "gene":
            #Exclude this feature                                                                                                   
            continue
        gd_feature_set.add_feature(feature, sigil="ARROW",
                                   color=gene_colors[i], label=True,
                                   name = str(i+1),
                                   label_position="start",
                                   label_size = 6, label_angle=0)
        i+=1

gd_diagram.draw(format="linear", pagesize='A4', fragments=1,
                start=0, end=max_len)
gd_diagram.write(name + ".pdf", "PDF")
gd_diagram.write(name + ".eps", "EPS")
gd_diagram.write(name + ".svg", "SVG")
</pre><p>
The result:</p><p><img src="images/three_track_simple.png" width=565, height=400></p><p>

I did wonder why in the original manuscript there were no red or orange genes
marked in the bottom phage. Another important point is here the phage are
shown with different lengths - this is because they are all drawn to the same
scale (they <em>are</em> different lengths).</p><p>The key difference from the published figure is they have color-coded links
between similar proteins &#X2013; which is what we will do in the next section.</p>
<!--TOC subsection id="sec332" Cross-Links between tracks-->
<h3 id="sec332" class="subsection">17.1.11&#XA0;&#XA0;Cross-Links between tracks</h3><!--SEC END --><p>
<a id="sec:gd_cross_links"></a></p><p>Biopython 1.59 added the ability to draw cross links between tracks - both
simple linear diagrams as we will show here, but also linear diagrams split
into fragments and circular diagrams.</p><p>Continuing the example from the previous section inspired by Figure 6 from
Proux <span style="font-style:italic">et al.</span> 2002 [<a href="#proux2002">5</a>],
we would need a list of cross links between pairs of genes, along with a score
or color to use. Realistically you might extract this from a BLAST file
computationally, but here I have manually typed them in.</p><p>My naming convention continues to refer to the three phage as A, B and C.
Here are the links we want to show between A and B, given as a list of
tuples (percentage similarity score, gene in A, gene in B).</p><pre class="verbatim">#Tuc2009 (NC_002703) vs bIL285 (AF323668)
A_vs_B = [
    (99, "Tuc2009_01", "int"),
    (33, "Tuc2009_03", "orf4"),
    (94, "Tuc2009_05", "orf6"),
    (100,"Tuc2009_06", "orf7"),
    (97, "Tuc2009_07", "orf8"),
    (98, "Tuc2009_08", "orf9"),
    (98, "Tuc2009_09", "orf10"),
    (100,"Tuc2009_10", "orf12"),
    (100,"Tuc2009_11", "orf13"),
    (94, "Tuc2009_12", "orf14"),
    (87, "Tuc2009_13", "orf15"),
    (94, "Tuc2009_14", "orf16"),
    (94, "Tuc2009_15", "orf17"),
    (88, "Tuc2009_17", "rusA"),
    (91, "Tuc2009_18", "orf20"),
    (93, "Tuc2009_19", "orf22"),
    (71, "Tuc2009_20", "orf23"),
    (51, "Tuc2009_22", "orf27"),
    (97, "Tuc2009_23", "orf28"),
    (88, "Tuc2009_24", "orf29"),
    (26, "Tuc2009_26", "orf38"),
    (19, "Tuc2009_46", "orf52"),
    (77, "Tuc2009_48", "orf54"),
    (91, "Tuc2009_49", "orf55"),
    (95, "Tuc2009_52", "orf60"), 
]
</pre><p>Likewise for B and C:</p><pre class="verbatim">#bIL285 (AF323668) vs Listeria innocua prophage 5 (in NC_003212)
B_vs_C = [
    (42, "orf39", "lin2581"),
    (31, "orf40", "lin2580"),
    (49, "orf41", "lin2579"), #terL
    (54, "orf42", "lin2578"), #portal
    (55, "orf43", "lin2577"), #protease
    (33, "orf44", "lin2576"), #mhp
    (51, "orf46", "lin2575"),
    (33, "orf47", "lin2574"),
    (40, "orf48", "lin2573"),
    (25, "orf49", "lin2572"),
    (50, "orf50", "lin2571"),
    (48, "orf51", "lin2570"),
    (24, "orf52", "lin2568"),
    (30, "orf53", "lin2567"),
    (28, "orf54", "lin2566"),
]
</pre><p>For the first and last phage these identifiers are locus tags, for the middle
phage there are no locus tags so I&#X2019;ve used gene names instead. The following
little helper function lets us lookup a feature using either a locus tag or
gene name:</p><pre class="verbatim">def get_feature(features, id, tags=["locus_tag", "gene"]):
    """Search list of SeqFeature objects for an identifier under the given tags."""
    for f in features:
        for key in tags:
            #tag may not be present in this feature 
            for x in f.qualifiers.get(key, []):
                if x == id:
                     return f
    raise KeyError(id)
</pre><p>We can now turn those list of identifier pairs into SeqFeature pairs, and thus
find their location co-ordinates. We can now add all that code and the following
snippet to the previous example (just before the <code>gd_diagram.draw(...)</code>
line &#X2013; see the finished example script
<a href="http://biopython.org/SRC/biopython/Doc/examples/Proux_et_al_2002_Figure_6.py">Proux_et_al_2002_Figure_6.py</a>
included in the <span style="font-family:monospace">Doc/examples</span> folder of the Biopython source code)
to add cross links to the figure:</p><pre class="verbatim">from Bio.Graphics.GenomeDiagram import CrossLink
from reportlab.lib import colors
#Note it might have been clearer to assign the track numbers explicitly...                                                          
for rec_X, tn_X, rec_Y, tn_Y, X_vs_Y in [(A_rec, 3, B_rec, 2, A_vs_B),
                                         (B_rec, 2, C_rec, 1, B_vs_C)]:
    track_X = gd_diagram.tracks[tn_X]
    track_Y = gd_diagram.tracks[tn_Y]
    for score, id_X, id_Y in X_vs_Y:
        feature_X = get_feature(rec_X.features, id_X)
        feature_Y = get_feature(rec_Y.features, id_Y)
        color = colors.linearlyInterpolatedColor(colors.white, colors.firebrick, 0, 100, score)
        link_xy = CrossLink((track_X, feature_X.location.start, feature_X.location.end),
                            (track_Y, feature_Y.location.start, feature_Y.location.end),
                            color, colors.lightgrey)
        gd_diagram.cross_track_links.append(link_xy)
</pre><p>There are several important pieces to this code. First the <code>GenomeDiagram</code> object
has a <code>cross_track_links</code> attribute which is just a list of <code>CrossLink</code> objects.
Each <code>CrossLink</code> object takes two sets of track-specific co-ordinates (here given
as tuples, you can alternatively use a <code>GenomeDiagram.Feature</code> object instead).
You can optionally supply a colour, border color, and say if this link should be drawn
flipped (useful for showing inversions).</p><p>You can also see how we turn the BLAST percentage identity score into a colour,
interpolating between white (0%) and a dark red (100%). In this example
we don&#X2019;t have any problems with overlapping cross-links. One way to tackle that
is to use transparency in ReportLab, by using colors with their alpha channel set.
However, this kind of shaded color scheme combined with overlap transparency
would be difficult to interpret.

The result:</p><p><img src="images/three_track_cl.png" width=565, height=400></p><p>There is still a lot more that can be done within Biopython to help
improve this figure. First of all, the cross links in this case are
between proteins which are drawn in a strand specific manor. It can
help to add a background region (a feature using the &#X2018;BOX&#X2019; sigil) on the
feature track to extend the cross link. Also, we could reduce the vertical
height of the feature tracks to allocate more to the links instead &#X2013; one
way to do that is to allocate space for empty tracks. Furthermore,
in cases like this where there are no large gene overlaps, we can use
the axis-straddling <code>BIGARROW</code> sigil, which allows us to further
reduce the vertical space needed for the track. These improvements
are demonstrated in the example script
<a href="http://biopython.org/SRC/biopython/Doc/examples/Proux_et_al_2002_Figure_6.py">Proux_et_al_2002_Figure_6.py</a>
included in the <span style="font-family:monospace">Doc/examples</span> folder of the Biopython source code.

The result:</p><p><img src="images/three_track_cl2a.png" width=565, height=400></p><p>Beyond that, finishing touches you might want to do manually in a vector
image editor include fine tuning the placement of gene labels, and adding
other custom annotation such as highlighting particular regions.</p><p>Although not really necessary in this example since none of the cross-links
overlap, using a transparent color in ReportLab is a very useful technique
for superimposing multiple links. However, in this case a shaded color
scheme should be avoided.</p>
<!--TOC subsection id="sec333" Further options-->
<h3 id="sec333" class="subsection">17.1.12&#XA0;&#XA0;Further options</h3><!--SEC END --><p>You can control the tick marks to show the scale &#X2013; after all every graph
should show its units, and the number of the grey-track labels.</p><p>Also, we have only used the <code>FeatureSet</code> so far. GenomeDiagram also has
a <code>GraphSet</code> which can be used for show line graphs, bar charts and heat
plots (e.g. to show plots of GC% on a track parallel to the features).</p><p>These options are not covered here yet, so for now we refer you to the
<a href="http://biopython.org/DIST/docs/GenomeDiagram/userguide.pdf">User Guide (PDF)</a> included with the standalone version of GenomeDiagram (but
please read the next section first), and the docstrings.</p>
<!--TOC subsection id="sec334" Converting old code-->
<h3 id="sec334" class="subsection">17.1.13&#XA0;&#XA0;Converting old code</h3><!--SEC END --><p>If you have old code written using the standalone version of GenomeDiagram, and
you want to switch it over to using the new version included with Biopython then
you will have to make a few changes - most importantly to your import statements.</p><p>Also, the older version of GenomeDiagram used only the UK spellings of color and
center (colour and centre). You will need to change to the American spellings,
although for several years the Biopython version of GenomeDiagram supported both.</p><p>For example, if you used to have:
</p><pre class="verbatim">from GenomeDiagram import GDFeatureSet, GDDiagram
gdd = GDDiagram("An example")
...
</pre><p>you could just switch the import statements like this:
</p><pre class="verbatim">from Bio.Graphics.GenomeDiagram import FeatureSet as GDFeatureSet, Diagram as GDDiagram
gdd = GDDiagram("An example")
...
</pre><p>and hopefully that should be enough. In the long term you might want to
switch to the new names, but you would have to change more of your code:
</p><pre class="verbatim">from Bio.Graphics.GenomeDiagram import FeatureSet, Diagram
gdd = Diagram("An example")
...
</pre><p>or:
</p><pre class="verbatim">from Bio.Graphics import GenomeDiagram
gdd = GenomeDiagram.Diagram("An example")
...
</pre><p>If you run into difficulties, please ask on the Biopython mailing list for
advice. One catch is that we have not included the old module
<code>GenomeDiagram.GDUtilities</code> yet. This included a number of
GC% related functions, which will probably be merged under
<code>Bio.SeqUtils</code> later on.
</p>
<!--TOC section id="sec335" Chromosomes-->
<h2 id="sec335" class="section">17.2&#XA0;&#XA0;Chromosomes</h2><!--SEC END --><p>The <code>Bio.Graphics.BasicChromosome</code> module allows drawing of chromosomes.
There is an example in Jupe <span style="font-style:italic">et al.</span> (2012) [<a href="#jupe2012">6</a>]
(open access) using colors to highlight different gene families.</p>
<!--TOC subsection id="sec336" Simple Chromosomes-->
<h3 id="sec336" class="subsection">17.2.1&#XA0;&#XA0;Simple Chromosomes</h3><!--SEC END --><p>
Here is a very simple example - for which we&#X2019;ll use <span style="font-style:italic">Arabidopsis thaliana</span>.</p><p>You can skip this bit, but first I downloaded the five sequenced chromosomes
from the NCBI&#X2019;s FTP site
<a href="ftp://ftp.ncbi.nlm.nih.gov/genomes/Arabidopsis_thaliana"><span style="font-family:monospace">ftp://ftp.ncbi.nlm.nih.gov/genomes/Arabidopsis_thaliana</span></a> and then parsed
them with <code>Bio.SeqIO</code> to find out their lengths. You could use the
GenBank files for this, but it is faster to use the FASTA files for the
whole chromosomes:</p><pre class="verbatim">from Bio import SeqIO
entries = [("Chr I", "CHR_I/NC_003070.fna"),
           ("Chr II", "CHR_II/NC_003071.fna"),
           ("Chr III", "CHR_III/NC_003074.fna"),
           ("Chr IV", "CHR_IV/NC_003075.fna"),
           ("Chr V", "CHR_V/NC_003076.fna")]
for (name, filename) in entries:
   record = SeqIO.read(filename,"fasta")
   print(name, len(record))
</pre><p>This gave the lengths of the five chromosomes, which we&#X2019;ll now use in
the following short demonstration of the <code>BasicChromosome</code> module:</p><pre class="verbatim">from reportlab.lib.units import cm
from Bio.Graphics import BasicChromosome

entries = [("Chr I", 30432563),
           ("Chr II", 19705359),
           ("Chr III", 23470805),
           ("Chr IV", 18585042),
           ("Chr V", 26992728)]

max_len = 30432563 #Could compute this
telomere_length = 1000000 #For illustration
         
chr_diagram = BasicChromosome.Organism()
chr_diagram.page_size = (29.7*cm, 21*cm) #A4 landscape

for name, length in entries:
    cur_chromosome = BasicChromosome.Chromosome(name)
    #Set the scale to the MAXIMUM length plus the two telomeres in bp,
    #want the same scale used on all five chromosomes so they can be
    #compared to each other
    cur_chromosome.scale_num = max_len + 2 * telomere_length

    #Add an opening telomere
    start = BasicChromosome.TelomereSegment()
    start.scale = telomere_length
    cur_chromosome.add(start)

    #Add a body - using bp as the scale length here.
    body = BasicChromosome.ChromosomeSegment()
    body.scale = length
    cur_chromosome.add(body)

    #Add a closing telomere
    end = BasicChromosome.TelomereSegment(inverted=True)
    end.scale = telomere_length
    cur_chromosome.add(end)

    #This chromosome is done
    chr_diagram.add(cur_chromosome)

chr_diagram.draw("simple_chrom.pdf", "Arabidopsis thaliana")
</pre><p>This should create a very simple PDF file, shown

here:</p><p><img src="images/simple_chrom.png" width=650, height=460></p><p>

This example is deliberately short and sweet. The next example shows the
location of features of interest.</p>
<!--TOC subsection id="sec337" Annotated Chromosomes-->
<h3 id="sec337" class="subsection">17.2.2&#XA0;&#XA0;Annotated Chromosomes</h3><!--SEC END --><p>Continuing from the previous example, let&#X2019;s also show the tRNA genes.
We&#X2019;ll get their locations by parsing the GenBank files for the five
<span style="font-style:italic">Arabidopsis thaliana</span> chromosomes. You&#X2019;ll need to download these
files from the NCBI FTP site
<a href="ftp://ftp.ncbi.nlm.nih.gov/genomes/Arabidopsis_thaliana"><span style="font-family:monospace">ftp://ftp.ncbi.nlm.nih.gov/genomes/Arabidopsis_thaliana</span></a>,
and preserve the subdirectory names or edit the paths below:</p><pre class="verbatim">from reportlab.lib.units import cm
from Bio import SeqIO
from Bio.Graphics import BasicChromosome

entries = [("Chr I", "CHR_I/NC_003070.gbk"),
           ("Chr II", "CHR_II/NC_003071.gbk"),
           ("Chr III", "CHR_III/NC_003074.gbk"),
           ("Chr IV", "CHR_IV/NC_003075.gbk"),
           ("Chr V", "CHR_V/NC_003076.gbk")]

max_len = 30432563 #Could compute this
telomere_length = 1000000 #For illustration

chr_diagram = BasicChromosome.Organism()
chr_diagram.page_size = (29.7*cm, 21*cm) #A4 landscape

for index, (name, filename) in enumerate(entries):
    record = SeqIO.read(filename,"genbank")
    length = len(record)
    features = [f for f in record.features if f.type=="tRNA"]
    #Record an Artemis style integer color in the feature's qualifiers,
    #1 = Black, 2 = Red, 3 = Green, 4 = blue, 5 =cyan, 6 = purple 
    for f in features: f.qualifiers["color"] = [index+2]

    cur_chromosome = BasicChromosome.Chromosome(name)
    #Set the scale to the MAXIMUM length plus the two telomeres in bp,
    #want the same scale used on all five chromosomes so they can be
    #compared to each other
    cur_chromosome.scale_num = max_len + 2 * telomere_length

    #Add an opening telomere
    start = BasicChromosome.TelomereSegment()
    start.scale = telomere_length
    cur_chromosome.add(start)

    #Add a body - again using bp as the scale length here.
    body = BasicChromosome.AnnotatedChromosomeSegment(length, features)
    body.scale = length
    cur_chromosome.add(body)

    #Add a closing telomere
    end = BasicChromosome.TelomereSegment(inverted=True)
    end.scale = telomere_length
    cur_chromosome.add(end)

    #This chromosome is done
    chr_diagram.add(cur_chromosome)

chr_diagram.draw("tRNA_chrom.pdf", "Arabidopsis thaliana")
</pre><p>It might warn you about the labels being too close together - have a look
at the forward strand (right hand side) of Chr I, but it should create a
colorful PDF file, shown

here:</p><p><img src="images/tRNA_chrom.png" width=650, height=460></p>
<!--TOC chapter id="sec338" Cookbook &#X2013; Cool things to do with it-->
<h1 id="sec338" class="chapter">Chapter&#XA0;18&#XA0;&#XA0;Cookbook &#X2013; Cool things to do with it</h1><!--SEC END --><p>
<a id="chapter:cookbook"></a></p><p>Biopython now has two collections of &#X201C;cookbook&#X201D; examples &#X2013; this chapter
(which has been included in this tutorial for many years and has gradually
grown), and <a href="http://biopython.org/wiki/Category:Cookbook"><span style="font-family:monospace">http://biopython.org/wiki/Category:Cookbook</span></a> which is a
user contributed collection on our wiki.</p><p>We&#X2019;re trying to encourage Biopython users to contribute their own examples
to the wiki. In addition to helping the community, one direct benefit of
sharing an example like this is that you could also get some feedback on
the code from other Biopython users and developers - which could help you
improve all your Python code.</p><p>In the long term, we may end up moving all of the examples in this chapter
to the wiki, or elsewhere within the tutorial.</p>
<!--TOC section id="sec339" Working with sequence files-->
<h2 id="sec339" class="section">18.1&#XA0;&#XA0;Working with sequence files</h2><!--SEC END --><p>
<a id="seq:cookbook-sequences"></a></p><p>This section shows some more examples of sequence input/output, using the
<code>Bio.SeqIO</code> module described in Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>.</p>
<!--TOC subsection id="sec340" Filtering a sequence file-->
<h3 id="sec340" class="subsection">18.1.1&#XA0;&#XA0;Filtering a sequence file</h3><!--SEC END --><p>Often you&#X2019;ll have a large file with many sequences in it (e.g. FASTA file
or genes, or a FASTQ or SFF file of reads), a separate shorter list of
the IDs for a subset of sequences of interest, and want to make a new
sequence file for this subset.</p><p>Let&#X2019;s say the list of IDs is in a simple text file, as the first word on
each line. This could be a tabular file where the first column is the ID.
Try something like this:</p><pre class="verbatim">from Bio import SeqIO
input_file = "big_file.sff"
id_file = "short_list.txt"
output_file = "short_list.sff"
wanted = set(line.rstrip("\n").split(None,1)[0] for line in open(id_file))
print("Found %i unique identifiers in %s" % (len(wanted), id_file))
records = (r for r in SeqIO.parse(input_file, "sff") if r.id in wanted)
count = SeqIO.write(records, output_file, "sff")
print("Saved %i records from %s to %s" % (count, input_file, output_file))
if count &lt; len(wanted):
    print("Warning %i IDs not found in %s" % (len(wanted)-count, input_file))
</pre><p>Note that we use a Python <code>set</code> rather than a <code>list</code>, this makes
testing membership faster.</p>
<!--TOC subsection id="sec341" Producing randomised genomes-->
<h3 id="sec341" class="subsection">18.1.2&#XA0;&#XA0;Producing randomised genomes</h3><!--SEC END --><p>Let&#X2019;s suppose you are looking at genome sequence, hunting for some sequence
feature &#X2013; maybe extreme local GC% bias, or possible restriction digest sites.
Once you&#X2019;ve got your Python code working on the real genome it may be sensible
to try running the same search on randomised versions of the same genome for
statistical analysis (after all, any &#X201C;features&#X201D; you&#X2019;ve found could just be
there just by chance).</p><p>For this discussion, we&#X2019;ll use the GenBank file for the pPCP1 plasmid from
<span style="font-style:italic">Yersinia pestis biovar Microtus</span>. The file is included with the
Biopython unit tests under the GenBank folder, or you can get it from our
website, <a href="http://biopython.org/SRC/biopython/Tests/GenBank/NC_005816.gb"><span style="font-family:monospace">NC_005816.gb</span></a>. 
This file contains one and only one record, so we can read it in as a
<code>SeqRecord</code> using the <code>Bio.SeqIO.read()</code> function:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; original_rec = SeqIO.read("NC_005816.gb", "genbank")
</pre><p>So, how can we generate a shuffled versions of the original sequence? I would
use the built in Python <code>random</code> module for this, in particular the function
<code>random.shuffle</code> &#X2013; but this works on a Python list. Our sequence is a
<code>Seq</code> object, so in order to shuffle it we need to turn it into a list:</p><pre class="verbatim">&gt;&gt;&gt; import random
&gt;&gt;&gt; nuc_list = list(original_rec.seq)
&gt;&gt;&gt; random.shuffle(nuc_list) #acts in situ!
</pre><p>Now, in order to use <code>Bio.SeqIO</code> to output the shuffled sequence, we need
to construct a new <code>SeqRecord</code> with a new <code>Seq</code> object using this
shuffled list. In order to do this, we need to turn the list of nucleotides
(single letter strings) into a long string &#X2013; the standard Python way to do
this is with the string object&#X2019;s join method.</p><pre class="verbatim">&gt;&gt;&gt; from Bio.Seq import Seq
&gt;&gt;&gt; from Bio.SeqRecord import SeqRecord
&gt;&gt;&gt; shuffled_rec = SeqRecord(Seq("".join(nuc_list), original_rec.seq.alphabet),
...                          id="Shuffled", description="Based on %s" % original_rec.id)
</pre><p>Let&#X2019;s put all these pieces together to make a complete Python script which
generates a single FASTA file containing 30 randomly shuffled versions of
the original sequence.</p><p>This first version just uses a big for loop and writes out the records one by one
(using the <code>SeqRecord</code>&#X2019;s format method described in
Section&#XA0;<a href="#sec%3ABio.SeqIO-and-StringIO">5.5.4</a>):</p><pre class="verbatim">import random
from Bio.Seq import Seq
from Bio.SeqRecord import SeqRecord
from Bio import SeqIO

original_rec = SeqIO.read("NC_005816.gb","genbank")

handle = open("shuffled.fasta", "w")
for i in range(30):
    nuc_list = list(original_rec.seq)
    random.shuffle(nuc_list)
    shuffled_rec = SeqRecord(Seq("".join(nuc_list), original_rec.seq.alphabet), \
                             id="Shuffled%i" % (i+1), \
                             description="Based on %s" % original_rec.id)
    handle.write(shuffled_rec.format("fasta"))
handle.close()
</pre><p>Personally I prefer the following version using a function to shuffle the record
and a generator expression instead of the for loop:</p><pre class="verbatim">import random
from Bio.Seq import Seq
from Bio.SeqRecord import SeqRecord
from Bio import SeqIO

def make_shuffle_record(record, new_id):
    nuc_list = list(record.seq)
    random.shuffle(nuc_list)
    return SeqRecord(Seq("".join(nuc_list), record.seq.alphabet), \
           id=new_id, description="Based on %s" % original_rec.id)
   
original_rec = SeqIO.read("NC_005816.gb","genbank")
shuffled_recs = (make_shuffle_record(original_rec, "Shuffled%i" % (i+1)) \
                 for i in range(30))
handle = open("shuffled.fasta", "w")
SeqIO.write(shuffled_recs, handle, "fasta")
handle.close()
</pre>
<!--TOC subsection id="sec342" Translating a FASTA file of CDS entries-->
<h3 id="sec342" class="subsection">18.1.3&#XA0;&#XA0;Translating a FASTA file of CDS entries</h3><!--SEC END --><p>
<a id="sec:SeqIO-translate"></a>
Suppose you&#X2019;ve got an input file of CDS entries for some organism, and you
want to generate a new FASTA file containing their protein sequences. i.e.
Take each nucleotide sequence from the original file, and translate it.
Back in Section&#XA0;<a href="#sec%3Atranslation">3.9</a> we saw how to use the <code>Seq</code>
object&#X2019;s <code>translate method</code>, and the optional <code>cds</code> argument
which enables correct translation of alternative start codons.</p><p>We can combine this with <code>Bio.SeqIO</code> as
shown in the reverse complement example in Section&#XA0;<a href="#sec%3ASeqIO-reverse-complement">5.5.3</a>.
The key point is that for each nucleotide <code>SeqRecord</code>, we need to create
a protein <code>SeqRecord</code> - and take care of naming it.</p><p>You can write you own function to do this, choosing suitable protein identifiers
for your sequences, and the appropriate genetic code. In this example we just
use the default table and add a prefix to the identifier:</p><pre class="verbatim">from Bio.SeqRecord import SeqRecord
def make_protein_record(nuc_record):
    """Returns a new SeqRecord with the translated sequence (default table)."""
    return SeqRecord(seq = nuc_record.seq.translate(cds=True), \
                     id = "trans_" + nuc_record.id, \
                     description = "translation of CDS, using default table")
</pre><p>We can then use this function to turn the input nucleotide records into protein
records ready for output. An elegant way and memory efficient way to do this
is with a generator expression:</p><pre class="verbatim">from Bio import SeqIO
proteins = (make_protein_record(nuc_rec) for nuc_rec in \
            SeqIO.parse("coding_sequences.fasta", "fasta"))
SeqIO.write(proteins, "translations.fasta", "fasta")
</pre><p>This should work on any FASTA file of complete coding sequences.
If you are working on partial coding sequences, you may prefer to use
<code>nuc_record.seq.translate(to_stop=True)</code> in the example above, as
this wouldn&#X2019;t check for a valid start codon etc.</p>
<!--TOC subsection id="sec343" Making the sequences in a FASTA file upper case-->
<h3 id="sec343" class="subsection">18.1.4&#XA0;&#XA0;Making the sequences in a FASTA file upper case</h3><!--SEC END --><p>Often you&#X2019;ll get data from collaborators as FASTA files, and sometimes the
sequences can be in a mixture of upper and lower case. In some cases this is
deliberate (e.g. lower case for poor quality regions), but usually it is not
important. You may want to edit the file to make everything consistent (e.g.
all upper case), and you can do this easily using the <code>upper()</code> method
of the <code>SeqRecord</code> object (added in Biopython 1.55):</p><pre class="verbatim">from Bio import SeqIO
records = (rec.upper() for rec in SeqIO.parse("mixed.fas", "fasta"))
count = SeqIO.write(records, "upper.fas", "fasta")
print("Converted %i records to upper case" % count)
</pre><p>How does this work? The first line is just importing the <code>Bio.SeqIO</code>
module. The second line is the interesting bit &#X2013; this is a Python
generator expression which gives an upper case version of each record
parsed from the input file (<span style="font-family:monospace">mixed.fas</span>). In the third line we give
this generator expression to the <code>Bio.SeqIO.write()</code> function and it
saves the new upper cases records to our output file (<span style="font-family:monospace">upper.fas</span>).</p><p>The reason we use a generator expression (rather than a list or list
comprehension) is this means only one record is kept in memory at a time.
This can be really important if you are dealing with large files with
millions of entries.</p>
<!--TOC subsection id="sec344" Sorting a sequence file-->
<h3 id="sec344" class="subsection">18.1.5&#XA0;&#XA0;Sorting a sequence file</h3><!--SEC END --><p>
<a id="sec:SeqIO-sort"></a></p><p>Suppose you wanted to sort a sequence file by length (e.g. a set of
contigs from an assembly), and you are working with a file format like
FASTA or FASTQ which <code>Bio.SeqIO</code> can read, write (and index).</p><p>If the file is small enough, you can load it all into memory at once
as a list of <code>SeqRecord</code> objects, sort the list, and save it:</p><pre class="verbatim">from Bio import SeqIO
records = list(SeqIO.parse("ls_orchid.fasta","fasta"))
records.sort(cmp=lambda x,y: cmp(len(x),len(y)))
SeqIO.write(records, "sorted_orchids.fasta", "fasta")
</pre><p>The only clever bit is specifying a comparison function for how to
sort the records (here we sort them by length). If you wanted the
longest records first, you could flip the comparison or use the
reverse argument:</p><pre class="verbatim">from Bio import SeqIO
records = list(SeqIO.parse("ls_orchid.fasta","fasta"))
records.sort(cmp=lambda x,y: cmp(len(y),len(x)))
SeqIO.write(records, "sorted_orchids.fasta", "fasta")
</pre><p>Now that&#X2019;s pretty straight forward - but what happens if you have a
very large file and you can&#X2019;t load it all into memory like this?
For example, you might have some next-generation sequencing reads
to sort by length. This can be solved using the
<code>Bio.SeqIO.index()</code> function.</p><pre class="verbatim">from Bio import SeqIO
#Get the lengths and ids, and sort on length         
len_and_ids = sorted((len(rec), rec.id) for rec in \
                     SeqIO.parse("ls_orchid.fasta","fasta"))
ids = reversed([id for (length, id) in len_and_ids])
del len_and_ids #free this memory
record_index = SeqIO.index("ls_orchid.fasta", "fasta")
records = (record_index[id] for id in ids)
SeqIO.write(records, "sorted.fasta", "fasta")
</pre><p>First we scan through the file once using <code>Bio.SeqIO.parse()</code>,
recording the record identifiers and their lengths in a list of tuples.
We then sort this list to get them in length order, and discard the lengths.
Using this sorted list of identifiers <code>Bio.SeqIO.index()</code> allows us to
retrieve the records one by one, and we pass them to <code>Bio.SeqIO.write()</code>
for output.</p><p>These examples all use <code>Bio.SeqIO</code> to parse the records into
<code>SeqRecord</code> objects which are output using <code>Bio.SeqIO.write()</code>.
What if you want to sort a file format which <code>Bio.SeqIO.write()</code> doesn&#X2019;t
support, like the plain text SwissProt format? Here is an alternative
solution using the <code>get_raw()</code> method added to <code>Bio.SeqIO.index()</code>
in Biopython 1.54 (see Section&#XA0;<a href="#sec%3Aseqio-index-getraw">5.4.2.2</a>).</p><pre class="verbatim">from Bio import SeqIO
#Get the lengths and ids, and sort on length         
len_and_ids = sorted((len(rec), rec.id) for rec in \
                     SeqIO.parse("ls_orchid.fasta","fasta"))
ids = reversed([id for (length, id) in len_and_ids])
del len_and_ids #free this memory
record_index = SeqIO.index("ls_orchid.fasta", "fasta")
handle = open("sorted.fasta", "w")
for id in ids:
    handle.write(record_index.get_raw(id))
handle.close()
</pre><p>As a bonus, because it doesn&#X2019;t parse the data into <code>SeqRecord</code> objects
a second time it should be faster.</p>
<!--TOC subsection id="sec345" Simple quality filtering for FASTQ files-->
<h3 id="sec345" class="subsection">18.1.6&#XA0;&#XA0;Simple quality filtering for FASTQ files</h3><!--SEC END --><p>
<a id="sec:FASTQ-filtering-example"></a></p><p>The FASTQ file format was introduced at Sanger and is now widely used for
holding nucleotide sequencing reads together with their quality scores.
FASTQ files (and the related QUAL files) are an excellent example of
per-letter-annotation, because for each nucleotide in the sequence there is
an associated quality score. Any per-letter-annotation is held in a
<code>SeqRecord</code> in the <code>letter_annotations</code> dictionary as a list,
tuple or string (with the same number of elements as the sequence length).</p><p>One common task is taking a large set of sequencing reads and filtering them
(or cropping them) based on their quality scores.
The following example is very simplistic, but should illustrate the basics of
working with quality data in a <code>SeqRecord</code> object. All we are going to
do here is read in a file of FASTQ data, and filter it to pick out only those
records whose PHRED quality scores are all above some threshold (here 20).</p><p>For this example we&#X2019;ll use some real data downloaded from the ENA sequence
read archive,
<a href="ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR020/SRR020192/SRR020192.fastq.gz"><span style="font-family:monospace">ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR020/SRR020192/SRR020192.fastq.gz</span></a>
(2MB) which unzips to a 19MB file <span style="font-family:monospace">SRR020192.fastq</span>. This is some
Roche 454 GS FLX single end data from virus infected California sea lions
(see <a href="http://www.ebi.ac.uk/ena/data/view/SRS004476"><span style="font-family:monospace">http://www.ebi.ac.uk/ena/data/view/SRS004476</span></a> for details).</p><p>First, let&#X2019;s count the reads:</p><pre class="verbatim">from Bio import SeqIO
count = 0
for rec in SeqIO.parse("SRR020192.fastq", "fastq"):
    count += 1
print("%i reads" % count)
</pre><p>Now let&#X2019;s do a simple filtering for a minimum PHRED quality of 20:</p><pre class="verbatim">from Bio import SeqIO
good_reads = (rec for rec in \
              SeqIO.parse("SRR020192.fastq", "fastq") \
              if min(rec.letter_annotations["phred_quality"]) &gt;= 20)
count = SeqIO.write(good_reads, "good_quality.fastq", "fastq")
print("Saved %i reads" % count)
</pre><p>This pulled out only 14580 reads out of the 41892 present.
A more sensible thing to do would be to quality trim the reads, but this
is intended as an example only.</p><p>FASTQ files can contain millions of entries, so it is best to avoid loading
them all into memory at once. This example uses a generator expression, which
means only one <code>SeqRecord</code> is created at a time - avoiding any memory
limitations.</p>
<!--TOC subsection id="sec346" Trimming off primer sequences-->
<h3 id="sec346" class="subsection">18.1.7&#XA0;&#XA0;Trimming off primer sequences</h3><!--SEC END --><p>
<a id="sec:FASTQ-slicing-off-primer"></a></p><p>For this example we&#X2019;re going to pretend that <span style="font-family:monospace">GATGACGGTGT</span> is a 5&#X2019; primer
sequence we want to look for in some FASTQ formatted read data. As in the example
above, we&#X2019;ll use the <span style="font-family:monospace">SRR020192.fastq</span> file downloaded from the ENA
(<a href="ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR020/SRR020192/SRR020192.fastq.gz"><span style="font-family:monospace">ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR020/SRR020192/SRR020192.fastq.gz</span></a>).
The same approach would work with any other supported file format (e.g. FASTA files).</p><p>This code uses <code>Bio.SeqIO</code> with a generator expression (to avoid loading
all the sequences into memory at once), and the <code>Seq</code> object&#X2019;s
<code>startswith</code> method to see if the read starts with the primer sequence:</p><pre class="verbatim">from Bio import SeqIO
primer_reads = (rec for rec in \
                SeqIO.parse("SRR020192.fastq", "fastq") \
                if rec.seq.startswith("GATGACGGTGT"))
count = SeqIO.write(primer_reads, "with_primer.fastq", "fastq")
print("Saved %i reads" % count)
</pre><p>That should find 13819 reads from <span style="font-family:monospace">SRR014849.fastq</span> and save them to
a new FASTQ file, <span style="font-family:monospace">with_primer.fastq</span>.</p><p>Now suppose that instead you wanted to make a FASTQ file containing these reads
but with the primer sequence removed? That&#X2019;s just a small change as we can slice the
<code>SeqRecord</code> (see Section&#XA0;<a href="#sec%3ASeqRecord-slicing">4.6</a>) to remove the first eleven
letters (the length of our primer):</p><pre class="verbatim">from Bio import SeqIO
trimmed_primer_reads = (rec[11:] for rec in \
                        SeqIO.parse("SRR020192.fastq", "fastq") \
                        if rec.seq.startswith("GATGACGGTGT"))
count = SeqIO.write(trimmed_primer_reads, "with_primer_trimmed.fastq", "fastq")
print("Saved %i reads" % count)
</pre><p>Again, that should pull out the 13819 reads from <span style="font-family:monospace">SRR020192.fastq</span>,
but this time strip off the first ten characters, and save them to another new
FASTQ file, <span style="font-family:monospace">with_primer_trimmed.fastq</span>.</p><p>Finally, suppose you want to create a new FASTQ file where these reads have
their primer removed, but all the other reads are kept as they were?
If we want to still use a generator expression, it is probably clearest to
define our own trim function:</p><pre class="verbatim">from Bio import SeqIO
def trim_primer(record, primer):
    if record.seq.startswith(primer):
        return record[len(primer):]
    else:
        return record

trimmed_reads = (trim_primer(record, "GATGACGGTGT") for record in \
                 SeqIO.parse("SRR020192.fastq", "fastq"))
count = SeqIO.write(trimmed_reads, "trimmed.fastq", "fastq")
print("Saved %i reads" % count)
</pre><p>This takes longer, as this time the output file contains all 41892 reads.
Again, we&#X2019;re used a generator expression to avoid any memory problems.
You could alternatively use a generator function rather than a generator
expression.</p><pre class="verbatim">from Bio import SeqIO
def trim_primers(records, primer):
    """Removes perfect primer sequences at start of reads.
    
    This is a generator function, the records argument should
    be a list or iterator returning SeqRecord objects.
    """
    len_primer = len(primer) #cache this for later
    for record in records:
        if record.seq.startswith(primer):
            yield record[len_primer:]
        else:
            yield record

original_reads = SeqIO.parse("SRR020192.fastq", "fastq")
trimmed_reads = trim_primers(original_reads, "GATGACGGTGT")
count = SeqIO.write(trimmed_reads, "trimmed.fastq", "fastq") 
print("Saved %i reads" % count)
</pre><p>This form is more flexible if you want to do something more complicated
where only some of the records are retained &#X2013; as shown in the next example.</p>
<!--TOC subsection id="sec347" Trimming off adaptor sequences-->
<h3 id="sec347" class="subsection">18.1.8&#XA0;&#XA0;Trimming off adaptor sequences</h3><!--SEC END --><p>
<a id="sec:FASTQ-slicing-off-adaptor"></a></p><p>This is essentially a simple extension to the previous example. We are going
to going to pretend <span style="font-family:monospace">GATGACGGTGT</span> is an adaptor sequence in some FASTQ
formatted read data, again the <span style="font-family:monospace">SRR020192.fastq</span> file from the NCBI
(<a href="ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR020/SRR020192/SRR020192.fastq.gz"><span style="font-family:monospace">ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR020/SRR020192/SRR020192.fastq.gz</span></a>).</p><p>This time however, we will look for the sequence <em>anywhere</em> in the reads,
not just at the very beginning:</p><pre class="verbatim">from Bio import SeqIO

def trim_adaptors(records, adaptor):
    """Trims perfect adaptor sequences.
    
    This is a generator function, the records argument should
    be a list or iterator returning SeqRecord objects.
    """
    len_adaptor = len(adaptor) #cache this for later
    for record in records:
        index = record.seq.find(adaptor)
        if index == -1:
            #adaptor not found, so won't trim
            yield record
        else:
            #trim off the adaptor
            yield record[index+len_adaptor:]

original_reads = SeqIO.parse("SRR020192.fastq", "fastq")
trimmed_reads = trim_adaptors(original_reads, "GATGACGGTGT")
count = SeqIO.write(trimmed_reads, "trimmed.fastq", "fastq") 
print("Saved %i reads" % count)
</pre><p>Because we are using a FASTQ input file in this example, the <code>SeqRecord</code>
objects have per-letter-annotation for the quality scores. By slicing the
<code>SeqRecord</code> object the appropriate scores are used on the trimmed
records, so we can output them as a FASTQ file too.</p><p>Compared to the output of the previous example where we only looked for
a primer/adaptor at the start of each read, you may find some of the
trimmed reads are quite short after trimming (e.g. if the adaptor was
found in the middle rather than near the start). So, let&#X2019;s add a minimum
length requirement as well:</p><pre class="verbatim">from Bio import SeqIO

def trim_adaptors(records, adaptor, min_len):
    """Trims perfect adaptor sequences, checks read length.
    
    This is a generator function, the records argument should
    be a list or iterator returning SeqRecord objects.
    """
    len_adaptor = len(adaptor) #cache this for later
    for record in records:
        len_record = len(record) #cache this for later
        if len(record) &lt; min_len:
           #Too short to keep
           continue
        index = record.seq.find(adaptor)
        if index == -1:
            #adaptor not found, so won't trim
            yield record
        elif len_record - index - len_adaptor &gt;= min_len:
            #after trimming this will still be long enough
            yield record[index+len_adaptor:]

original_reads = SeqIO.parse("SRR020192.fastq", "fastq")
trimmed_reads = trim_adaptors(original_reads, "GATGACGGTGT", 100)
count = SeqIO.write(trimmed_reads, "trimmed.fastq", "fastq") 
print("Saved %i reads" % count)
</pre><p>By changing the format names, you could apply this to FASTA files instead.
This code also could be extended to do a fuzzy match instead of an exact
match (maybe using a pairwise alignment, or taking into account the read
quality scores), but that will be much slower.</p>
<!--TOC subsection id="sec348" Converting FASTQ files-->
<h3 id="sec348" class="subsection">18.1.9&#XA0;&#XA0;Converting FASTQ files</h3><!--SEC END --><p>
<a id="sec:SeqIO-fastq-conversion"></a></p><p>Back in Section&#XA0;<a href="#sec%3ASeqIO-conversion">5.5.2</a> we showed how to use
<code>Bio.SeqIO</code> to convert between two file formats. Here we&#X2019;ll go into a
little more detail regarding FASTQ files which are used in second generation
DNA sequencing. Please refer to Cock <span style="font-style:italic">et al.</span> (2009) [<a href="#cock2010">7</a>]
for a longer description. FASTQ files store both the DNA sequence (as a string)
and the associated read qualities. </p><p>PHRED scores (used in most FASTQ files, and also in QUAL files, ACE files
and SFF files) have become a <span style="font-style:italic">de facto</span> standard for representing
the probability of a sequencing error (here denoted by <span style="font-style:italic">P</span><sub><span style="font-style:italic">e</span></sub>) at a given
base using a simple base ten log transformation:</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell">
<span style="font-style:italic">Q</span><sub>PHRED</sub>&#XA0;=&#XA0;&#X2212;&#XA0;10&#XA0;&#XD7;&#XA0;log<sub>10</sub>&#XA0;(&#XA0;<span style="font-style:italic">P</span><sub><span style="font-style:italic">e</span></sub>&#XA0;)
&#XA0;&#XA0;&#XA0;&#XA0;(18.1)</td></tr>
</table><p>This means a wrong read (<span style="font-style:italic">P</span><sub><span style="font-style:italic">e</span></sub> = 1) gets a PHRED quality of 0, while a very
good read like <span style="font-style:italic">P</span><sub><span style="font-style:italic">e</span></sub> = 0.00001 gets a PHRED quality of 50. While for raw
sequencing data qualities higher than this are rare, with post processing
such as read mapping or assembly, qualities of up to about 90 are possible
(indeed, the MAQ tool allows for PHRED scores in the range 0 to 93 inclusive).</p><p>The FASTQ format has the potential to become a <span style="font-style:italic">de facto</span> standard for
storing the letters and quality scores for a sequencing read in a single plain
text file. The only fly in the ointment is that there are at least three
versions of the FASTQ format which are incompatible and difficult to
distinguish...</p><ol class="enumerate" type=1><li class="li-enumerate">
The original Sanger FASTQ format uses PHRED qualities encoded with an
ASCII offset of 33. The NCBI are using this format in their Short Read
Archive. We call this the <span style="font-family:monospace">fastq</span> (or <span style="font-family:monospace">fastq-sanger</span>) format
in <code>Bio.SeqIO</code>.
</li><li class="li-enumerate">Solexa (later bought by Illumina) introduced their own version using
Solexa qualities encoded with an ASCII offset of 64. We call this the
<span style="font-family:monospace">fastq-solexa</span> format.
</li><li class="li-enumerate">Illumina pipeline 1.3 onwards produces FASTQ files with PHRED qualities
(which is more consistent), but encoded with an ASCII offset of 64. We call
this the <span style="font-family:monospace">fastq-illumina</span> format.
</li></ol><p>The Solexa quality scores are defined using a different log transformation:</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell">
<span style="font-style:italic">Q</span><sub>Solexa</sub>&#XA0;=&#XA0;&#X2212;&#XA0;10&#XA0;&#XD7;&#XA0;log<sub>10</sub>&#XA0;</td><td class="dcell">&#X239B;<br>
&#X239C;<br>
&#X239C;<br>
&#X239D;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">P</span><sub><span style="font-style:italic">e</span></sub></td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center">1&#X2212;<span style="font-style:italic">P</span><sub><span style="font-style:italic">e</span></sub></td></tr>
</table></td><td class="dcell">&#XA0;</td><td class="dcell">&#X239E;<br>
&#X239F;<br>
&#X239F;<br>
&#X23A0;</td><td class="dcell">
&#XA0;&#XA0;&#XA0;&#XA0;(18.2)</td></tr>
</table><p>Given Solexa/Illumina have now moved to using PHRED scores in version 1.3 of
their pipeline, the Solexa quality scores will gradually fall out of use.
If you equate the error estimates (<span style="font-style:italic">P</span><sub><span style="font-style:italic">e</span></sub>) these two equations allow conversion
between the two scoring systems - and Biopython includes functions to do this
in the <code>Bio.SeqIO.QualityIO</code> module, which are called if you use
<code>Bio.SeqIO</code> to convert an old Solexa/Illumina file into a standard Sanger
FASTQ file:</p><pre class="verbatim">from Bio import SeqIO
SeqIO.convert("solexa.fastq", "fastq-solexa", "standard.fastq", "fastq")
</pre><p>If you want to convert a new Illumina 1.3+ FASTQ file, all that gets changed
is the ASCII offset because although encoded differently the scores are all
PHRED qualities:</p><pre class="verbatim">from Bio import SeqIO
SeqIO.convert("illumina.fastq", "fastq-illumina", "standard.fastq", "fastq")
</pre><p>Note that using <code>Bio.SeqIO.convert()</code> like this is <em>much</em> faster
than combining <code>Bio.SeqIO.parse()</code> and <code>Bio.SeqIO.write()</code>
because optimised code is used for converting between FASTQ variants
(and also for FASTQ to FASTA conversion).</p><p>For good quality reads, PHRED and Solexa scores are approximately equal,
which means since both the <span style="font-family:monospace">fasta-solexa</span> and <span style="font-family:monospace">fastq-illumina</span>
formats use an ASCII offset of 64 the files are almost the same. This was a
deliberate design choice by Illumina, meaning applications expecting the old
<span style="font-family:monospace">fasta-solexa</span> style files will probably be OK using the newer
<span style="font-family:monospace">fastq-illumina</span> files (on good data). Of course, both variants are
very different from the original FASTQ standard as used by Sanger,
the NCBI, and elsewhere (format name <span style="font-family:monospace">fastq</span> or <span style="font-family:monospace">fastq-sanger</span>).</p><p>For more details, see the built in help (also <a href="http://www.biopython.org/DIST/docs/api/Bio.SeqIO.QualityIO-module.html">online</a>):</p><pre class="verbatim">&gt;&gt;&gt; from Bio.SeqIO import QualityIO
&gt;&gt;&gt; help(QualityIO)
...
</pre>
<!--TOC subsection id="sec349" Converting FASTA and QUAL files into FASTQ files-->
<h3 id="sec349" class="subsection">18.1.10&#XA0;&#XA0;Converting FASTA and QUAL files into FASTQ files</h3><!--SEC END --><p>
<a id="sec:SeqIO-fasta-qual-conversion"></a></p><p>FASTQ files hold <em>both</em> sequences and their quality strings.
FASTA files hold <em>just</em> sequences, while QUAL files hold <em>just</em>
the qualities. Therefore a single FASTQ file can be converted to or from
<em>paired</em> FASTA and QUAL files.</p><p>Going from FASTQ to FASTA is easy:</p><pre class="verbatim">from Bio import SeqIO
SeqIO.convert("example.fastq", "fastq", "example.fasta", "fasta")
</pre><p>Going from FASTQ to QUAL is also easy:</p><pre class="verbatim">from Bio import SeqIO
SeqIO.convert("example.fastq", "fastq", "example.qual", "qual")
</pre><p>However, the reverse is a little more tricky. You can use <code>Bio.SeqIO.parse()</code>
to iterate over the records in a <em>single</em> file, but in this case we have
two input files. There are several strategies possible, but assuming that the
two files are really paired the most memory efficient way is to loop over both
together. The code is a little fiddly, so we provide a function called
<code>PairedFastaQualIterator</code> in the <code>Bio.SeqIO.QualityIO</code> module to do
this. This takes two handles (the FASTA file and the QUAL file) and returns
a <code>SeqRecord</code> iterator:</p><pre class="verbatim">from Bio.SeqIO.QualityIO import PairedFastaQualIterator
for record in PairedFastaQualIterator(open("example.fasta"), open("example.qual")):
   print(record)
</pre><p>This function will check that the FASTA and QUAL files are consistent (e.g.
the records are in the same order, and have the same sequence length).
You can combine this with the <code>Bio.SeqIO.write()</code> function to convert a
pair of FASTA and QUAL files into a single FASTQ files:</p><pre class="verbatim">from Bio import SeqIO
from Bio.SeqIO.QualityIO import PairedFastaQualIterator
handle = open("temp.fastq", "w") #w=write
records = PairedFastaQualIterator(open("example.fasta"), open("example.qual"))
count = SeqIO.write(records, handle, "fastq")
handle.close()
print("Converted %i records" % count)
</pre>
<!--TOC subsection id="sec350" Indexing a FASTQ file-->
<h3 id="sec350" class="subsection">18.1.11&#XA0;&#XA0;Indexing a FASTQ file</h3><!--SEC END --><p>
<a id="sec:fastq-indexing"></a></p><p>FASTQ files are often very large, with millions of reads in them. Due to the
sheer amount of data, you can&#X2019;t load all the records into memory at once.
This is why the examples above (filtering and trimming) iterate over the file
looking at just one <code>SeqRecord</code> at a time.</p><p>However, sometimes you can&#X2019;t use a big loop or an iterator - you may need
random access to the reads. Here the <code>Bio.SeqIO.index()</code> function
may prove very helpful, as it allows you to access any read in the FASTQ file
by its name (see Section&#XA0;<a href="#sec%3ASeqIO-index">5.4.2</a>).</p><p>Again we&#X2019;ll use the <span style="font-family:monospace">SRR020192.fastq</span> file from the ENA
(<a href="ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR020/SRR020192/SRR020192.fastq.gz"><span style="font-family:monospace">ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR020/SRR020192/SRR020192.fastq.gz</span></a>),
although this is actually quite a small FASTQ file with less than 50,000 reads:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; fq_dict = SeqIO.index("SRR020192.fastq", "fastq")
&gt;&gt;&gt; len(fq_dict)
41892
&gt;&gt;&gt; fq_dict.keys()[:4]
['SRR020192.38240', 'SRR020192.23181', 'SRR020192.40568', 'SRR020192.23186']
&gt;&gt;&gt; fq_dict["SRR020192.23186"].seq
Seq('GTCCCAGTATTCGGATTTGTCTGCCAAAACAATGAAATTGACACAGTTTACAAC...CCG', SingleLetterAlphabet())
</pre><p>When testing this on a FASTQ file with seven million reads,
indexing took about a minute, but record access was almost instant.</p><p>The example in Section&#XA0;<a href="#sec%3ASeqIO-sort">18.1.5</a> show how you can use the
<code>Bio.SeqIO.index()</code> function to sort a large FASTA file &#X2013; this
could also be used on FASTQ files.</p>
<!--TOC subsection id="sec351" Converting SFF files-->
<h3 id="sec351" class="subsection">18.1.12&#XA0;&#XA0;Converting SFF files</h3><!--SEC END --><p>
<a id="sec:SeqIO-sff-conversion"></a></p><p>If you work with 454 (Roche) sequence data, you will probably have access
to the raw data as a Standard Flowgram Format (SFF) file. This contains
the sequence reads (called bases) with quality scores and the original
flow information.</p><p>A common task is to convert from SFF to a pair of FASTA and QUAL files,
or to a single FASTQ file. These operations are trivial using the
<code>Bio.SeqIO.convert()</code> function (see Section&#XA0;<a href="#sec%3ASeqIO-conversion">5.5.2</a>):</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; SeqIO.convert("E3MFGYR02_random_10_reads.sff", "sff", "reads.fasta", "fasta")
10
&gt;&gt;&gt; SeqIO.convert("E3MFGYR02_random_10_reads.sff", "sff", "reads.qual", "qual")
10
&gt;&gt;&gt; SeqIO.convert("E3MFGYR02_random_10_reads.sff", "sff", "reads.fastq", "fastq")
10
</pre><p>Remember the convert function returns the number of records, in
this example just ten. This will give you the <em>untrimmed</em> reads, where
the leading and trailing poor quality sequence or adaptor will be in lower
case. If you want the <em>trimmed</em> reads (using the clipping information
recorded within the SFF file) use this:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; SeqIO.convert("E3MFGYR02_random_10_reads.sff", "sff-trim", "trimmed.fasta", "fasta")
10
&gt;&gt;&gt; SeqIO.convert("E3MFGYR02_random_10_reads.sff", "sff-trim", "trimmed.qual", "qual")
10
&gt;&gt;&gt; SeqIO.convert("E3MFGYR02_random_10_reads.sff", "sff-trim", "trimmed.fastq", "fastq")
10
</pre><p>If you run Linux, you could ask Roche for a copy of their &#X201C;off instrument&#X201D;
tools (often referred to as the Newbler tools). This offers an alternative way to
do SFF to FASTA or QUAL conversion at the command line (but currently FASTQ output
is not supported), e.g.</p><pre class="verbatim">$ sffinfo -seq -notrim E3MFGYR02_random_10_reads.sff &gt; reads.fasta
$ sffinfo -qual -notrim E3MFGYR02_random_10_reads.sff &gt; reads.qual
$ sffinfo -seq -trim E3MFGYR02_random_10_reads.sff &gt; trimmed.fasta
$ sffinfo -qual -trim E3MFGYR02_random_10_reads.sff &gt; trimmed.qual
</pre><p>The way Biopython uses mixed case sequence strings to represent
the trimming points deliberately mimics what the Roche tools do.</p><p>For more information on the Biopython SFF support, consult the built in help:</p><pre class="verbatim">&gt;&gt;&gt; from Bio.SeqIO import SffIO
&gt;&gt;&gt; help(SffIO)
...
</pre>
<!--TOC subsection id="sec352" Identifying open reading frames-->
<h3 id="sec352" class="subsection">18.1.13&#XA0;&#XA0;Identifying open reading frames</h3><!--SEC END --><p>A very simplistic first step at identifying possible genes is to look for
open reading frames (ORFs). By this we mean look in all six frames for long
regions without stop codons &#X2013; an ORF is just a region of nucleotides with
no in frame stop codons.</p><p>Of course, to find a gene you would also need to worry about locating a start
codon, possible promoters &#X2013; and in Eukaryotes there are introns to worry about
too. However, this approach is still useful in viruses and Prokaryotes.</p><p>To show how you might approach this with Biopython, we&#X2019;ll need a sequence to
search, and as an example we&#X2019;ll again use the bacterial plasmid &#X2013; although
this time we&#X2019;ll start with a plain FASTA file with no pre-marked genes:
<a href="http://biopython.org/SRC/biopython/Tests/GenBank/NC_005816.fna"><span style="font-family:monospace">NC_005816.fna</span></a>. This is a bacterial sequence, so we&#X2019;ll want to use
NCBI codon table 11 (see Section&#XA0;<a href="#sec%3Atranslation">3.9</a> about translation).</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO 
&gt;&gt;&gt; record = SeqIO.read("NC_005816.fna", "fasta")
&gt;&gt;&gt; table = 11
&gt;&gt;&gt; min_pro_len = 100
</pre><p>Here is a neat trick using the <code>Seq</code> object&#X2019;s <code>split</code> method to
get a list of all the possible ORF translations in the six reading frames:</p><pre class="verbatim">&gt;&gt;&gt; for strand, nuc in [(+1, record.seq), (-1, record.seq.reverse_complement())]:
...     for frame in range(3):
...         length = 3 * ((len(record)-frame) // 3) #Multiple of three
...         for pro in nuc[frame:frame+length].translate(table).split("*"):
...             if len(pro) &gt;= min_pro_len:
...                 print("%s...%s - length %i, strand %i, frame %i" \
...                       % (pro[:30], pro[-3:], len(pro), strand, frame))
GCLMKKSSIVATIITILSGSANAASSQLIP...YRF - length 315, strand 1, frame 0
KSGELRQTPPASSTLHLRLILQRSGVMMEL...NPE - length 285, strand 1, frame 1
GLNCSFFSICNWKFIDYINRLFQIIYLCKN...YYH - length 176, strand 1, frame 1
VKKILYIKALFLCTVIKLRRFIFSVNNMKF...DLP - length 165, strand 1, frame 1
NQIQGVICSPDSGEFMVTFETVMEIKILHK...GVA - length 355, strand 1, frame 2
RRKEHVSKKRRPQKRPRRRRFFHRLRPPDE...PTR - length 128, strand 1, frame 2
TGKQNSCQMSAIWQLRQNTATKTRQNRARI...AIK - length 100, strand 1, frame 2
QGSGYAFPHASILSGIAMSHFYFLVLHAVK...CSD - length 114, strand -1, frame 0
IYSTSEHTGEQVMRTLDEVIASRSPESQTR...FHV - length 111, strand -1, frame 0
WGKLQVIGLSMWMVLFSQRFDDWLNEQEDA...ESK - length 125, strand -1, frame 1
RGIFMSDTMVVNGSGGVPAFLFSGSTLSSY...LLK - length 361, strand -1, frame 1
WDVKTVTGVLHHPFHLTFSLCPEGATQSGR...VKR - length 111, strand -1, frame 1
LSHTVTDFTDQMAQVGLCQCVNVFLDEVTG...KAA - length 107, strand -1, frame 2
RALTGLSAPGIRSQTSCDRLRELRYVPVSL...PLQ - length 119, strand -1, frame 2
</pre><p>Note that here we are counting the frames from the 5&#X2019; end (start) of
<em>each</em> strand. It is sometimes easier to always count from the 5&#X2019; end
(start) of the <em>forward</em> strand.</p><p>You could easily edit the above loop based code to build up a list of the
candidate proteins, or convert this to a list comprehension. Now, one thing
this code doesn&#X2019;t do is keep track of where the proteins are.</p><p>You could tackle this in several ways. For example, the following code tracks
the locations in terms of the protein counting, and converts back to the
parent sequence by multiplying by three, then adjusting for the frame and
strand:</p><pre class="verbatim">from Bio import SeqIO 
record = SeqIO.read("NC_005816.gb","genbank")
table = 11
min_pro_len = 100

def find_orfs_with_trans(seq, trans_table, min_protein_length):
    answer = []
    seq_len = len(seq)
    for strand, nuc in [(+1, seq), (-1, seq.reverse_complement())]:
        for frame in range(3):
            trans = str(nuc[frame:].translate(trans_table))
            trans_len = len(trans)
            aa_start = 0
            aa_end = 0
            while aa_start &lt; trans_len:
                aa_end = trans.find("*", aa_start)
                if aa_end == -1:
                    aa_end = trans_len
                if aa_end-aa_start &gt;= min_protein_length:
                    if strand == 1:
                        start = frame+aa_start*3
                        end = min(seq_len,frame+aa_end*3+3)
                    else:
                        start = seq_len-frame-aa_end*3-3
                        end = seq_len-frame-aa_start*3                        
                    answer.append((start, end, strand,
                                   trans[aa_start:aa_end]))
                aa_start = aa_end+1
    answer.sort()
    return answer

orf_list = find_orfs_with_trans(record.seq, table, min_pro_len)
for start, end, strand, pro in orf_list:
    print("%s...%s - length %i, strand %i, %i:%i" \
          % (pro[:30], pro[-3:], len(pro), strand, start, end))
</pre><p>And the output:</p><pre class="verbatim">NQIQGVICSPDSGEFMVTFETVMEIKILHK...GVA - length 355, strand 1, 41:1109
WDVKTVTGVLHHPFHLTFSLCPEGATQSGR...VKR - length 111, strand -1, 491:827
KSGELRQTPPASSTLHLRLILQRSGVMMEL...NPE - length 285, strand 1, 1030:1888
RALTGLSAPGIRSQTSCDRLRELRYVPVSL...PLQ - length 119, strand -1, 2830:3190
RRKEHVSKKRRPQKRPRRRRFFHRLRPPDE...PTR - length 128, strand 1, 3470:3857
GLNCSFFSICNWKFIDYINRLFQIIYLCKN...YYH - length 176, strand 1, 4249:4780
RGIFMSDTMVVNGSGGVPAFLFSGSTLSSY...LLK - length 361, strand -1, 4814:5900
VKKILYIKALFLCTVIKLRRFIFSVNNMKF...DLP - length 165, strand 1, 5923:6421
LSHTVTDFTDQMAQVGLCQCVNVFLDEVTG...KAA - length 107, strand -1, 5974:6298
GCLMKKSSIVATIITILSGSANAASSQLIP...YRF - length 315, strand 1, 6654:7602
IYSTSEHTGEQVMRTLDEVIASRSPESQTR...FHV - length 111, strand -1, 7788:8124
WGKLQVIGLSMWMVLFSQRFDDWLNEQEDA...ESK - length 125, strand -1, 8087:8465
TGKQNSCQMSAIWQLRQNTATKTRQNRARI...AIK - length 100, strand 1, 8741:9044
QGSGYAFPHASILSGIAMSHFYFLVLHAVK...CSD - length 114, strand -1, 9264:9609
</pre><p>If you comment out the sort statement, then the protein sequences will be
shown in the same order as before, so you can check this is doing the same
thing. Here we have sorted them by location to make it easier to compare
to the actual annotation in the GenBank file (as visualised in
Section&#XA0;<a href="#sec%3Agd_nice_example">17.1.9</a>).</p><p>If however all you want to find are the locations of the open reading frames,
then it is a waste of time to translate every possible codon, including doing
the reverse complement to search the reverse strand too. All you need to do
is search for the possible stop codons (and their reverse complements). Using
regular expressions is an obvious approach here (see the Python module
<code>re</code>). These are an extremely powerful (but rather complex) way of
describing search strings, which are supported in lots of programming
languages and also command line tools like <span style="font-family:monospace">grep</span> as well). You can
find whole books about this topic!</p>
<!--TOC section id="sec353" Sequence parsing plus simple plots-->
<h2 id="sec353" class="section">18.2&#XA0;&#XA0;Sequence parsing plus simple plots</h2><!--SEC END --><p>
<a id="seq:sequence-parsing-plus-pylab"></a></p><p>This section shows some more examples of sequence parsing, using the <code>Bio.SeqIO</code>
module described in Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>, plus the Python library matplotlib&#X2019;s <code>pylab</code> plotting interface (see <a href="http://matplotlib.sourceforge.net/">the matplotlib website for a tutorial</a>). Note that to follow these examples you will need matplotlib installed - but without it you can still try the data parsing bits.</p>
<!--TOC subsection id="sec354" Histogram of sequence lengths-->
<h3 id="sec354" class="subsection">18.2.1&#XA0;&#XA0;Histogram of sequence lengths</h3><!--SEC END --><p>There are lots of times when you might want to visualise the distribution of sequence
lengths in a dataset &#X2013; for example the range of contig sizes in a genome assembly
project. In this example we&#X2019;ll reuse our orchid FASTA file <a href="http://biopython.org/DIST/docs/tutorial/examples/ls_orchid.fasta"><span style="font-family:monospace">ls_orchid.fasta</span></a> which has only 94 sequences.</p><p>First of all, we will use <code>Bio.SeqIO</code> to parse the FASTA file and compile a list
of all the sequence lengths. You could do this with a for loop, but I find a list
comprehension more pleasing:</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SeqIO
&gt;&gt;&gt; sizes = [len(rec) for rec in SeqIO.parse("ls_orchid.fasta", "fasta")]
&gt;&gt;&gt; len(sizes), min(sizes), max(sizes)
(94, 572, 789)
&gt;&gt;&gt; sizes
[740, 753, 748, 744, 733, 718, 730, 704, 740, 709, 700, 726, ..., 592]
</pre><p>Now that we have the lengths of all the genes (as a list of integers), we can use the
matplotlib histogram function to display it.</p><pre class="verbatim">from Bio import SeqIO
sizes = [len(rec) for rec in SeqIO.parse("ls_orchid.fasta", "fasta")]

import pylab
pylab.hist(sizes, bins=20)
pylab.title("%i orchid sequences\nLengths %i to %i" \
            % (len(sizes),min(sizes),max(sizes)))
pylab.xlabel("Sequence length (bp)")
pylab.ylabel("Count")
pylab.show()
</pre><p>
That should pop up a new window containing the following graph:</p><p><img src="images/hist_plot.png" width=600, height=450></p><p>

Notice that most of these orchid sequences are about 740 bp long, and there could be
two distinct classes of sequence here with a subset of shorter sequences.</p><p><em>Tip:</em> Rather than using <code>pylab.show()</code> to show the plot in a window, you can also use <code>pylab.savefig(...)</code> to save the figure to a file (e.g. as a PNG or PDF).</p>
<!--TOC subsection id="sec355" Plot of sequence GC%-->
<h3 id="sec355" class="subsection">18.2.2&#XA0;&#XA0;Plot of sequence GC%</h3><!--SEC END --><p>Another easily calculated quantity of a nucleotide sequence is the GC%. You might
want to look at the GC% of all the genes in a bacterial genome for example, and
investigate any outliers which could have been recently acquired by horizontal gene
transfer. Again, for this example we&#X2019;ll reuse our orchid FASTA file <a href="http://biopython.org/DIST/docs/tutorial/examples/ls_orchid.fasta"><span style="font-family:monospace">ls_orchid.fasta</span></a>.</p><p>First of all, we will use <code>Bio.SeqIO</code> to parse the FASTA file and compile a list
of all the GC percentages. Again, you could do this with a for loop, but I prefer this:</p><pre class="verbatim">from Bio import SeqIO
from Bio.SeqUtils import GC

gc_values = sorted(GC(rec.seq) for rec in SeqIO.parse("ls_orchid.fasta", "fasta"))
</pre><p>Having read in each sequence and calculated the GC%, we then sorted them into ascending
order. Now we&#X2019;ll take this list of floating point values and plot them with matplotlib:</p><pre class="verbatim">import pylab
pylab.plot(gc_values)
pylab.title("%i orchid sequences\nGC%% %0.1f to %0.1f" \
            % (len(gc_values),min(gc_values),max(gc_values)))
pylab.xlabel("Genes")
pylab.ylabel("GC%")
pylab.show()
</pre><p>
As in the previous example, that should pop up a new window containing a graph:</p><p><img src="images/gc_plot.png" width=600, height=450></p><p>

If you tried this on the full set of genes from one organism, you&#X2019;d probably get a much
smoother plot than this.</p>
<!--TOC subsection id="sec356" Nucleotide dot plots-->
<h3 id="sec356" class="subsection">18.2.3&#XA0;&#XA0;Nucleotide dot plots</h3><!--SEC END --><p>
A dot plot is a way of visually comparing two nucleotide sequences for similarity to
each other. A sliding window is used to compare short sub-sequences to each other,
often with a mis-match threshold. Here for simplicity we&#X2019;ll only look for perfect
matches (shown in black


in the plot below).
</p><p>To start off, we&#X2019;ll need two sequences. For the sake of argument, we&#X2019;ll just take
the first two from our orchid FASTA file <a href="http://biopython.org/DIST/docs/tutorial/examples/ls_orchid.fasta"><span style="font-family:monospace">ls_orchid.fasta</span></a>:</p><pre class="verbatim">from Bio import SeqIO
handle = open("ls_orchid.fasta")
record_iterator = SeqIO.parse(handle, "fasta")
rec_one = next(record_iterator)
rec_two = next(record_iterator)
handle.close()
</pre><p>We&#X2019;re going to show two approaches. Firstly, a simple naive implementation
which compares all the window sized sub-sequences to each other to compiles a
similarity matrix. You could construct a matrix or array object, but here we
just use a list of lists of booleans created with a nested list
comprehension:</p><pre class="verbatim">window = 7
seq_one = str(rec_one.seq).upper()
seq_two = str(rec_two.seq).upper()
data = [[(seq_one[i:i+window] &lt;&gt; seq_two[j:j+window]) \
        for j in range(len(seq_one)-window)] \
       for i in range(len(seq_two)-window)]
</pre><p>Note that we have <em>not</em> checked for reverse complement matches here.
Now we&#X2019;ll use the matplotlib&#X2019;s <code>pylab.imshow()</code> function to display this
data, first requesting the gray color scheme so this is done in black and
white:</p><pre class="verbatim">import pylab
pylab.gray()
pylab.imshow(data)
pylab.xlabel("%s (length %i bp)" % (rec_one.id, len(rec_one)))
pylab.ylabel("%s (length %i bp)" % (rec_two.id, len(rec_two)))
pylab.title("Dot plot using window size %i\n(allowing no mis-matches)" % window)
pylab.show()
</pre><p>
That should pop up a new window containing a graph like this:</p><p><img src="images/dot_plot.png" width=600, height=450></p><p>

As you might have expected, these two sequences are very similar with a
partial line of window sized matches along the diagonal. There are no off
diagonal matches which would be indicative of inversions or other interesting
events.</p><p>The above code works fine on small examples, but there are two problems
applying this to larger sequences, which we will address below.
First off all, this brute force approach to the all against all comparisons
is very slow. Instead, we&#X2019;ll compile dictionaries mapping the window sized
sub-sequences to their locations, and then take the set intersection to find
those sub-sequences found in both sequences. This uses more memory, but is
<em>much</em> faster. Secondly, the <code>pylab.imshow()</code> function is limited
in the size of matrix it can display. As an alternative, we&#X2019;ll use the
<code>pylab.scatter()</code> function.</p><p>We start by creating dictionaries mapping the window-sized sub-sequences to locations:
</p><pre class="verbatim">window = 7
dict_one = {}
dict_two = {}
for (seq, section_dict) in [(str(rec_one.seq).upper(), dict_one),
                            (str(rec_two.seq).upper(), dict_two)]:
    for i in range(len(seq)-window):
        section = seq[i:i+window]
        try:
            section_dict[section].append(i)
        except KeyError:
            section_dict[section] = [i]
#Now find any sub-sequences found in both sequences
#(Python 2.3 would require slightly different code here)
matches = set(dict_one).intersection(dict_two)
print("%i unique matches" % len(matches))
</pre><p>In order to use the <code>pylab.scatter()</code> we need separate lists for the <span style="font-style:italic">x</span> and <span style="font-style:italic">y</span> co-ordinates:
</p><pre class="verbatim">#Create lists of x and y co-ordinates for scatter plot
x = []
y = []
for section in matches:
    for i in dict_one[section]:
        for j in dict_two[section]:
            x.append(i)
            y.append(j)
</pre><p>We are now ready to draw the revised dot plot as a scatter plot:
</p><pre class="verbatim">import pylab
pylab.cla() #clear any prior graph
pylab.gray()
pylab.scatter(x,y)
pylab.xlim(0, len(rec_one)-window)
pylab.ylim(0, len(rec_two)-window)
pylab.xlabel("%s (length %i bp)" % (rec_one.id, len(rec_one)))
pylab.ylabel("%s (length %i bp)" % (rec_two.id, len(rec_two)))
pylab.title("Dot plot using window size %i\n(allowing no mis-matches)" % window)
pylab.show()
</pre><p>
That should pop up a new window containing a graph like this:</p><p><img src="images/dot_plot_scatter.png" width=600, height=450></p><p>

Personally I find this second plot much easier to read!
Again note that we have <em>not</em> checked for reverse complement matches here
&#X2013; you could extend this example to do this, and perhaps plot the forward
matches in one color and the reverse matches in another.</p>
<!--TOC subsection id="sec357" Plotting the quality scores of sequencing read data-->
<h3 id="sec357" class="subsection">18.2.4&#XA0;&#XA0;Plotting the quality scores of sequencing read data</h3><!--SEC END --><p>If you are working with second generation sequencing data, you may want to try plotting
the quality data. Here is an example using two FASTQ files containing paired end reads,
<span style="font-family:monospace">SRR001666_1.fastq</span> for the forward reads, and <span style="font-family:monospace">SRR001666_2.fastq</span> for
the reverse reads. These were downloaded from the ENA sequence read archive FTP site
(<a href="ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR001/SRR001666/SRR001666_1.fastq.gz"><span style="font-family:monospace">ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR001/SRR001666/SRR001666_1.fastq.gz</span></a> and
<a href="ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR001/SRR001666/SRR001666_2.fastq.gz"><span style="font-family:monospace">ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR001/SRR001666/SRR001666_2.fastq.gz</span></a>), and
are from <span style="font-style:italic">E. coli</span> &#X2013; see <a href="http://www.ebi.ac.uk/ena/data/view/SRR001666"><span style="font-family:monospace">http://www.ebi.ac.uk/ena/data/view/SRR001666</span></a>
for details.
</p><p>In the following code the <code>pylab.subplot(...)</code> function is used in order to show
the forward and reverse qualities on two subplots, side by side. There is also a little
bit of code to only plot the first fifty reads.</p><pre class="verbatim">import pylab
from Bio import SeqIO
for subfigure in [1,2]:
    filename = "SRR001666_%i.fastq" % subfigure
    pylab.subplot(1, 2, subfigure)
    for i,record in enumerate(SeqIO.parse(filename, "fastq")):
        if i &gt;= 50 : break #trick!
        pylab.plot(record.letter_annotations["phred_quality"])
    pylab.ylim(0,45)
    pylab.ylabel("PHRED quality score")
    pylab.xlabel("Position")
pylab.savefig("SRR001666.png")
print("Done")
</pre><p>You should note that we are using the <code>Bio.SeqIO</code> format name <span style="font-family:monospace">fastq</span>
here because the NCBI has saved these reads using the standard Sanger FASTQ format
with PHRED scores. However, as you might guess from the read lengths, this data was
from an Illumina Genome Analyzer and was probably originally in one of the two
Solexa/Illumina FASTQ variant file formats instead.</p><p>This example uses the <code>pylab.savefig(...)</code> function instead of
<code>pylab.show(...)</code>, but as mentioned before both are useful.


Here is the result:</p><p><img src="images/SRR001666.png" width=600, height=600></p>
<!--TOC section id="sec358" Dealing with alignments-->
<h2 id="sec358" class="section">18.3&#XA0;&#XA0;Dealing with alignments</h2><!--SEC END --><p>This section can been seen as a follow on to Chapter&#XA0;<a href="#chapter%3ABio.AlignIO">6</a>.</p>
<!--TOC subsection id="sec359" Calculating summary information-->
<h3 id="sec359" class="subsection">18.3.1&#XA0;&#XA0;Calculating summary information</h3><!--SEC END --><p>
<a id="sec:summary_info"></a></p><p>Once you have an alignment, you are very likely going to want to find out information about it. Instead of trying to have all of the functions that can generate information about an alignment in the alignment object itself, we&#X2019;ve tried to separate out the functionality into separate classes, which act on the alignment.</p><p>Getting ready to calculate summary information about an object is quick to do. Let&#X2019;s say we&#X2019;ve got an alignment object called <code>alignment</code>, for example read in using <code>Bio.AlignIO.read(...)</code> as described in Chapter&#XA0;<a href="#chapter%3ABio.AlignIO">6</a>. All we need to do to get an object that will calculate summary information is:</p><pre class="verbatim">from Bio.Align import AlignInfo
summary_align = AlignInfo.SummaryInfo(alignment)
</pre><p>The <code>summary_align</code> object is very useful, and will do the following neat things for you:</p><ol class="enumerate" type=1><li class="li-enumerate">
Calculate a quick consensus sequence &#X2013; see section&#XA0;<a href="#sec%3Aconsensus">18.3.2</a>
</li><li class="li-enumerate">Get a position specific score matrix for the alignment &#X2013; see section&#XA0;<a href="#sec%3Apssm">18.3.3</a>
</li><li class="li-enumerate">Calculate the information content for the alignment &#X2013; see section&#XA0;<a href="#sec%3Agetting_info_content">18.3.4</a>
</li><li class="li-enumerate">Generate information on substitutions in the alignment &#X2013; section&#XA0;<a href="#sec%3Asub_matrix">18.4</a> details using this to generate a substitution matrix.
</li></ol>
<!--TOC subsection id="sec360" Calculating a quick consensus sequence-->
<h3 id="sec360" class="subsection">18.3.2&#XA0;&#XA0;Calculating a quick consensus sequence</h3><!--SEC END --><p>
<a id="sec:consensus"></a></p><p>The <code>SummaryInfo</code> object, described in section&#XA0;<a href="#sec%3Asummary_info">18.3.1</a>, provides functionality to calculate a quick consensus of an alignment. Assuming we&#X2019;ve got a <code>SummaryInfo</code> object called <code>summary_align</code> we can calculate a consensus by doing:</p><pre class="verbatim">consensus = summary_align.dumb_consensus()
</pre><p>As the name suggests, this is a really simple consensus calculator, and will just add up all of the residues at each point in the consensus, and if the most common value is higher than some threshold value will add the common residue to the consensus. If it doesn&#X2019;t reach the threshold, it adds an ambiguity character to the consensus. The returned consensus object is Seq object whose alphabet is inferred from the alphabets of the sequences making up the consensus. So doing a <code>print consensus</code> would give:</p><pre class="verbatim">consensus Seq('TATACATNAAAGNAGGGGGATGCGGATAAATGGAAAGGCGAAAGAAAGAAAAAAATGAAT
...', IUPACAmbiguousDNA())
</pre><p>You can adjust how <code>dumb_consensus</code> works by passing optional parameters:</p><dl class="description"><dt class="dt-description">
<span style="font-weight:bold">the threshold</span></dt><dd class="dd-description"> This is the threshold specifying how common a particular residue has to be at a position before it is added. The default is 0.7 (meaning 70%).</dd><dt class="dt-description"><span style="font-weight:bold">the ambiguous character</span></dt><dd class="dd-description"> This is the ambiguity character to use. The default is &#X2019;N&#X2019;.</dd><dt class="dt-description"><span style="font-weight:bold">the consensus alphabet</span></dt><dd class="dd-description"> This is the alphabet to use for the consensus sequence. If an alphabet is not specified than we will try to guess the alphabet based on the alphabets of the sequences in the alignment.
</dd></dl>
<!--TOC subsection id="sec361" Position Specific Score Matrices-->
<h3 id="sec361" class="subsection">18.3.3&#XA0;&#XA0;Position Specific Score Matrices</h3><!--SEC END --><p>
<a id="sec:pssm"></a></p><p>Position specific score matrices (PSSMs) summarize the alignment information in a different way than a consensus, and may be useful for different tasks. Basically, a PSSM is a count matrix. For each column in the alignment, the number of each alphabet letters is counted and totaled. The totals are displayed relative to some representative sequence along the left axis. This sequence may be the consesus sequence, but can also be any sequence in the alignment. For instance for the alignment,</p><pre class="verbatim">GTATC
AT--C
CTGTC
</pre><p>the PSSM is:</p><pre class="verbatim">      G A T C
    G 1 1 0 1
    T 0 0 3 0
    A 1 1 0 0
    T 0 0 2 0
    C 0 0 0 3
</pre><p>Let&#X2019;s assume we&#X2019;ve got an alignment object called <code>c_align</code>. To get a PSSM with the consensus sequence along the side we first get a summary object and calculate the consensus sequence:</p><pre class="verbatim">summary_align = AlignInfo.SummaryInfo(c_align)
consensus = summary_align.dumb_consensus()
</pre><p>Now, we want to make the PSSM, but ignore any <code>N</code> ambiguity residues when calculating this:</p><pre class="verbatim">my_pssm = summary_align.pos_specific_score_matrix(consensus,
                                                  chars_to_ignore = ['N'])
</pre><p>Two notes should be made about this:</p><ol class="enumerate" type=1><li class="li-enumerate">
To maintain strictness with the alphabets, you can only include characters along the top of the PSSM that are in the alphabet of the alignment object. Gaps are not included along the top axis of the PSSM.</li><li class="li-enumerate">The sequence passed to be displayed along the left side of the axis does not need to be the consensus. For instance, if you wanted to display the second sequence in the alignment along this axis, you would need to do:<pre class="verbatim">second_seq = alignment.get_seq_by_num(1)
my_pssm = summary_align.pos_specific_score_matrix(second_seq
                                                  chars_to_ignore = ['N'])
</pre></li></ol><p>The command above returns a <code>PSSM</code> object.
To print out the PSSM as shown above,
we simply need to do a <code>print(my_pssm)</code>, which gives:</p><pre class="verbatim">    A   C   G   T
T  0.0 0.0 0.0 7.0
A  7.0 0.0 0.0 0.0
T  0.0 0.0 0.0 7.0
A  7.0 0.0 0.0 0.0
C  0.0 7.0 0.0 0.0
A  7.0 0.0 0.0 0.0
T  0.0 0.0 0.0 7.0
T  1.0 0.0 0.0 6.0
...
</pre><p>You can access any element of the PSSM by subscripting like <code>your_pssm[sequence_number][residue_count_name]</code>. For instance, to get the counts for the &#X2019;A&#X2019; residue in the second element of the above PSSM you would do:</p><pre class="verbatim">&gt;&gt;&gt; print(my_pssm[1]["A"])
7.0
</pre><p>The structure of the PSSM class hopefully makes it easy both to access elements and to pretty print the matrix.</p>
<!--TOC subsection id="sec362" Information Content-->
<h3 id="sec362" class="subsection">18.3.4&#XA0;&#XA0;Information Content</h3><!--SEC END --><p>
<a id="sec:getting_info_content"></a></p><p>A potentially useful measure of evolutionary conservation is the information content of a sequence.</p><p>A useful introduction to information theory targeted towards molecular biologists can be found at <a href="http://www.lecb.ncifcrf.gov/~toms/paper/primer/"><span style="font-family:monospace">http://www.lecb.ncifcrf.gov/~toms/paper/primer/</span></a>. For our purposes, we will be looking at the information content of a consesus sequence, or a portion of a consensus sequence. We calculate information content at a particular column in a multiple sequence alignment using the following formula:</p><table class="display dcenter"><tr style="vertical-align:middle"><td class="dcell"><span style="font-style:italic">IC</span><sub><span style="font-style:italic">j</span></sub>&#XA0;=&#XA0;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">N</span><sub><span style="font-style:italic">a</span></sub></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-size:xx-large">&#X2211;</span></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">i</span>=1</td></tr>
</table></td><td class="dcell">&#XA0;<span style="font-style:italic">P</span><sub><span style="font-style:italic">ij</span></sub>&#XA0;<span style="font-style:italic">log</span></td><td class="dcell">&#X239B;<br>
&#X239C;<br>
&#X239C;<br>
&#X239D;</td><td class="dcell"><table class="display"><tr><td class="dcell" style="text-align:center"><span style="font-style:italic">P</span><sub><span style="font-style:italic">ij</span></sub></td></tr>
<tr><td class="hbar"></td></tr>
<tr><td class="dcell" style="text-align:center"><span style="font-style:italic">Q</span><sub><span style="font-style:italic">i</span></sub></td></tr>
</table></td><td class="dcell">&#X239E;<br>
&#X239F;<br>
&#X239F;<br>
&#X23A0;</td></tr>
</table><p>where:</p><ul class="itemize"><li class="li-itemize">
<span style="font-style:italic">IC</span><sub><span style="font-style:italic">j</span></sub> &#X2013; The information content for the <span style="font-style:italic">j</span>-th column in an alignment.
</li><li class="li-itemize"><span style="font-style:italic">N</span><sub><span style="font-style:italic">a</span></sub> &#X2013; The number of letters in the alphabet.
</li><li class="li-itemize"><span style="font-style:italic">P</span><sub><span style="font-style:italic">ij</span></sub> &#X2013; The frequency of a particular letter <span style="font-style:italic">i</span> in the <span style="font-style:italic">j</span>-th column (i.&#XA0;e.&#XA0;if G occurred 3 out of 6 times in an aligment column, this would be 0.5)
</li><li class="li-itemize"><span style="font-style:italic">Q</span><sub><span style="font-style:italic">i</span></sub> &#X2013; The expected frequency of a letter <span style="font-style:italic">i</span>. This is an
optional argument, usage of which is left at the user&#X2019;s
discretion. By default, it is automatically assigned to 0.05 = 1/20 for a
protein alphabet, and 0.25 = 1/4 for a nucleic acid alphabet. This is for
geting the information content without any assumption of prior
distributions. When assuming priors, or when using a non-standard
alphabet, you should supply the values for <span style="font-style:italic">Q</span><sub><span style="font-style:italic">i</span></sub>.
</li></ul><p>Well, now that we have an idea what information content is being calculated in Biopython, let&#X2019;s look at how to get it for a particular region of the alignment.</p><p>First, we need to use our alignment to get an alignment summary object, which we&#X2019;ll assume is called <code>summary_align</code> (see section&#XA0;<a href="#sec%3Asummary_info">18.3.1</a>) for instructions on how to get this. Once we&#X2019;ve got this object, calculating the information content for a region is as easy as:</p><pre class="verbatim">info_content = summary_align.information_content(5, 30,
                                                 chars_to_ignore = ['N'])
</pre><p>Wow, that was much easier then the formula above made it look! The variable <code>info_content</code> now contains a float value specifying the information content over the specified region (from 5 to 30 of the alignment). We specifically ignore the ambiguity residue &#X2019;N&#X2019; when calculating the information content, since this value is not included in our alphabet (so we shouldn&#X2019;t be interested in looking at it!).</p><p>As mentioned above, we can also calculate relative information content by supplying the expected frequencies:</p><pre class="verbatim">expect_freq = {
    'A' : .3,
    'G' : .2,
    'T' : .3,
    'C' : .2}
</pre><p>The expected should not be passed as a raw dictionary, but instead by passed as a <code>SubsMat.FreqTable</code> object (see section&#XA0;<a href="#sec%3Afreq_table">20.2.2</a> for more information about FreqTables). The FreqTable object provides a standard for associating the dictionary with an Alphabet, similar to how the Biopython Seq class works.</p><p>To create a FreqTable object, from the frequency dictionary you just need to do:</p><pre class="verbatim">from Bio.Alphabet import IUPAC
from Bio.SubsMat import FreqTable

e_freq_table = FreqTable.FreqTable(expect_freq, FreqTable.FREQ,
                                   IUPAC.unambiguous_dna)
</pre><p>Now that we&#X2019;ve got that, calculating the relative information content for our region of the alignment is as simple as:</p><pre class="verbatim">info_content = summary_align.information_content(5, 30,
                                                 e_freq_table = e_freq_table,
                                                 chars_to_ignore = ['N'])
</pre><p>Now, <code>info_content</code> will contain the relative information content over the region in relation to the expected frequencies.</p><p>The value return is calculated using base 2 as the logarithm base in the formula above. You can modify this by passing the parameter <code>log_base</code> as the base you want:</p><pre class="verbatim">info_content = summary_align.information_content(5, 30, log_base = 10,
                                                 chars_to_ignore = ['N'])
</pre><p>Well, now you are ready to calculate information content. If you want to try applying this to some real life problems, it would probably be best to dig into the literature on information content to get an idea of how it is used. Hopefully your digging won&#X2019;t reveal any mistakes made in coding this function!</p>
<!--TOC section id="sec363" Substitution Matrices-->
<h2 id="sec363" class="section">18.4&#XA0;&#XA0;Substitution Matrices</h2><!--SEC END --><p>
<a id="sec:sub_matrix"></a></p><p>Substitution matrices are an extremely important part of everyday bioinformatics work. They provide the scoring terms for classifying how likely two different residues are to substitute for each other. This is essential in doing sequence comparisons. The book &#X201C;Biological Sequence Analysis&#X201D; by Durbin et al. provides a really nice introduction to Substitution Matrices and their uses. Some famous substitution matrices are the PAM and BLOSUM series of matrices.</p><p>Biopython provides a ton of common substitution matrices, and also provides functionality for creating your own substitution matrices.</p>
<!--TOC subsection id="sec364" Using common substitution matrices-->
<h3 id="sec364" class="subsection">18.4.1&#XA0;&#XA0;Using common substitution matrices</h3><!--SEC END -->
<!--TOC subsection id="sec365" Creating your own substitution matrix from an alignment-->
<h3 id="sec365" class="subsection">18.4.2&#XA0;&#XA0;Creating your own substitution matrix from an alignment</h3><!--SEC END --><p>
<a id="sec:subs_mat_ex"></a></p><p>A very cool thing that you can do easily with the substitution matrix
classes is to create your own substitution matrix from an
alignment. In practice, this is normally done with protein
alignments. In this example, we&#X2019;ll first get a Biopython alignment
object and then get a summary object to calculate info about the
alignment. The file containing <a href="examples/protein.aln">protein.aln</a>
(also available online
<a href="http://biopython.org/DIST/docs/tutorial/examples/protein.aln">here</a>)
contains the Clustalw alignment output.</p><pre class="verbatim">&gt;&gt;&gt; from Bio import AlignIO
&gt;&gt;&gt; from Bio import Alphabet
&gt;&gt;&gt; from Bio.Alphabet import IUPAC
&gt;&gt;&gt; from Bio.Align import AlignInfo
&gt;&gt;&gt; filename = "protein.aln"
&gt;&gt;&gt; alpha = Alphabet.Gapped(IUPAC.protein)
&gt;&gt;&gt; c_align = AlignIO.read(filename, "clustal", alphabet=alpha)
&gt;&gt;&gt; summary_align = AlignInfo.SummaryInfo(c_align)
</pre><p>Sections&#XA0;<a href="#sec%3Aalign_clustal">6.4.1</a> and&#XA0;<a href="#sec%3Asummary_info">18.3.1</a> contain
more information on doing this.</p><p>Now that we&#X2019;ve got our <code>summary_align</code> object, we want to use it
to find out the number of times different residues substitute for each
other. To make the example more readable, we&#X2019;ll focus on only amino
acids with polar charged side chains. Luckily, this can be done easily
when generating a replacement dictionary, by passing in all of the
characters that should be ignored. Thus we&#X2019;ll create a dictionary of
replacements for only charged polar amino acids using:</p><pre class="verbatim">&gt;&gt;&gt; replace_info = summary_align.replacement_dictionary(["G", "A", "V", "L", "I",
...                                                      "M", "P", "F", "W", "S",
...                                                      "T", "N", "Q", "Y", "C"])
</pre><p>This information about amino acid replacements is represented as a
python dictionary which will look something like (the order can vary):</p><pre class="verbatim">{('R', 'R'): 2079.0, ('R', 'H'): 17.0, ('R', 'K'): 103.0, ('R', 'E'): 2.0,
('R', 'D'): 2.0, ('H', 'R'): 0, ('D', 'H'): 15.0, ('K', 'K'): 3218.0,
('K', 'H'): 24.0, ('H', 'K'): 8.0, ('E', 'H'): 15.0, ('H', 'H'): 1235.0,
('H', 'E'): 18.0, ('H', 'D'): 0, ('K', 'D'): 0, ('K', 'E'): 9.0,
('D', 'R'): 48.0, ('E', 'R'): 2.0, ('D', 'K'): 1.0, ('E', 'K'): 45.0,
('K', 'R'): 130.0, ('E', 'D'): 241.0, ('E', 'E'): 3305.0,
('D', 'E'): 270.0, ('D', 'D'): 2360.0}
</pre><p>This information gives us our accepted number of replacements, or how
often we expect different things to substitute for each other. It
turns out, amazingly enough, that this is all of the information we
need to go ahead and create a substitution matrix. First, we use the
replacement dictionary information to create an Accepted Replacement
Matrix (ARM):</p><pre class="verbatim">&gt;&gt;&gt; from Bio import SubsMat
&gt;&gt;&gt; my_arm = SubsMat.SeqMat(replace_info)
</pre><p>With this accepted replacement matrix, we can go right ahead and
create our log odds matrix (i.&#XA0;e.&#XA0;a standard type Substitution Matrix):</p><pre class="verbatim">&gt;&gt;&gt; my_lom = SubsMat.make_log_odds_matrix(my_arm)
</pre><p>The log odds matrix you create is customizable with the following
optional arguments:</p><ul class="itemize"><li class="li-itemize">
<code>exp_freq_table</code> &#X2013; You can pass a table of expected
frequencies for each alphabet. If supplied, this will be used
instead of the passed accepted replacement matrix when calculate
expected replacments.</li><li class="li-itemize"><code>logbase</code> - The base of the logarithm taken to create the
log odd matrix. Defaults to base 10.</li><li class="li-itemize"><code>factor</code> - The factor to multiply each matrix entry
by. This defaults to 10, which normally makes the matrix numbers
easy to work with.</li><li class="li-itemize"><code>round_digit</code> - The digit to round to in the matrix. This
defaults to 0 (i.&#XA0;e.&#XA0;no digits).</li></ul><p>Once you&#X2019;ve got your log odds matrix, you can display it prettily
using the function <code>print_mat</code>. Doing this on our created matrix
gives:</p><pre class="verbatim">&gt;&gt;&gt; my_lom.print_mat()
D   2
E  -1   1
H  -5  -4   3
K -10  -5  -4   1
R  -4  -8  -4  -2   2
   D   E   H   K   R
</pre><p>Very nice. Now we&#X2019;ve got our very own substitution matrix to play with!</p>
<!--TOC section id="sec366" BioSQL &#X2013; storing sequences in a relational database-->
<h2 id="sec366" class="section">18.5&#XA0;&#XA0;BioSQL &#X2013; storing sequences in a relational database</h2><!--SEC END --><p>
<a id="sec:BioSQL"></a>
<a href="http://www.biosql.org/">BioSQL</a> is a joint effort between the
<a href="http://open-bio.org/">OBF</a> projects (BioPerl, BioJava etc) to support a
shared database schema for storing sequence data. In theory, you could load a
GenBank file into the database with BioPerl, then using Biopython extract this
from the database as a record object with features - and get more or less the same
thing as if you had loaded the GenBank file directly as a SeqRecord using
<code>Bio.SeqIO</code> (Chapter&#XA0;<a href="#chapter%3ABio.SeqIO">5</a>).</p><p>Biopython&#X2019;s BioSQL module is currently documented at
<a href="http://biopython.org/wiki/BioSQL"><span style="font-family:monospace">http://biopython.org/wiki/BioSQL</span></a> which is part of our wiki pages.</p>
<!--TOC chapter id="sec367" The Biopython testing framework-->
<h1 id="sec367" class="chapter">Chapter&#XA0;19&#XA0;&#XA0;The Biopython testing framework</h1><!--SEC END --><p>
<a id="sec:regr_test"></a></p><p>Biopython has a regression testing framework (the file
<code>run_tests.py</code>) based on
<a href="http://docs.python.org/library/unittest.html">unittest</a>,
the standard unit testing framework for Python. Providing comprehensive
tests for modules is one of the most important aspects of making sure that
the Biopython code is as bug-free as possible before going out.
It also tends to be one of the most undervalued aspects of contributing.
This chapter is designed to make running the Biopython tests and
writing good test code as easy as possible.
Ideally, every module that goes into Biopython
should have a test (and should also have documentation!).
All our developers, and anyone installing Biopython from source,
are strongly encouraged to run the unit tests.</p>
<!--TOC section id="sec368" Running the tests-->
<h2 id="sec368" class="section">19.1&#XA0;&#XA0;Running the tests</h2><!--SEC END --><p>When you download the Biopython source code, or check it out from
our source code repository, you should find a subdirectory call
<code>Tests</code>. This contains the key script <code>run_tests.py</code>,
lots of individual scripts named <code>test_XXX.py</code>, a subdirectory
called <code>output</code> and lots of other subdirectories which
contain input files for the test suite.</p><p>As part of building and installing Biopython you will typically
run the full test suite at the command line from the Biopython
source top level directory using the following:
</p><pre class="verbatim">python setup.py test
</pre><p>This is actually equivalent to going to the <code>Tests</code>
subdirectory and running:
</p><pre class="verbatim">python run_tests.py
</pre><p>You&#X2019;ll often want to run just some of the tests, and this is done
like this:
</p><pre class="verbatim">python run_tests.py test_SeqIO.py test_AlignIO.py
</pre><p>When giving the list of tests, the <code>.py</code> extension is optional,
so you can also just type:
</p><pre class="verbatim">python run_tests.py test_SeqIO test_AlignIO
</pre><p>To run the docstring tests (see section <a href="#section%3Adoctest">19.3</a>), you can use
</p><pre class="verbatim">python run_tests.py doctest
</pre><p>By default, <code>run_tests.py</code> runs all tests, including the docstring tests.</p><p>If an individual test is failing, you can also try running it
directly, which may give you more information.</p><p>Importantly, note that the individual unit tests come in two types:
</p><ul class="itemize"><li class="li-itemize">
Simple print-and-compare scripts. These unit tests are
essentially short example Python programs, which print out
various output text. For a test file named <code>test_XXX.py</code>
there will be a matching text file called <code>test_XXX</code> under
the <code>output</code> subdirectory which contains the expected
output. All that the test framework does to is run the script,
and check the output agrees.
</li><li class="li-itemize">Standard <code>unittest</code>- based tests. These will <code>import unittest</code>
and then define <code>unittest.TestCase</code> classes, each with one
or more sub-tests as methods starting with <code>test_</code> which
check some specific aspect of the code.
These tests should not print any output directly.
</li></ul><p>
Currently, about half of the Biopython tests are <code>unittest</code>-style tests, and half are print-and-compare tests.</p><p>Running a simple print-and-compare test directly will usually give lots
of output on screen, but does not check the output matches the expected
output. If the test is failing with an exception error, it should be
very easy to locate where exactly the script is failing.
For an example of a print-and-compare test, try:
</p><pre class="verbatim">python test_SeqIO.py
</pre><p>The <code>unittest</code>-based tests instead show you exactly which sub-section(s) of
the test are failing. For example,
</p><pre class="verbatim">python test_Cluster.py
</pre>
<!--TOC section id="sec369" Writing tests-->
<h2 id="sec369" class="section">19.2&#XA0;&#XA0;Writing tests</h2><!--SEC END --><p>Let&#X2019;s say you want to write some tests for a module called <code>Biospam</code>.
This can be a module you wrote, or an existing module that doesn&#X2019;t have
any tests yet. In the examples below, we assume that
<code>Biospam</code> is a module that does simple math.</p><p>Each Biopython test can have three important files and directories involved with it:</p><ol class="enumerate" type=1><li class="li-enumerate">
<code>test_Biospam.py</code> &#X2013; The actual test code for your module.
</li><li class="li-enumerate"><code>Biospam</code> [optional]&#X2013; A directory where any necessary input files
will be located. Any output files that will be generated should also
be written here (and preferably cleaned up after the tests are
done) to prevent clogging up the main Tests directory.
</li><li class="li-enumerate"><code>output/Biospam</code> &#X2013; [for print-and-compare tests only] This
file contains the expected output from running <code>test_Biospam.py</code>.
This file is not needed for <code>unittest</code>-style tests, since there
the validation is done in the test script <code>test_Biospam.py</code> itself.
</li></ol><p>It&#X2019;s up to you to decide whether you want to write a print-and-compare test script or a <code>unittest</code>-style test script. The important thing is that you cannot mix these two styles in a single test script. Particularly, don&#X2019;t use <code>unittest</code> features in a print-and-compare test.</p><p>Any script with a <code>test_</code> prefix in the <code>Tests</code> directory will be found and run by <code>run_tests.py</code>. Below, we show an example test script <code>test_Biospam.py</code> both for a print-and-compare test and for a <code>unittest</code>-based test. If you put this script in the Biopython <code>Tests</code> directory, then <code>run_tests.py</code> will find it and execute the tests contained in it:
</p><pre class="verbatim">$ python run_tests.py     
test_Ace ... ok
test_AlignIO ... ok
test_BioSQL ... ok
test_BioSQL_SeqIO ... ok
test_Biospam ... ok
test_CAPS ... ok
test_Clustalw ... ok
</pre><p>&#X2026;</p><pre class="verbatim">----------------------------------------------------------------------
Ran 107 tests in 86.127 seconds
</pre>
<!--TOC subsection id="sec370" Writing a print-and-compare test-->
<h3 id="sec370" class="subsection">19.2.1&#XA0;&#XA0;Writing a print-and-compare test</h3><!--SEC END --><p>A print-and-compare style test should be much simpler for beginners
or novices to write - essentially it is just an example script using
your new module.</p><p>Here is what you should do to make a print-and-compare test for the
<code>Biospam</code> module.</p><ol class="enumerate" type=1><li class="li-enumerate">
Write a script called <code>test_Biospam.py</code><ul class="itemize"><li class="li-itemize">This script should live in the Tests directory</li><li class="li-itemize">The script should test all of the important functionality
of the module (the more you test the better your test is, of course!).</li><li class="li-itemize">Try to avoid anything which might be platform specific,
such as printing floating point numbers without using an explicit
formatting string to avoid having too many decimal places
(different platforms can give very slightly different values).</li></ul></li><li class="li-enumerate">If the script requires files to do the testing, these should go in
the directory Tests/Biospam (if you just need something generic, like
a FASTA sequence file, or a GenBank record, try and use an existing
sample input file instead).</li><li class="li-enumerate">Write out the test output and verify the output to be correct.<p>There are two ways to do this:</p><ol class="enumerate" type=a><li class="li-enumerate">
The long way:<ul class="itemize"><li class="li-itemize">Run the script and write its output to a file. On UNIX (including
Linux and Mac OS X) machines, you would do something like:
<code>python test_Biospam.py &gt; test_Biospam</code> which would write the
output to the file <code>test_Biospam</code>.</li><li class="li-itemize">Manually look at the file <code>test_Biospam</code> to make sure the output is correct. When you are sure it is all right and there are no bugs, you need to quickly edit the <code>test_Biospam</code> file so that the first line is: &#X2018;<code>test_Biospam</code>&#X2019; (no quotes).</li><li class="li-itemize">copy the <code>test_Biospam</code> file to the directory Tests/output</li></ul></li><li class="li-enumerate">The quick way:<ul class="itemize"><li class="li-itemize">
Run <code>python run_tests.py -g test_Biospam.py</code>. The
regression testing framework is nifty enough that it&#X2019;ll put
the output in the right place in just the way it likes it. </li><li class="li-itemize">Go to the output (which should be in <code>Tests/output/test_Biospam</code>) and double check the output to make sure it is all correct.</li></ul></li></ol></li><li class="li-enumerate">Now change to the Tests directory and run the regression tests
with <code>python run_tests.py</code>. This will run all of the tests, and
you should see your test run (and pass!).</li><li class="li-enumerate">That&#X2019;s it! Now you&#X2019;ve got a nice test for your module ready to check in,
or submit to Biopython. Congratulations!
</li></ol><p>As an example, the <code>test_Biospam.py</code> test script to test the
<code>addition</code> and <code>multiplication</code> functions in the <code>Biospam</code>
module could look as follows:</p><pre class="verbatim">from __future__ import print_function
from Bio import Biospam

print("2 + 3 =", Biospam.addition(2, 3))
print("9 - 1 =", Biospam.addition(9, -1))
print("2 * 3 =", Biospam.multiplication(2, 3))
print("9 * (- 1) =", Biospam.multiplication(9, -1))
</pre><p>We generate the corresponding output with <code>python run_tests.py -g test_Biospam.py</code>, and check the output file <code>output/test_Biospam</code>:</p><pre class="verbatim">test_Biospam
2 + 3 = 5
9 - 1 = 8
2 * 3 = 6
9 * (- 1) = -9
</pre><p>Often, the difficulty with larger print-and-compare tests is to keep track which line in the output corresponds to which command in the test script. For this purpose, it is important to print out some markers to help you match lines in the input script with the generated output.</p>
<!--TOC subsection id="sec371" Writing a unittest-based test-->
<h3 id="sec371" class="subsection">19.2.2&#XA0;&#XA0;Writing a unittest-based test</h3><!--SEC END --><p>We want all the modules in Biopython to have unit tests, and a simple
print-and-compare test is better than no test at all. However, although
there is a steeper learning curve, using the <code>unittest</code> framework
gives a more structured result, and if there is a test failure this can
clearly pinpoint which part of the test is going wrong. The sub-tests can
also be run individually which is helpful for testing or debugging.</p><p>The <code>unittest</code>-framework has been included with Python since version
2.1, and is documented in the Python Library Reference (which I know you
are keeping under your pillow, as recommended). There is also
<a href="http://docs.python.org/library/unittest.html">online documentaion
for unittest</a>.
If you are familiar with the <code>unittest</code> system (or something similar
like the nose test framework), you shouldn&#X2019;t have any trouble. You may
find looking at the existing example within Biopython helpful too.</p><p>Here&#X2019;s a minimal <code>unittest</code>-style test script for <code>Biospam</code>,
which you can copy and paste to get started:</p><pre class="verbatim">import unittest
from Bio import Biospam

class BiospamTestAddition(unittest.TestCase):

    def test_addition1(self):
        result = Biospam.addition(2, 3)
        self.assertEqual(result, 5)

    def test_addition2(self):
        result = Biospam.addition(9, -1)
        self.assertEqual(result, 8)

class BiospamTestDivision(unittest.TestCase):

    def test_division1(self):
        result = Biospam.division(3.0, 2.0)
        self.assertAlmostEqual(result, 1.5)

    def test_division2(self):
        result = Biospam.division(10.0, -2.0)
        self.assertAlmostEqual(result, -5.0)


if __name__ == "__main__":
    runner = unittest.TextTestRunner(verbosity = 2)
    unittest.main(testRunner=runner)
</pre><p>In the division tests, we use <code>assertAlmostEqual</code> instead of <code>assertEqual</code> to avoid tests failing due to roundoff errors; see the <code>unittest</code> chapter in the Python documentation for details and for other functionality available in <code>unittest</code> (<a href="http://docs.python.org/library/unittest.html">online reference</a>).</p><p>These are the key points of <code>unittest</code>-based tests:</p><ul class="itemize"><li class="li-itemize">
Test cases are stored in classes that derive from
<code>unittest.TestCase</code> and cover one basic aspect of your code</li><li class="li-itemize">You can use methods <code>setUp</code> and <code>tearDown</code> for any repeated
code which should be run before and after each test method. For example,
the <code>setUp</code> method might be used to create an instance of the object
you are testing, or open a file handle. The <code>tearDown</code> should do any
&#X201C;tidying up&#X201D;, for example closing the file handle.</li><li class="li-itemize">The tests are prefixed with <code>test_</code> and each test should cover
one specific part of what you are trying to test. You can have as
many tests as you want in a class.</li><li class="li-itemize">At the end of the test script, you can use
<pre class="verbatim">if __name__ == "__main__":
    runner = unittest.TextTestRunner(verbosity = 2)
    unittest.main(testRunner=runner)
</pre>to execute the tests when the script is run by itself (rather than
imported from <code>run_tests.py</code>).
If you run this script, then you&#X2019;ll see something like the following:<pre class="verbatim">$ python test_BiospamMyModule.py
test_addition1 (__main__.TestAddition) ... ok
test_addition2 (__main__.TestAddition) ... ok
test_division1 (__main__.TestDivision) ... ok
test_division2 (__main__.TestDivision) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.059s

OK
</pre></li><li class="li-itemize">To indicate more clearly what each test is doing, you can add
docstrings to each test. These are shown when running the tests,
which can be useful information if a test is failing.<pre class="verbatim">import unittest
from Bio import Biospam

class BiospamTestAddition(unittest.TestCase):

    def test_addition1(self):
        """An addition test"""
        result = Biospam.addition(2, 3)
        self.assertEqual(result, 5)

    def test_addition2(self):
        """A second addition test"""
        result = Biospam.addition(9, -1)
        self.assertEqual(result, 8)

class BiospamTestDivision(unittest.TestCase):

    def test_division1(self):
        """Now let's check division"""
        result = Biospam.division(3.0, 2.0)
        self.assertAlmostEqual(result, 1.5)

    def test_division2(self):
        """A second division test"""
        result = Biospam.division(10.0, -2.0)
        self.assertAlmostEqual(result, -5.0)


if __name__ == "__main__":
    runner = unittest.TextTestRunner(verbosity = 2)
    unittest.main(testRunner=runner)
</pre><p>Running the script will now show you:</p><pre class="verbatim">$ python test_BiospamMyModule.py
An addition test ... ok
A second addition test ... ok
Now let's check division ... ok
A second division test ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.001s

OK
</pre></li></ul><p>If your module contains docstring tests (see section <a href="#section%3Adoctest">19.3</a>),
you may want to include those in the tests to be run. You can do so as
follows by modifying the code under <code>if __name__ == "__main__":</code>
to look like this:</p><pre class="verbatim">if __name__ == "__main__":
    unittest_suite = unittest.TestLoader().loadTestsFromName("test_Biospam")
    doctest_suite = doctest.DocTestSuite(Biospam)
    suite = unittest.TestSuite((unittest_suite, doctest_suite))
    runner = unittest.TextTestRunner(sys.stdout, verbosity = 2)
    runner.run(suite)
</pre><p>This is only relevant if you want to run the docstring tests when you
execute <code>python test_Biospam.py</code>; with
<code>python run_tests.py</code>, the docstring tests are run automatically
(assuming they are included in the list of docstring tests in
<code>run_tests.py</code>, see the section below).</p>
<!--TOC section id="sec372" Writing doctests-->
<h2 id="sec372" class="section">19.3&#XA0;&#XA0;Writing doctests</h2><!--SEC END --><p>
<a id="section:doctest"></a></p><p>Python modules, classes and functions support built in documentation using
docstrings. The <a href="http://docs.python.org/library/doctest.html">doctest
framework</a> (included with Python) allows the developer to embed working
examples in the docstrings, and have these examples automatically tested.</p><p>Currently only a small part of Biopython includes doctests. The
<code>run_tests.py</code> script takes care of running the doctests.
For this purpose, at the top of the <code>run_tests.py</code> script is a
manually compiled list of modules to test, which
allows us to skip modules with optional external dependencies which may
not be installed (e.g. the Reportlab and NumPy libraries). So, if you&#X2019;ve
added some doctests to the docstrings in a Biopython module, in order to
have them included in the Biopython test suite, you must update
<code>run_tests.py</code> to include your module. Currently, the relevant part
of <code>run_tests.py</code> looks as follows:</p><pre class="verbatim"># This is the list of modules containing docstring tests.
# If you develop docstring tests for other modules, please add
# those modules here.
DOCTEST_MODULES = ["Bio.Seq",
                   "Bio.SeqRecord",
                   "Bio.SeqIO",
                   "...",
                  ]
#Silently ignore any doctests for modules requiring numpy!
try:
    import numpy
    DOCTEST_MODULES.extend(["Bio.Statistics.lowess"])
except ImportError:
    pass
</pre><p>Note that we regard doctests primarily as documentation, so you should
stick to typical usage. Generally complicated examples dealing with error
conditions and the like would be best left to a dedicated unit test.</p><p>Note that if you want to write doctests involving file parsing, defining
the file location complicates matters. Ideally use relative paths assuming
the code will be run from the <code>Tests</code> directory, see the
<code>Bio.SeqIO</code> doctests for an example of this.</p><p>To run the docstring tests only, use
</p><pre class="verbatim">$ python run_tests.py doctest
</pre>
<!--TOC chapter id="sec373" Advanced-->
<h1 id="sec373" class="chapter">Chapter&#XA0;20&#XA0;&#XA0;Advanced</h1><!--SEC END --><p>
<a id="chapter:advanced"></a></p>
<!--TOC section id="sec374" Parser Design-->
<h2 id="sec374" class="section">20.1&#XA0;&#XA0;Parser Design</h2><!--SEC END --><p>Many of the older Biopython parsers were built around an event-oriented
design that includes Scanner and Consumer objects.</p><p>Scanners take input from a data source and analyze it line by line,
sending off an event whenever it recognizes some information in the
data. For example, if the data includes information about an organism
name, the scanner may generate an <code>organism_name</code> event whenever it
encounters a line containing the name.</p><p>Consumers are objects that receive the events generated by Scanners.
Following the previous example, the consumer receives the
<code>organism_name</code> event, and the processes it in whatever manner
necessary in the current application.</p><p>This is a very flexible framework, which is advantageous if you want to
be able to parse a file format into more than one representation. For
example, the <code>Bio.GenBank</code> module uses this to construct either
<code>SeqRecord</code> objects or file-format-specific record objects.</p><p>More recently, many of the parsers added for <code>Bio.SeqIO</code> and
<code>Bio.AlignIO</code> take a much simpler approach, but only generate a
single object representation (<code>SeqRecord</code> and
<code>MultipleSeqAlignment</code> objects respectively). In some cases the
<code>Bio.SeqIO</code> parsers actually wrap
another Biopython parser - for example, the <code>Bio.SwissProt</code> parser
produces SwissProt format specific record objects, which get converted
into <code>SeqRecord</code> objects.</p>
<!--TOC section id="sec375" Substitution Matrices-->
<h2 id="sec375" class="section">20.2&#XA0;&#XA0;Substitution Matrices</h2><!--SEC END -->
<!--TOC subsection id="sec376" SubsMat-->
<h3 id="sec376" class="subsection">20.2.1&#XA0;&#XA0;SubsMat</h3><!--SEC END --><p>This module provides a class and a few routines for generating substitution matrices, similar to BLOSUM or PAM matrices, but based on user-provided data. Additionally, you may select a matrix from MatrixInfo.py, a collection of established substitution matrices. The <code>SeqMat</code> class derives from a dictionary:
</p><pre class="verbatim">class SeqMat(dict)
</pre><p>The dictionary is of the form <code>{(i1,j1):n1, (i1,j2):n2,...,(ik,jk):nk}</code> where i, j are alphabet letters, and n is a value.</p><ol class="enumerate" type=1><li class="li-enumerate">
Attributes
<ol class="enumerate" type=a><li class="li-enumerate">
<code>self.alphabet</code>: a class as defined in Bio.Alphabet</li><li class="li-enumerate"><code>self.ab_list</code>: a list of the alphabet&#X2019;s letters, sorted. Needed mainly for internal purposes
</li></ol></li><li class="li-enumerate">Methods<ol class="enumerate" type=a><li class="li-enumerate"><pre class="verbatim">__init__(self,data=None,alphabet=None, mat_name='', build_later=0):
</pre><ol class="enumerate" type=i><li class="li-enumerate"><code>data</code>: can be either a dictionary, or another SeqMat instance.
</li><li class="li-enumerate"><code>alphabet</code>: a Bio.Alphabet instance. If not provided, construct an alphabet from data.</li><li class="li-enumerate"><code>mat_name</code>: matrix name, such as "BLOSUM62" or "PAM250"</li><li class="li-enumerate"><code>build_later</code>: default false. If true, user may supply only alphabet and empty dictionary, if intending to build the matrix later. this skips the sanity check of alphabet size vs. matrix size.</li></ol></li><li class="li-enumerate"><pre class="verbatim">entropy(self,obs_freq_mat)
</pre><ol class="enumerate" type=i><li class="li-enumerate">
<code>obs_freq_mat</code>: an observed frequency matrix. Returns the matrix&#X2019;s entropy, based on the frequency in <code>obs_freq_mat</code>. The matrix instance should be LO or SUBS.
</li></ol></li><li class="li-enumerate"><pre class="verbatim">sum(self)
</pre>Calculates the sum of values for each letter in the matrix&#X2019;s alphabet, and returns it as a dictionary of the form <code>{i1: s1, i2: s2,...,in:sn}</code>, where:
<ul class="itemize"><li class="li-itemize">
i: an alphabet letter;
</li><li class="li-itemize">s: sum of all values in a half-matrix for that letter;
</li><li class="li-itemize">n: number of letters in alphabet.
</li></ul></li><li class="li-enumerate"><pre class="verbatim">print_mat(self,f,format="%4d",bottomformat="%4s",alphabet=None)
</pre><p>prints the matrix to file handle f. <code>format</code> is the format field for the matrix values; <code>bottomformat</code> is the format field for the bottom row, containing matrix letters. Example output for a 3-letter alphabet matrix:</p><pre class="verbatim">A 23
B 12 34
C 7  22  27
  A   B   C
</pre><p>The <code>alphabet</code> optional argument is a string of all characters in the alphabet. If supplied, the order of letters along the axes is taken from the string, rather than by alphabetical order.</p></li></ol></li><li class="li-enumerate">Usage<p>The following section is laid out in the order by which most people wish to generate a log-odds matrix. Of course, interim matrices can be generated and
investigated. Most people just want a log-odds matrix, that&#X2019;s all.</p><ol class="enumerate" type=a><li class="li-enumerate">Generating an Accepted Replacement Matrix<p>Initially, you should generate an accepted replacement matrix (ARM) from your data. The values in ARM are the counted number of replacements according to your data. The data could be a set of pairs or multiple alignments. So for instance if Alanine was replaced by Cysteine 10 times, and Cysteine by Alanine 12 times, the corresponding ARM entries would be:</p><pre class="verbatim">('A','C'): 10, ('C','A'): 12
</pre><p>as order doesn&#X2019;t matter, user can already provide only one entry:</p><pre class="verbatim">('A','C'): 22
</pre><p>A SeqMat instance may be initialized with either a full (first method of counting: 10, 12) or half (the latter method, 22) matrices. A full protein
alphabet matrix would be of the size 20x20 = 400. A half matrix of that alphabet would be 20x20/2 + 20/2 = 210. That is because same-letter entries don&#X2019;t
change. (The matrix diagonal). Given an alphabet size of N:</p><ol class="enumerate" type=i><li class="li-enumerate">
Full matrix size:N*N</li><li class="li-enumerate">Half matrix size: N(N+1)/2
</li></ol><p>The SeqMat constructor automatically generates a half-matrix, if a full matrix is passed. If a half matrix is passed, letters in the key should be provided in alphabetical order: (&#X2019;A&#X2019;,&#X2019;C&#X2019;) and not (&#X2019;C&#X2019;,A&#X2019;).</p><p>At this point, if all you wish to do is generate a log-odds matrix, please go to the section titled Example of Use. The following text describes the nitty-gritty of internal functions, to be used by people who wish to investigate their nucleotide/amino-acid frequency data more thoroughly.</p></li><li class="li-enumerate">Generating the observed frequency matrix (OFM)<p>Use:
</p><pre class="verbatim">OFM = SubsMat._build_obs_freq_mat(ARM)
</pre><p>The OFM is generated from the ARM, only instead of replacement counts, it contains replacement frequencies.</p></li><li class="li-enumerate">Generating an expected frequency matrix (EFM)<p>Use:</p><pre class="verbatim">EFM = SubsMat._build_exp_freq_mat(OFM,exp_freq_table)
</pre><ol class="enumerate" type=i><li class="li-enumerate">
<code>exp_freq_table</code>: should be a FreqTable instance. See section&#XA0;<a href="#sec%3Afreq_table">20.2.2</a> for detailed information on FreqTable. Briefly, the expected frequency table has the frequencies of appearance for each member of the alphabet. It is
implemented as a dictionary with the alphabet letters as keys, and each letter&#X2019;s frequency as a value. Values sum to 1.
</li></ol><p>The expected frequency table can (and generally should) be generated from the observed frequency matrix. So in most cases you will generate <code>exp_freq_table</code> using:</p><pre class="verbatim">&gt;&gt;&gt; exp_freq_table = SubsMat._exp_freq_table_from_obs_freq(OFM)
&gt;&gt;&gt; EFM = SubsMat._build_exp_freq_mat(OFM, exp_freq_table)
</pre><p>But you can supply your own <code>exp_freq_table</code>, if you wish</p></li><li class="li-enumerate">Generating a substitution frequency matrix (SFM)<p>Use:</p><pre class="verbatim">SFM = SubsMat._build_subs_mat(OFM,EFM)
</pre><p>Accepts an OFM, EFM. Provides the division product of the corresponding values.</p></li><li class="li-enumerate">Generating a log-odds matrix (LOM)<p>Use:
</p><pre class="verbatim">LOM=SubsMat._build_log_odds_mat(SFM[,logbase=10,factor=10.0,round_digit=1])
</pre><ol class="enumerate" type=i><li class="li-enumerate">
Accepts an SFM.</li><li class="li-enumerate"><code>logbase</code>: base of the logarithm used to generate the log-odds values.</li><li class="li-enumerate"><code>factor</code>: factor used to multiply the log-odds values. Each entry is generated by log(LOM[key])*factor And rounded to the <code>round_digit</code> place after the decimal point, if required.</li></ol></li></ol></li><li class="li-enumerate">Example of use<p>As most people would want to generate a log-odds matrix, with minimum hassle, SubsMat provides one function which does it all:</p><pre class="verbatim">make_log_odds_matrix(acc_rep_mat,exp_freq_table=None,logbase=10,
                      factor=10.0,round_digit=0):
</pre><ol class="enumerate" type=a><li class="li-enumerate">
<code>acc_rep_mat</code>: user provided accepted replacements matrix
</li><li class="li-enumerate"><code>exp_freq_table</code>: expected frequencies table. Used if provided, if not, generated from the <code>acc_rep_mat</code>.
</li><li class="li-enumerate"><code>logbase</code>: base of logarithm for the log-odds matrix. Default base 10.
</li><li class="li-enumerate"><code>round_digit</code>: number after decimal digit to which result should be rounded. Default zero.
</li></ol></li></ol>
<!--TOC subsection id="sec377" FreqTable-->
<h3 id="sec377" class="subsection">20.2.2&#XA0;&#XA0;FreqTable</h3><!--SEC END --><p>
<a id="sec:freq_table"></a></p><pre class="verbatim">FreqTable.FreqTable(UserDict.UserDict)
</pre><ol class="enumerate" type=1><li class="li-enumerate">Attributes:<ol class="enumerate" type=a><li class="li-enumerate">
<code>alphabet</code>: A Bio.Alphabet instance.
</li><li class="li-enumerate"><code>data</code>: frequency dictionary
</li><li class="li-enumerate"><code>count</code>: count dictionary (in case counts are provided).
</li></ol></li><li class="li-enumerate">Functions:
<ol class="enumerate" type=a><li class="li-enumerate">
<code>read_count(f)</code>: read a count file from stream f. Then convert to frequencies
</li><li class="li-enumerate"><code>read_freq(f)</code>: read a frequency data file from stream f. Of course, we then don&#X2019;t have the counts, but it is usually the letter frquencies which are interesting.
</li></ol></li><li class="li-enumerate">Example of use:
The expected count of the residues in the database is sitting in a file, whitespace delimited, in the following format (example given for a 3-letter alphabet):<pre class="verbatim">A   35
B   65
C   100
</pre><p>And will be read using the <code>FreqTable.read_count(file_handle)</code> function.</p><p>An equivalent frequency file:</p><pre class="verbatim">A  0.175
B  0.325
C  0.5
</pre><p>Conversely, the residue frequencies or counts can be passed as a dictionary.
Example of a count dictionary (3-letter alphabet):</p><pre class="verbatim">{'A': 35, 'B': 65, 'C': 100}
</pre><p>Which means that an expected data count would give a 0.5 frequency
for &#X2019;C&#X2019;, a 0.325 probability of &#X2019;B&#X2019; and a 0.175 probability of &#X2019;A&#X2019;
out of 200 total, sum of A, B and C)</p><p>A frequency dictionary for the same data would be:</p><pre class="verbatim">{'A': 0.175, 'B': 0.325, 'C': 0.5}
</pre><p>Summing up to 1.</p><p>When passing a dictionary as an argument, you should indicate whether it is a count or a frequency dictionary. Therefore the FreqTable class constructor requires two arguments: the dictionary itself, and FreqTable.COUNT or FreqTable.FREQ indicating counts or frequencies, respectively.</p><p>Read expected counts. readCount will already generate the frequencies
Any one of the following may be done to geerate the frequency table (ftab):</p><pre class="verbatim">&gt;&gt;&gt; from SubsMat import *
&gt;&gt;&gt; ftab = FreqTable.FreqTable(my_frequency_dictionary, FreqTable.FREQ)
&gt;&gt;&gt; ftab = FreqTable.FreqTable(my_count_dictionary, FreqTable.COUNT)
&gt;&gt;&gt; ftab = FreqTable.read_count(open('myCountFile'))
&gt;&gt;&gt; ftab = FreqTable.read_frequency(open('myFrequencyFile'))
</pre></li></ol>
<!--TOC chapter id="sec378" Where to go from here &#X2013; contributing to Biopython-->
<h1 id="sec378" class="chapter">Chapter&#XA0;21&#XA0;&#XA0;Where to go from here &#X2013; contributing to Biopython</h1><!--SEC END -->
<!--TOC section id="sec379" Bug Reports + Feature Requests-->
<h2 id="sec379" class="section">21.1&#XA0;&#XA0;Bug Reports + Feature Requests</h2><!--SEC END --><p>Getting feedback on the Biopython modules is very important to us. Open-source projects like this benefit greatly from feedback, bug-reports (and patches!) from a wide variety of contributors.</p><p>The main forums for discussing feature requests and potential bugs are the
<a href="http://biopython.org/wiki/Mailing_lists">Biopython mailing lists</a>:</p><ul class="itemize"><li class="li-itemize">
<a href="mailto:biopython@biopython.org">biopython@biopython.org</a> &#X2013; An unmoderated list for discussion of anything to do with Biopython.</li><li class="li-itemize"><a href="mailto:biopython-dev@biopython.org">biopython-dev@biopython.org</a> &#X2013; A more development oriented list that is mainly used by developers (but anyone is free to contribute!).
</li></ul><p>Additionally, if you think you&#X2019;ve found a new bug, you can submit it to
our issue tracker at <a href="https://github.com/biopython/biopython/issues"><span style="font-family:monospace">https://github.com/biopython/biopython/issues</span></a>
(this has replaced the older tracker hosted at
<a href="http://redmine.open-bio.org/projects/biopython"><span style="font-family:monospace">http://redmine.open-bio.org/projects/biopython</span></a>).
This way, it won&#X2019;t get buried in anyone&#X2019;s Inbox and forgotten about.</p>
<!--TOC section id="sec380" Mailing lists and helping newcomers-->
<h2 id="sec380" class="section">21.2&#XA0;&#XA0;Mailing lists and helping newcomers</h2><!--SEC END --><p>We encourage all our uses to sign up to the main Biopython mailing list.
Once you&#X2019;ve got the hang of an area of Biopython, we&#X2019;d encourage you to
help answer questions from beginners. After all, you were a beginner once.</p>
<!--TOC section id="sec381" Contributing Documentation-->
<h2 id="sec381" class="section">21.3&#XA0;&#XA0;Contributing Documentation</h2><!--SEC END --><p>We&#X2019;re happy to take feedback or contributions - either via a bug-report or on the Mailing List.
While reading this tutorial, perhaps you noticed some topics you were interested in which were missing, or not clearly explained. There is also Biopython&#X2019;s built in documentation (the docstrings, these are also 
<a href="http://biopython.org/DIST/docs/api">online</a>), where again, you may be able to help fill in any blanks.</p>
<!--TOC section id="sec382" Contributing cookbook examples-->
<h2 id="sec382" class="section">21.4&#XA0;&#XA0;Contributing cookbook examples</h2><!--SEC END --><p>
As explained in Chapter&#XA0;<a href="#chapter%3Acookbook">18</a>, Biopython now has a wiki
collection of user contributed &#X201C;cookbook&#X201D; examples,
<a href="http://biopython.org/wiki/Category:Cookbook"><span style="font-family:monospace">http://biopython.org/wiki/Category:Cookbook</span></a> &#X2013; maybe you can add
to this?</p>
<!--TOC section id="sec383" Maintaining a distribution for a platform-->
<h2 id="sec383" class="section">21.5&#XA0;&#XA0;Maintaining a distribution for a platform</h2><!--SEC END --><p>
<a id="sec:maintain_dist"></a></p><p>We currently provide source code archives (suitable for any OS, if you have the right build tools installed), and Windows Installers which are just click and run. This covers all the major operating systems.</p><p>Most major Linux distributions have volunteers who take these source code releases, and compile them into packages for Linux users to easily install (taking care of dependencies etc). This is really great and we are of course very grateful. If you would like to contribute to this work, please find out more about how your Linux distribution handles this.</p><p>Below are some tips for certain platforms to maybe get people started with helping out:</p><dl class="description"><dt class="dt-description"><span style="font-weight:bold">Windows</span></dt><dd class="dd-description"> &#X2013; Windows products typically have a nice graphical installer that installs all of the essential components in the right place. We use Distutils to create a installer of this type fairly easily.<p>You must first make sure you have a C compiler on your Windows computer, and that you can compile and install things (this is the hard bit - see the Biopython installation instructions for info on how to do this).</p><p>Once you are setup with a C compiler, making the installer just requires doing:</p><pre class="verbatim">python setup.py bdist_wininst
</pre><p>Now you&#X2019;ve got a Windows installer. Congrats! At the moment we have no trouble shipping installers built on 32 bit windows. If anyone would like to look into supporting 64 bit Windows that would be great.</p></dd><dt class="dt-description"><span style="font-weight:bold">RPMs</span></dt><dd class="dd-description"> &#X2013; RPMs are pretty popular package systems on some Linux platforms. There is lots of documentation on RPMs available at <a href="http://www.rpm.org"><span style="font-family:monospace">http://www.rpm.org</span></a> to help you get started with them. To create an RPM for your platform is really easy. You just need to be able to build the package from source (having a C compiler that works is thus essential) &#X2013; see the Biopython installation instructions for more info on this.<p>To make the RPM, you just need to do:</p><pre class="verbatim">python setup.py bdist_rpm
</pre><p>This will create an RPM for your specific platform and a source RPM in the directory <code>dist</code>. This RPM should be good and ready to go, so this is all you need to do! Nice and easy.</p></dd><dt class="dt-description"><span style="font-weight:bold">Macintosh</span></dt><dd class="dd-description"> &#X2013; Since Apple moved to Mac OS X, things have become much easier on the Mac. We generally
treat it as just another Unix variant, and installing Biopython from source is just as easy as on Linux.
The easiest way to get all the GCC compilers etc installed is to install Apple&#X2019;s X-Code.
We might be able to provide click and run installers for Mac OS X, but to date there hasn&#X2019;t been any demand.</dd></dl><p>Once you&#X2019;ve got a package, please test it on your system to make sure it installs everything in a good way and seems to work properly. Once you feel good about it, send it off to one of the Biopython developers (write to our main mailing list at biopython@biopython.org if you&#X2019;re not sure who to send it to) and you&#X2019;ve done it. Thanks!</p>
<!--TOC section id="sec384" Contributing Unit Tests-->
<h2 id="sec384" class="section">21.6&#XA0;&#XA0;Contributing Unit Tests</h2><!--SEC END --><p>Even if you don&#X2019;t have any new functionality to add to Biopython, but you want to write some code, please
consider extending our unit test coverage. We&#X2019;ve devoted all of Chapter&#XA0;<a href="#sec%3Aregr_test">19</a> to this topic.</p>
<!--TOC section id="sec385" Contributing Code-->
<h2 id="sec385" class="section">21.7&#XA0;&#XA0;Contributing Code</h2><!--SEC END --><p>There are no barriers to joining Biopython code development other
than an interest in creating biology-related code in Python. The
best place to express an interest is on the Biopython mailing lists
&#X2013; just let us know you are interested in coding and what kind of
stuff you want to work on. Normally, we try to have some discussion
on modules before coding them, since that helps generate good ideas
&#X2013; then just feel free to jump right in and start coding!</p><p>The main Biopython release tries to be fairly uniform and interworkable,
to make it easier for users. You can read about some of (fairly
informal) coding style guidelines we try to use in Biopython in the
contributing documentation at
<a href="http://biopython.org/wiki/Contributing"><span style="font-family:monospace">http://biopython.org/wiki/Contributing</span></a>. We also try to add code to the distribution along with tests (see Chapter&#XA0;<a href="#sec%3Aregr_test">19</a> for more info on the regression testing framework) and documentation, so that everything can stay as workable and well documented as possible (including docstrings). This is, of course, the most ideal situation, under many situations you&#X2019;ll be able to find other people on the list who will be willing to help add documentation or more tests for your code once you make it available. So, to end this paragraph like the last, feel free to start working!</p><p>Please note that to make a code contribution you must have the legal right to contribute it and license it under the Biopython license. If you wrote it all yourself, and it is not based on any other code, this shouldn&#X2019;t be a problem. However, there are issues if you want to contribute a derivative work - for example something based on GPL or LPGL licenced code would not be compatible with our license. If you have any queries on this, please discuss the issue on the biopython-dev mailing list.</p><p>Another point of concern for any additions to Biopython regards any build time or run time dependencies. Generally speaking, writing code to interact with a standalone tool (like BLAST, EMBOSS or ClustalW) doesn&#X2019;t present a big problem. However, any dependency on another library - even a Python library (especially one needed in order to compile and install Biopython like NumPy) would need further discussion.</p><p>Additionally, if you have code that you don&#X2019;t think fits in the
distribution, but that you want to make available, we maintain Script
Central (<a href="http://biopython.org/wiki/Scriptcentral"><span style="font-family:monospace">http://biopython.org/wiki/Scriptcentral</span></a>)
which has pointers to freely available code in Python for bioinformatics.</p><p>Hopefully this documentation has got you excited enough about
Biopython to try it out (and most importantly, contribute!). Thanks
for reading all the way through!</p>
<!--TOC chapter id="sec386" Appendix: Useful stuff about Python-->
<h1 id="sec386" class="chapter">Chapter&#XA0;22&#XA0;&#XA0;Appendix: Useful stuff about Python</h1><!--SEC END --><p>
<a id="sec:appendix"></a></p><p>If you haven&#X2019;t spent a lot of time programming in Python, many
questions and problems that come up in using Biopython are often
related to Python itself. This section tries to present some ideas and
code that come up often (at least for us!) while using the Biopython
libraries. If you have any suggestions for useful pointers that could
go here, please contribute!</p>
<!--TOC section id="sec387" What the heck is a handle?-->
<h2 id="sec387" class="section">22.1&#XA0;&#XA0;What the heck is a handle?</h2><!--SEC END --><p>
<a id="sec:appendix-handles"></a></p><p>Handles are mentioned quite frequently throughout this documentation,
and are also fairly confusing (at least to me!). Basically, you can
think of a handle as being a &#X201C;wrapper&#X201D; around text information.</p><p>Handles provide (at least) two benefits over plain text information:</p><ol class="enumerate" type=1><li class="li-enumerate">
They provide a standard way to deal with information stored in
different ways. The text information can be in a file, or in a
string stored in memory, or the output from a command line program,
or at some remote website, but the handle provides a common way of
dealing with information in all of these formats.</li><li class="li-enumerate">They allow text information to be read incrementally, instead
of all at once. This is really important when you are dealing with
huge text files which would use up all of your memory if you had to
load them all.
</li></ol><p>Handles can deal with text information that is being read (e.&#XA0;g.&#XA0;reading
from a file) or written (e.&#XA0;g.&#XA0;writing information to a file). In the
case of a &#X201C;read&#X201D; handle, commonly used functions are <code>read()</code>,
which reads the entire text information from the handle, and
<code>readline()</code>, which reads information one line at a time. For
&#X201C;write&#X201D; handles, the function <code>write()</code> is regularly used.</p><p>The most common usage for handles is reading information from a file,
which is done using the built-in Python function <code>open</code>. Here, we open a
handle to the file <a href="examples/m_cold.fasta">m_cold.fasta</a>
(also available online
<a href="http://biopython.org/DIST/docs/tutorial/examples/m_cold.fasta">here</a>):</p><pre class="verbatim">&gt;&gt;&gt; handle = open("m_cold.fasta", "r")
&gt;&gt;&gt; handle.readline()
"&gt;gi|8332116|gb|BE037100.1|BE037100 MP14H09 MP Mesembryanthemum ...\n"
</pre><p>Handles are regularly used in Biopython for passing information to parsers.
For example, since Biopython 1.54 the main functions in <code>Bio.SeqIO</code>
and <code>Bio.AlignIO</code> have allowed you to use a filename instead of a
handle:</p><pre class="verbatim">from Bio import SeqIO
for record in SeqIO.parse("m_cold.fasta", "fasta"):
    print(record.id, len(record))
</pre><p>On older versions of Biopython you had to use a handle, e.g.</p><pre class="verbatim">from Bio import SeqIO
handle = open("m_cold.fasta", "r")
for record in SeqIO.parse(handle, "fasta"):
    print(record.id, len(record))
handle.close()
</pre><p>This pattern is still useful - for example suppose you have a gzip
compressed FASTA file you want to parse:</p><pre class="verbatim">import gzip
from Bio import SeqIO
handle = gzip.open("m_cold.fasta.gz")
for record in SeqIO.parse(handle, "fasta"):
    print(record.id, len(record))
handle.close()
</pre><p>See Section&#XA0;<a href="#sec%3ASeqIO_compressed">5.2</a> for more examples like this,
including reading bzip2 compressed files.</p>
<!--TOC subsection id="sec388" Creating a handle from a string-->
<h3 id="sec388" class="subsection">22.1.1&#XA0;&#XA0;Creating a handle from a string</h3><!--SEC END --><p>One useful thing is to be able to turn information contained in a
string into a handle. The following example shows how to do this using
<code>cStringIO</code> from the Python standard library:</p><pre class="verbatim">&gt;&gt;&gt; my_info = 'A string\n with multiple lines.'
&gt;&gt;&gt; print(my_info)
A string
 with multiple lines.
&gt;&gt;&gt; from StringIO import StringIO
&gt;&gt;&gt; my_info_handle = StringIO(my_info)
&gt;&gt;&gt; first_line = my_info_handle.readline()
&gt;&gt;&gt; print(first_line)
A string
&lt;BLANKLINE&gt;
&gt;&gt;&gt; second_line = my_info_handle.readline()
&gt;&gt;&gt; print(second_line)
 with multiple lines.
</pre><!--TOC chapter id="sec389" References-->
<h1 id="sec389" class="chapter">References</h1><!--SEC END --><dl class="thebibliography"><dt class="dt-thebibliography">

<a id="cock2009">[1]</a></dt><dd class="dd-thebibliography">
Peter J. A. Cock, Tiago Antao, Jeffrey T. Chang, Brad A. Chapman, Cymon J. Cox, Andrew Dalke, Iddo Friedberg, Thomas Hamelryck, Frank Kauff, Bartek Wilczynski, Michiel J. L. de Hoon: &#X201C;Biopython: freely available Python tools for computational molecular biology and bioinformatics&#X201D;. <span style="font-style:italic">Bioinformatics</span> <span style="font-weight:bold">25</span> (11), 1422&#X2013;1423 (2009). <a href="http://dx.doi.org/10.1093/bioinformatics/btp163">doi:10.1093/bioinformatics/btp163</a>,
</dd><dt class="dt-thebibliography"><a id="pritchard2006">[2]</a></dt><dd class="dd-thebibliography">
Leighton Pritchard, Jennifer A. White, Paul R.J. Birch, Ian K. Toth: &#X201C;GenomeDiagram: a python package for the visualization of large-scale genomic data&#X201D;. <span style="font-style:italic">Bioinformatics</span> <span style="font-weight:bold">22</span> (5): 616&#X2013;617 (2006).
<a href="http://dx.doi.org/10.1093/bioinformatics/btk021">doi:10.1093/bioinformatics/btk021</a>,
</dd><dt class="dt-thebibliography"><a id="toth2006">[3]</a></dt><dd class="dd-thebibliography">
Ian K. Toth, Leighton Pritchard, Paul R. J. Birch: &#X201C;Comparative genomics reveals what makes an enterobacterial plant pathogen&#X201D;. <span style="font-style:italic">Annual Review of Phytopathology</span> <span style="font-weight:bold">44</span>: 305&#X2013;336 (2006).
<a href="http://dx.doi.org/10.1146/annurev.phyto.44.070505.143444">doi:10.1146/annurev.phyto.44.070505.143444</a>,
</dd><dt class="dt-thebibliography"><a id="vanderauwera2009">[4]</a></dt><dd class="dd-thebibliography">
G&#XE9;raldine A. van der Auwera, Jaroslaw E. Kr&#XF3;l, Haruo Suzuki, Brian Foster, Rob van Houdt, Celeste J. Brown, Max Mergeay, Eva M. Top: &#X201C;Plasmids captured in C. metallidurans CH34: defining the PromA family of broad-host-range plasmids&#X201D;.
<span style="font-style:italic">Antonie van Leeuwenhoek</span> <span style="font-weight:bold">96</span> (2): 193&#X2013;204 (2009).
<a href="http://dx.doi.org/10.1007/s10482-009-9316-9">doi:10.1007/s10482-009-9316-9</a>
</dd><dt class="dt-thebibliography"><a id="proux2002">[5]</a></dt><dd class="dd-thebibliography">
Caroline Proux, Douwe van Sinderen, Juan Suarez, Pilar Garcia, Victor Ladero, Gerald F. Fitzgerald, Frank Desiere, Harald Br&#XFC;ssow:
&#X201C;The dilemma of phage taxonomy illustrated by comparative genomics of Sfi21-Like Siphoviridae in lactic acid bacteria&#X201D;. <span style="font-style:italic">Journal of Bacteriology</span> <span style="font-weight:bold">184</span> (21): 6026&#X2013;6036 (2002).
<a href="http://dx.doi.org/10.1128/JB.184.21.6026-6036.2002">http://dx.doi.org/10.1128/JB.184.21.6026-6036.2002</a>
</dd><dt class="dt-thebibliography"><a id="jupe2012">[6]</a></dt><dd class="dd-thebibliography">
Florian Jupe, Leighton Pritchard, Graham J. Etherington, Katrin MacKenzie, Peter JA Cock, Frank Wright, Sanjeev Kumar Sharma1, Dan Bolser, Glenn J Bryan, Jonathan DG Jones, Ingo Hein: &#X201C;Identification and localisation of the NB-LRR gene family within the potato genome&#X201D;. <span style="font-style:italic">BMC Genomics</span> <span style="font-weight:bold">13</span>: 75 (2012).
<a href="http://dx.doi.org/10.1186/1471-2164-13-75">http://dx.doi.org/10.1186/1471-2164-13-75</a>
</dd><dt class="dt-thebibliography"><a id="cock2010">[7]</a></dt><dd class="dd-thebibliography">
Peter J. A. Cock, Christopher J. Fields, Naohisa Goto, Michael L. Heuer, Peter M. Rice: &#X201C;The Sanger FASTQ file format for sequences with quality scores, and the Solexa/Illumina FASTQ variants&#X201D;. <span style="font-style:italic">Nucleic Acids Research</span> <span style="font-weight:bold">38</span> (6): 1767&#X2013;1771 (2010). <a href="http://dx.doi.org/10.1093/nar/gkp1137">doi:10.1093/nar/gkp1137</a>
</dd><dt class="dt-thebibliography"><a id="brown1999">[8]</a></dt><dd class="dd-thebibliography">
Patrick O. Brown, David Botstein: &#X201C;Exploring the new world of the genome with DNA microarrays&#X201D;. <span style="font-style:italic">Nature Genetics</span> <span style="font-weight:bold">21</span> (Supplement 1), 33&#X2013;37 (1999). <a href="http://dx.doi.org/10.1038/4462">doi:10.1038/4462</a>
</dd><dt class="dt-thebibliography"><a id="talevich2012">[9]</a></dt><dd class="dd-thebibliography">
Eric Talevich, Brandon M. Invergo, Peter J.A. Cock, Brad A. Chapman: &#X201C;Bio.Phylo: A unified toolkit for processing, analyzing and visualizing phylogenetic trees in Biopython&#X201D;. <span style="font-style:italic">BMC Bioinformatics</span> <span style="font-weight:bold">13</span>: 209 (2012). <a href="http://dx.doi.org/10.1186/1471-2105-13-209">doi:10.1186/1471-2105-13-209</a>
</dd><dt class="dt-thebibliography"><a id="cornish1985">[10]</a></dt><dd class="dd-thebibliography">
Athel Cornish-Bowden: &#X201C;Nomenclature for incompletely specified bases in nucleic acid sequences: Recommendations 1984.&#X201D; <span style="font-style:italic">Nucleic Acids Research</span> <span style="font-weight:bold">13</span> (9): 3021&#X2013;3030 (1985). <a href="http://dx.doi.org/10.1093/nar/13.9.3021">doi:10.1093/nar/13.9.3021</a>
</dd><dt class="dt-thebibliography"><a id="cavener1987">[11]</a></dt><dd class="dd-thebibliography">
Douglas R. Cavener: &#X201C;Comparison of the consensus sequence flanking translational start sites in Drosophila and vertebrates.&#X201D; <span style="font-style:italic">Nucleic Acids Research</span> <span style="font-weight:bold">15</span> (4): 1353&#X2013;1361 (1987). <a href="http://dx.doi.org/10.1093/nar/15.4.1353">doi:10.1093/nar/15.4.1353</a>
</dd><dt class="dt-thebibliography"><a id="bailey1994">[12]</a></dt><dd class="dd-thebibliography">
Timothy L. Bailey and Charles Elkan: &#X201C;Fitting a mixture model by expectation maximization to discover motifs in biopolymers&#X201D;, <span style="font-style:italic">Proceedings of the Second International Conference on Intelligent Systems for Molecular Biology</span> 28&#X2013;36. AAAI Press, Menlo Park, California (1994).
</dd><dt class="dt-thebibliography"><a id="chapman2000">[13]</a></dt><dd class="dd-thebibliography">
Brad Chapman and Jeff Chang: &#X201C;Biopython: Python tools for computational biology&#X201D;. <span style="font-style:italic">ACM SIGBIO Newsletter</span> <span style="font-weight:bold">20</span> (2): 15&#X2013;19 (August 2000).
</dd><dt class="dt-thebibliography"><a id="dehoon2004">[14]</a></dt><dd class="dd-thebibliography">
Michiel J. L. de Hoon, Seiya Imoto, John Nolan, Satoru Miyano: &#X201C;Open source clustering software&#X201D;. <span style="font-style:italic">Bioinformatics</span> <span style="font-weight:bold">20</span> (9): 1453&#X2013;1454 (2004). <a href="http://dx.doi.org/10.1093/bioinformatics/bth078">doi:10.1093/bioinformatics/bth078</a>
</dd><dt class="dt-thebibliography"><a id="eisen1998">[15]</a></dt><dd class="dd-thebibliography">
Michiel B. Eisen, Paul T. Spellman, Patrick O. Brown, David Botstein: &#X201C;Cluster analysis and display of genome-wide expression patterns&#X201D;. <span style="font-style:italic">Proceedings of the National Academy of Science USA</span> <span style="font-weight:bold">95</span> (25): 14863&#X2013;14868 (1998). <a href="http://dx.doi.org/10.1073/pnas.96.19.10943-c">doi:10.1073/pnas.96.19.10943-c</a>
</dd><dt class="dt-thebibliography"><a id="golub1971">[16]</a></dt><dd class="dd-thebibliography">
Gene H. Golub, Christian Reinsch: &#X201C;Singular value decomposition and least squares solutions&#X201D;. In <span style="font-style:italic">Handbook for Automatic Computation</span>, <span style="font-weight:bold">2</span>, (Linear Algebra) (J. H. Wilkinson and C. Reinsch, eds), 134&#X2013;151. New York: Springer-Verlag (1971).
</dd><dt class="dt-thebibliography"><a id="golub1989">[17]</a></dt><dd class="dd-thebibliography">
Gene H. Golub, Charles F. Van Loan: <span style="font-style:italic">Matrix computations</span>, 2nd edition (1989).
</dd><dt class="dt-thebibliography"><a id="hamelryck2003a">[18]</a></dt><dd class="dd-thebibliography">
Thomas Hamelryck and Bernard Manderick: 11PDB parser and structure class
implemented in Python&#X201D;. <span style="font-style:italic">Bioinformatics</span>, <span style="font-weight:bold">19</span> (17): 2308&#X2013;2310 (2003) <a href="http://dx.doi.org/10.1093/bioinformatics/btg299">doi: 10.1093/bioinformatics/btg299</a>. 
</dd><dt class="dt-thebibliography"><a id="hamelryck2003b">[19]</a></dt><dd class="dd-thebibliography">
Thomas Hamelryck: &#X201C;Efficient identification of side-chain patterns using a multidimensional index tree&#X201D;. <span style="font-style:italic">Proteins</span> <span style="font-weight:bold">51</span> (1): 96&#X2013;108 (2003). <a href="http://dx.doi.org/10.1002/prot.10338">doi:10.1002/prot.10338</a>
</dd><dt class="dt-thebibliography"><a id="hamelryck2005">[20]</a></dt><dd class="dd-thebibliography">
Thomas Hamelryck: &#X201C;An amino acid has two sides; A new 2D measure provides a different view of solvent exposure&#X201D;. <span style="font-style:italic">Proteins</span> <span style="font-weight:bold">59</span> (1): 29&#X2013;48 (2005). <a href="http://dx.doi.org/10.1002/prot.20379">doi:10.1002/prot.20379</a>.
</dd><dt class="dt-thebibliography"><a id="hartigan1975">[21]</a></dt><dd class="dd-thebibliography">
John A. Hartiga. <span style="font-style:italic">Clustering algorithms</span>. New York: Wiley (1975).
</dd><dt class="dt-thebibliography"><a id="jain1988">[22]</a></dt><dd class="dd-thebibliography">
Anil L. Jain, Richard C. Dubes: <span style="font-style:italic">Algorithms for clustering data</span>. Englewood Cliffs, N.J.: Prentice Hall (1988).
</dd><dt class="dt-thebibliography"><a id="kachitvichyanukul1988">[23]</a></dt><dd class="dd-thebibliography">
Voratas Kachitvichyanukul, Bruce W. Schmeiser: Binomial Random Variate Generation. <span style="font-style:italic">Communications of the ACM</span> <span style="font-weight:bold">31</span> (2): 216&#X2013;222 (1988). <a href="http://dx.doi.org/10.1145/42372.42381">doi:10.1145/42372.42381</a>
</dd><dt class="dt-thebibliography"><a id="kohonen1997">[24]</a></dt><dd class="dd-thebibliography">
Teuvo Kohonen: &#X201C;Self-organizing maps&#X201D;, 2nd Edition. Berlin; New York: Springer-Verlag (1997).
</dd><dt class="dt-thebibliography"><a id="lecuyer1988">[25]</a></dt><dd class="dd-thebibliography">
Pierre L&#X2019;Ecuyer: &#X201C;Efficient and Portable Combined Random Number Generators.&#X201D;
<span style="font-style:italic">Communications of the ACM</span> <span style="font-weight:bold">31</span> (6): 742&#X2013;749,774 (1988). <a href="http://dx.doi.org/10.1145/62959.62969">doi:10.1145/62959.62969</a>
</dd><dt class="dt-thebibliography"><a id="majumdar2005">[26]</a></dt><dd class="dd-thebibliography">
Indraneel Majumdar, S. Sri Krishna, Nick V. Grishin: &#X201C;PALSSE: A program to delineate linear secondary structural elements from protein structures.&#X201D; <span style="font-style:italic">BMC Bioinformatics</span>, <span style="font-weight:bold">6</span>: 202 (2005). <a href="http://dx.doi.org/10.1186/1471-2105-6-202">doi:10.1186/1471-2105-6-202</a>.
</dd><dt class="dt-thebibliography"><a id="matys2003">[27]</a></dt><dd class="dd-thebibliography">
V. Matys, E. Fricke, R. Geffers, E. G&#XF6;ssling, M. Haubrock, R. Hehl, K. Hornischer, D. Karas, A.E. Kel, O.V. Kel-Margoulis, D.U. Kloos, S. Land, B. Lewicki-Potapov, H. Michael, R. M&#XFC;nch, I. Reuter, S. Rotert, H. Saxel, M. Scheer, S. Thiele, E. Wingender E: &#X201C;TRANSFAC: transcriptional regulation, from patterns to profiles.&#X201D; Nucleic Acids Research <span style="font-weight:bold">31</span> (1): 374&#X2013;378 (2003). <a href="http://dx.doi.org/10.1093/nar/gkg108">doi:10.1093/nar/gkg108</a>
</dd><dt class="dt-thebibliography"><a id="sibson1973">[28]</a></dt><dd class="dd-thebibliography">
Robin Sibson: &#X201C;SLINK: An optimally efficient algorithm for the single-link cluster method&#X201D;. <span style="font-style:italic">The Computer Journal</span> <span style="font-weight:bold">16</span> (1): 30&#X2013;34 (1973). <a href="http://dx.doi.org/10.1093/comjnl/16.1.30">doi:10.1093/comjnl/16.1.30</a>
</dd><dt class="dt-thebibliography"><a id="snedecor1989">[29]</a></dt><dd class="dd-thebibliography">
George W. Snedecor, William G. Cochran: <span style="font-style:italic">Statistical methods</span>. Ames, Iowa: Iowa State University Press (1989).
</dd><dt class="dt-thebibliography"><a id="tamayo1999">[30]</a></dt><dd class="dd-thebibliography">
Pablo Tamayo, Donna Slonim, Jill Mesirov, Qing Zhu, Sutisak Kitareewan, Ethan Dmitrovsky, Eric S. Lander, Todd R. Golub: &#X201C;Interpreting patterns of gene expression with self-organizing maps: Methods and application to hematopoietic differentiation&#X201D;. <span style="font-style:italic">Proceedings of the National Academy of Science USA</span> <span style="font-weight:bold">96</span> (6): 2907&#X2013;2912 (1999). <a href="http://dx.doi.org/10.1073/pnas.96.6.2907">doi:10.1073/pnas.96.6.2907</a>
</dd><dt class="dt-thebibliography"><a id="tryon1970">[31]</a></dt><dd class="dd-thebibliography">
Robert C. Tryon, Daniel E. Bailey: <span style="font-style:italic">Cluster analysis</span>. New York: McGraw-Hill (1970).
</dd><dt class="dt-thebibliography"><a id="tukey1977">[32]</a></dt><dd class="dd-thebibliography">
John W. Tukey: &#X201C;Exploratory data analysis&#X201D;. Reading, Mass.: Addison-Wesley Pub. Co. (1977).
</dd><dt class="dt-thebibliography"><a id="yeung2001">[33]</a></dt><dd class="dd-thebibliography">
Ka Yee Yeung, Walter L. Ruzzo: &#X201C;Principal Component Analysis for clustering gene expression data&#X201D;. <span style="font-style:italic">Bioinformatics</span> <span style="font-weight:bold">17</span> (9): 763&#X2013;774 (2001). <a href="http://dx.doi.org/10.1093/bioinformatics/17.9.763">doi:10.1093/bioinformatics/17.9.763</a>
</dd><dt class="dt-thebibliography"><a id="saldanha2004">[34]</a></dt><dd class="dd-thebibliography">
Alok Saldanha: &#X201C;Java Treeview&#X2014;extensible visualization of microarray data&#X201D;. <span style="font-style:italic">Bioinformatics</span> <span style="font-weight:bold">20</span> (17): 3246&#X2013;3248 (2004). 
<a href="http://dx.doi.org/10.1093/bioinformatics/bth349">http://dx.doi.org/10.1093/bioinformatics/bth349</a>
</dd></dl><!--CUT END -->
<!--HTMLFOOT-->
<!--ENDHTML-->
<!--FOOTER-->
<hr style="height:2"><blockquote class="quote"><em>This document was translated from L<sup>A</sup>T<sub>E</sub>X by
</em><a href="http://hevea.inria.fr/index.html"><em>H</em><em><span style="font-size:small"><sup>E</sup></span></em><em>V</em><em><span style="font-size:small"><sup>E</sup></span></em><em>A</em></a><em>.</em></blockquote></body>
</html>