File: last-parallel.html

package info (click to toggle)
last-align 830-1
links: PTS, VCS
area: main
in suites: stretch
size: 3,240 kB
ctags: 3,201
sloc: cpp: 40,808; python: 1,910; ansic: 1,188; makefile: 385; sh: 232
file content (395 lines) | stat: -rw-r--r-- 9,819 bytes
parent folder | download | duplicates (2)
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.6: http://docutils.sourceforge.net/" />
<title>Running LAST in parallel</title>
<style type="text/css">

/*
:Author: David Goodger (goodger@python.org)
:Id: $Id: html4css1.css 5951 2009-05-18 18:03:10Z milde $
:Copyright: This stylesheet has been placed in the public domain.

Default cascading style sheet for the HTML output of Docutils.

See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/

/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
  border: 0 }

table.borderless td, table.borderless th {
  /* Override padding for "table.docutils td" with "! important".
     The right padding separates the table cells. */
  padding: 0 0.5em 0 0 ! important }

.first {
  /* Override more specific margin styles with "! important". */
  margin-top: 0 ! important }

.last, .with-subtitle {
  margin-bottom: 0 ! important }

.hidden {
  display: none }

a.toc-backref {
  text-decoration: none ;
  color: black }

blockquote.epigraph {
  margin: 2em 5em ; }

dl.docutils dd {
  margin-bottom: 0.5em }

/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
  font-weight: bold }
*/

div.abstract {
  margin: 2em 5em }

div.abstract p.topic-title {
  font-weight: bold ;
  text-align: center }

div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
  margin: 2em ;
  border: medium outset ;
  padding: 1em }

div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
  font-weight: bold ;
  font-family: sans-serif }

div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title {
  color: red ;
  font-weight: bold ;
  font-family: sans-serif }

/* Uncomment (and remove this text!) to get reduced vertical space in
   compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
  margin-bottom: 0.5em }

div.compound .compound-last, div.compound .compound-middle {
  margin-top: 0.5em }
*/

div.dedication {
  margin: 2em 5em ;
  text-align: center ;
  font-style: italic }

div.dedication p.topic-title {
  font-weight: bold ;
  font-style: normal }

div.figure {
  margin-left: 2em ;
  margin-right: 2em }

div.footer, div.header {
  clear: both;
  font-size: smaller }

div.line-block {
  display: block ;
  margin-top: 1em ;
  margin-bottom: 1em }

div.line-block div.line-block {
  margin-top: 0 ;
  margin-bottom: 0 ;
  margin-left: 1.5em }

div.sidebar {
  margin: 0 0 0.5em 1em ;
  border: medium outset ;
  padding: 1em ;
  background-color: #ffffee ;
  width: 40% ;
  float: right ;
  clear: right }

div.sidebar p.rubric {
  font-family: sans-serif ;
  font-size: medium }

div.system-messages {
  margin: 5em }

div.system-messages h1 {
  color: red }

div.system-message {
  border: medium outset ;
  padding: 1em }

div.system-message p.system-message-title {
  color: red ;
  font-weight: bold }

div.topic {
  margin: 2em }

h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
  margin-top: 0.4em }

h1.title {
  text-align: center }

h2.subtitle {
  text-align: center }

hr.docutils {
  width: 75% }

img.align-left, .figure.align-left{
  clear: left ;
  float: left ;
  margin-right: 1em }

img.align-right, .figure.align-right {
  clear: right ;
  float: right ;
  margin-left: 1em }

.align-left {
  text-align: left }

.align-center {
  clear: both ;
  text-align: center }

.align-right {
  text-align: right }

/* reset inner alignment in figures */
div.align-right {
  text-align: left }

/* div.align-center * { */
/*   text-align: left } */

ol.simple, ul.simple {
  margin-bottom: 1em }

ol.arabic {
  list-style: decimal }

ol.loweralpha {
  list-style: lower-alpha }

ol.upperalpha {
  list-style: upper-alpha }

ol.lowerroman {
  list-style: lower-roman }

ol.upperroman {
  list-style: upper-roman }

p.attribution {
  text-align: right ;
  margin-left: 50% }

p.caption {
  font-style: italic }

p.credits {
  font-style: italic ;
  font-size: smaller }

p.label {
  white-space: nowrap }

p.rubric {
  font-weight: bold ;
  font-size: larger ;
  color: maroon ;
  text-align: center }

p.sidebar-title {
  font-family: sans-serif ;
  font-weight: bold ;
  font-size: larger }

p.sidebar-subtitle {
  font-family: sans-serif ;
  font-weight: bold }

p.topic-title {
  font-weight: bold }

pre.address {
  margin-bottom: 0 ;
  margin-top: 0 ;
  font: inherit }

pre.literal-block, pre.doctest-block {
  margin-left: 2em ;
  margin-right: 2em }

span.classifier {
  font-family: sans-serif ;
  font-style: oblique }

span.classifier-delimiter {
  font-family: sans-serif ;
  font-weight: bold }

span.interpreted {
  font-family: sans-serif }

span.option {
  white-space: nowrap }

span.pre {
  white-space: pre }

span.problematic {
  color: red }

span.section-subtitle {
  /* font-size relative to parent (h1..h6 element) */
  font-size: 80% }

table.citation {
  border-left: solid 1px gray;
  margin-left: 1px }

table.docinfo {
  margin: 2em 4em }

table.docutils {
  margin-top: 0.5em ;
  margin-bottom: 0.5em }

table.footnote {
  border-left: solid 1px black;
  margin-left: 1px }

table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
  padding-left: 0.5em ;
  padding-right: 0.5em ;
  vertical-align: top }

table.docutils th.field-name, table.docinfo th.docinfo-name {
  font-weight: bold ;
  text-align: left ;
  white-space: nowrap ;
  padding-left: 0 }

h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
  font-size: 100% }

ul.auto-toc {
  list-style-type: none }

</style>
<style type="text/css">

/* Style sheet for LAST HTML documents */
h1 { color: navy }
h2 { color: teal }
div.document { margin-left: auto; margin-right: auto; max-width: 45em }
strong { color: red }
.option-list td { padding-bottom: 1em }
table.field-list { border: thin solid green }

</style>
</head>
<body>
<div class="document" id="running-last-in-parallel">
<h1 class="title">Running LAST in parallel</h1>

<p>You can make LAST faster by running it on multiple CPUs / cores.  The
easiest way is with lastal's -P option:</p>
<pre class="literal-block">
lastal -P4 my-index queries.fasta &gt; out.maf
</pre>
<p>This will use 4 parallel threads.  If you specify -P0, it will use as
many threads as your computer claims it can handle simultaneously.</p>
<p>This works by aligning different query sequences in different threads
- so if you only have one query you won't get any parallelization!</p>
<div class="section" id="dealing-with-very-long-query-sequences">
<h2>Dealing with very long query sequences</h2>
<p>lastal aligns one &quot;batch&quot; of queries at a time, so if the batch has
only one query you won't get any parallelization.  This can be fixed
by increasing the batch size, with option -i:</p>
<pre class="literal-block">
lastal -P4 -i3G my-index queries.fasta &gt; out.maf
</pre>
<p>This specifies a batch size of 3 gibi-bytes.  The downside is that
more memory is needed to hold the batch and its alignments.</p>
</div>
<div class="section" id="dealing-with-pipelines">
<h2>Dealing with pipelines</h2>
<p>If you have a multi-command &quot;pipeline&quot;, such as:</p>
<pre class="literal-block">
lastal -P4 my-index queries.fasta | last-split &gt; out.maf
</pre>
<p>then the -P option may help, because lastal is often the slowest step,
but it would be nice to parallelize the whole thing.  Unfortunately,
last-split doesn't have a -P option, and even if it did, the pipe
between the commands would become a bottleneck.</p>
<p>You can use parallel-fasta and parallel-fastq (which accompany LAST,
but require <a class="reference external" href="http://www.gnu.org/software/parallel/">GNU parallel</a> to
be installed).  These commands read sequence data, split it into
blocks (with a whole number of sequences per block), and run the
blocks in parallel through any command or pipeline you specify, using
all your CPU cores.  Here are some examples.</p>
<p>Instead of this:</p>
<pre class="literal-block">
lastal mydb queries.fa &gt; myalns.maf
</pre>
<p>try this:</p>
<pre class="literal-block">
parallel-fasta &quot;lastal mydb&quot; &lt; queries.fa &gt; myalns.maf
</pre>
<p>Instead of this:</p>
<pre class="literal-block">
lastal -Q1 -D100 db q.fastq | last-split &gt; out.maf
</pre>
<p>try this:</p>
<pre class="literal-block">
parallel-fastq &quot;lastal -Q1 -D100 db | last-split&quot; &lt; q.fastq &gt; out.maf
</pre>
<p>Instead of this:</p>
<pre class="literal-block">
zcat queries.fa.gz | lastal mydb &gt; myalns.maf
</pre>
<p>try this:</p>
<pre class="literal-block">
zcat queries.fa.gz | parallel-fasta &quot;lastal mydb&quot; &gt; myalns.maf
</pre>
<p>Notes:</p>
<ul>
<li><p class="first">parallel-fasta and parallel-fastq simply execute GNU parallel with a
few options for fasta or fastq: you can specify other GNU parallel
options to control the number of simultaneous jobs, use remote
computers, get the output in the same order as the input, etc.</p>
</li>
<li><p class="first">parallel-fastq assumes that each fastq record is 4 lines, so there
should be no line wrapping or blank lines.</p>
</li>
</ul>
</div>
</div>
</body>
</html>